A/B testing software Always Slows Down The Site

Every A/B testing software application that is available right now and will ever be created slows down your site. Period! Tweet This
A/B testing softwareSince we sell A/B testing software this is not easy to admit, but it’s part of the optimization effort. Every system you use to optimize conversion will have a negative impact on the website performance. It might be 0.05 or even 0.2 seconds, the difference in these different systems is for one part related to the softwares’ infrastructure and the other part.. your own budget for a tool (even 200ms does not seem to even affect conversion rates)

There are software solutions that replace your current DNS solution where all the traffic goes through their DNS and there are solutions where additional tracking beacons and tags are used to alter the user’s experience by presenting a variation. Both solutions will have a negative impact on site’s performance since there are either new routes introduced or new tags installed. The site performance is really important but the main A/B testing tools available on the market all manage to keep their load under 0.2 seconds (most of the time) and that is a great improvement compared to where we were a few years ago.

There is no research that shows a direct correlation between conversion decrease between a site that loads two seconds and the same site that loads three seconds, see some research that is related to this subject. But over three seconds there are strong signals that there is a negative effect on conversion rates around 6.7% for each second delay over that (see also TagMan revenue loss calculator here, but that one starts at 4 seconds) . Make sure that you page loads around 3 seconds and you will be making a lot of visitors happy (see O’Reilly post) . If you can make your site load in two seconds, you are my hero! The real risk of adding a tag to your site and start A/B testing is not really the 0.05 to 0.02 seconds delay, there are bigger risks you are taking than those milliseconds of added delay. The big risk in A/B testing is actually the company/person designing the test not the Javascript delay Tweet This. More about that problem in next the blogpost.

Update: August 29, 2013: Added link to new blogpost: Seven Out Of Ten of All A/B Tests Fail