When to Stop A/B Testing

When to Stop A/B Testing

Always be up to Date subscribe to updates - November 26, 2014

In her latest webinar, Jana Fung of Mixrank, talks about A/B testing and knowing when to stop. Fung shared that it is important that we perform tests on our landing pages in order to improve our conversion rates. Through testing, we can gather statistical information on what our audiences prefer to see on our landing pages. Testing a landing page should only have one goal – to be well A/B testing software (Convert Experiments)” href=”https://www.convert.com”>optimized for conversion.

Testing Options

We previously discussed about the four different types of landing page test, namely:

  • Template Variation Test: Wherein we use the cookie technology to measure results
  • Usability Test: A test using eye-tracking technology
  • A/B Test or Split Test: A simple comparison test of two different variations where we only need a web analytics tool to see which version gets a higher conversion. Aka A/B testing software.
  • Multivariate Test: This is similar to split test but has many versions that you can test on.

When to Stop A/B Testing

Effective testing

We have also gone through a checklist of variables that are required to conduct a successful landing page test. This is where the scientific method behind conversion optimization comes in. The checklist involves setting a baseline, making a hypothesis, the actual testing, ensuring every version gets the same traffic as the others, and should be ran simultaneously.

Knowing the right moment

But, how will we know when to keep our test running or when to stop it.

Either of these causes high opportunity costs because you will miss more conversion from the winning landing page. For example, if you’re running a landing page test and you stop it too early and end up picking the wrong landing page, you’re going to miss an opportunity to increase your conversion rate because you chose to run the losing landing page. Likewise, if you keep a test run for too long even after you have determined which landing page has the higher conversion, you will miss those who dropped off from the losing landing page. And you lose conversion by running the losing landing page for too long.

Make sure that you don’t stop your testing too early. Keep in mind that results from testing has a possibility of randomness. Any conversion or drop off has the chance of being random and unusual out of the norm. The point of a landing page test is to reach significant results. You can only reach a statistically significant result if you run a landing page test long enough to collect enough data wherein you can draw conclusions. As mentioned in this article in ConversionXL:

We need to rule out seasonality and test for full weeks. Did you start the test on Monday? Then you need to end it on a Monday as well. Why? Because your conversion rate can vary greatly depending on the day of the week.

How long is long enough?

As a rule of thumb, 30 conversion goals from a single version of a landing page will give enough data to see the Trans and identify which landing page performed better over that period of time. Around 30 conversion goals must be reached by any landing page you are testing. With this information you will be able to see which one outperforms the others.

As a side note, testing may take a little longer when you’re using a multivariate test. As discussed previously, because you’re splitting up your traffic between several variations of your landing page, it may take time to reach 30 conversions. But once you reach that goal you’ll be able to determine which landing page is dragging in all of your conversions and boosting the overall conversion rate. From there you can stop running the other variations. If you want to find out more about testing, you can view Jana Fung’s entire webinar here.

Download this infographic

Embed Our Infographic On Your Site!

Originally published November 26, 2014 - Updated October 20, 2017


Signup to our monthly newsletter to get the best of our content with the latest
Conversion and A/B Testing resources right in your inbox.

Written by Dennis van der Heijden

Co-founder and CEO of Convert.com passionate in building communities that care. Trying to make that happen inside and outside Convert. I love working with my team to make our A/B testing software better for agencies and e-commerce clients.
Guest Post Form

We have brought thought leaders, influencers, visionaries and veterans to our tribe. Now it’s your turn. If you have something worthwhile to share with a large community of savvy testers, go ahead and pitch your post idea. We’re listening.

Fill out our guest post form

CATEGORIES Articles Blogs

[hclightbox id='5' text='Anchor text']