When to Stop A/B Testing

In her latest webinar, Jana Fung of Mixrank, talks about A/B testing and knowing when to stop. Fung shared that it is important that we perform tests on our landing pages in order to improve our conversion rates. Through testing, we can gather statistical information on what our audiences prefer to see on our landing pages. Testing a landing page should only have one goal – to be well A/B testing software (Convert Experiments)” href=”https://www.convert.com”>optimized for conversion.

Testing Options

We previously discussed about the four different types of landing page test, namely:

• Template Variation Test: Wherein we use the cookie technology to measure results
• Usability Test: A test using eye-tracking technology
• A/B Test or Split Test: A simple comparison test of two different variations where we only need a web analytics tool to see which version gets a higher conversion. Aka A/B testing software.
• Multivariate Test: This is similar to split test but has many versions that you can test on.

Effective testing

We have also gone through a checklist of variables that are required to conduct a successful landing page test. This is where the scientific method behind conversion optimization comes in. The checklist involves setting a baseline, making a hypothesis, the actual testing, ensuring every version gets the same traffic as the others, and should be ran simultaneously.

Knowing the right moment

But, how will we know when to keep our test running or when to stop it.

[Tweet “A common mistake among marketers is that they stop a test too early or run the test too long”]

Either of these causes high opportunity costs because you will miss more conversion from the winning landing page. For example, if you’re running a landing page test and you stop it too early and end up picking the wrong landing page, you’re going to miss an opportunity to increase your conversion rate because you chose to run the losing landing page. Likewise, if you keep a test run for too long even after you have determined which landing page has the higher conversion, you will miss those who dropped off from the losing landing page. And you lose conversion by running the losing landing page for too long.

Make sure that you don’t stop your testing too early. Keep in mind that results from testing has a possibility of randomness. Any conversion or drop off has the chance of being random and unusual out of the norm. The point of a landing page test is to reach significant results. You can only reach a statistically significant result if you run a landing page test long enough to collect enough data wherein you can draw conclusions. As mentioned in this article in ConversionXL:

We need to rule out seasonality and test for full weeks. Did you start the test on Monday? Then you need to end it on a Monday as well. Why? Because your conversion rate can vary greatly depending on the day of the week.

How long is long enough?

As a rule of thumb, 30 conversion goals from a single version of a landing page will give enough data to see the Trans and identify which landing page performed better over that period of time. Around 30 conversion goals must be reached by any landing page you are testing. With this information you will be able to see which one outperforms the others.

As a side note, testing may take a little longer when you’re using a multivariate test. As discussed previously, because you’re splitting up your traffic between several variations of your landing page, it may take time to reach 30 conversions. But once you reach that goal you’ll be able to determine which landing page is dragging in all of your conversions and boosting the overall conversion rate. From there you can stop running the other variations. If you want to find out more about testing, you can view Jana Fung’s entire webinar here.

Download this infographic