6 Mistakes People Make While Running Split Tests, And How to Avoid Them

In 2011 Google executed more than 7,000 split tests on their algorithm!

Yes, you read that right. The brand that has become a verb fully endorses data backed decision making over gut instinct.

While this may seem like a no-brainer and the phrase “A/B testing” is starting to catch on, there is a lot of misconception around the concept of split testing, how it differs from its more popular cousin and the pitfalls that businesses should expect to find (and avoid) on the journey to growth with conversion rate optimization.

Popular but Mis-Executed:

While 64 percent of companies agree that split testing testing is easy to execute, 7 percent say it’s not easy to perform the tests (source: finance online).

TRUTH: Split testing is a popular form of conversion rate optimization testing for marketers these days.

ALSO, A TRUTH: Not all businesses are getting the most out of it. A study has shown that only 12.5% of split tests yield better results.

In fact, many companies that run split tests for the first time complain that they don’t get the anticipated lift from their efforts.

Well, getting great results with your split tests isn’t a walk in the park. You must know the most common mistakes marketers make and how to avoid them, and that’s exactly what we will talk about in this article.

Let’s get rolling.

What is Split Testing and How does it Work?

Split testing is a conversion rate optimization practice where marketers compare two completely different versions of a web page to identify which one provides better results according to explicit metrics.

For instance, one webpage could have a call-to-action above the fold and another one would have a call-to-action below the fold.

Once the test is done, you can identify which version drove visitors to take action.

The aim of running a split test on a website is to improve conversions.

You can also run a split test to compare your landing page and homepage to identify which one better converts visitors in the context of a quantifiable goal. Like sign-ups. Or purchases.

Traffic coming to your site is spread randomly between the two pages and the performance of each page is tracked and analyzed. The results are then compared to determine the winner.

Now That You Know What Split Tests are All About, Let’s Look at Some Common Mistakes that Derail Efforts

1. Pre-Testing and After-Testing

This is perhaps the common mistake businesses make while running a split test. To get better results with your tests, you need to test two versions without changing them during the course of the experiment. With pre-testing and after-testing, marketers measure conversions for a given period, make adjustments and then measure conversions again for a given period.

That means different versions are tested for different periods.

Given that different days of the week have different traffic quality and the constant change in economic climate amongst people, running before and after tests might not give you better results.

This change in traffic could hurt conversion rates and that’s why it is recommended to avoid running before and after tests but to test two or more versions simultaneously.

This way, the two versions are shown to visitors throughout the testing period and this increases the likelihood of getting relevant results.

When you run pre-tests and after-tests you will not be able to ascertain how relevant the results are and this may not provide you with conclusive data about the best version.

2. Implementing Other People’s Ideas

This one is a no-brainer but worth iterating.

There are plenty of best practices and case studies out there. And they serve as inspiration for testers.

But at no point of time should the work or the result of another business inspire your split tests.

This applies to other classes of experimentation like A/B testing as well.

This is because of two reasons:

• You defeat the purpose of testing, which is to remove bias for your specific case and move forward with a confident data backed decision. In essence, you don’t “test”. You might as well replicate what the other site has done.
• You miss out on exploring low hanging fruits for your business. The site you are emulating may have pitched two landing pages against each other. But in your case, maybe pitching your landing page against your home page makes better sense given how your traffic finds you. (Look at your analytics set up to determine this!)

3. Anticipating Great Results from Minor Changes

Well, you don’t have to make huge changes to see great results. You need to make reasonable changes backed by data.

But another mistake marketers make is to anticipate huge results from small changes.

According to Edu birdie sometimes modifying the headline or changing the color of the call-to-action button can bring better results but not always.

A good rule of thumb is to test radical changes to ascertain how they impact conversions. After you have identified how a change increases conversions, you can then keep on adjusting more till you see great results.

An example of a change that produced great results is when Crazy Egg changed their homepage to a lengthy sales letter. After this change delivered a huge win, they tested their call-to-action buttons and made several other changes to increase conversions.

The whole purpose of a split test is to gauge the general preference of your audience. So, use the flexibility to test radically different pages and versions.

Once the trend is clear, A/B testing and multivariate testing can help zero in on the granular aspects of what works better for your traffic.

4. Using the Wrong Tool

As seen in the papers owl reviews almost all content management platforms offer split testing features. Nevertheless, if you want to run a serious split test, you would get great results and better experience when you test with a tool designed specifically for the job.

There are plenty of split testing tools out there – Convert Experiences, for example is a platform that makes split testing easy and cost effective.

Many people make the mistake of assuming that add-ons or plugins are enough to run serious split tests. This is rarely ever the case.

5. Ignoring Mobile Traffic

According to Statista Mobile traffic has surpassed desktop. Unfortunately, many people make the mistake of choosing not to split test with mobile traffic.

Users who use mobile phones are inherently more impatient in their browsing habits. They may not have the patience of scrolling through long form sales pages or home pages choc full of links.

Split test your regular page versus a version made especially for mobile traffic. This may just add lots of conversion percentage points to your micro and macro goals.

6. Frenzied Testing

You could have tons of people visiting your website but it’s not recommended that you run several tests within a short time frame.

After all, when will you be able to collect statistical data, analyze it, evaluate the results, and run another test? You might end up not measuring results accurately and worse of all – not knowing what decision to make.

The expert marketers at zipjob advise people to collect useful statistical data, evaluate the outcome, and then run an informed test based on the information collected.

While test frequency is a mark of CRO program maturity … quality should never be compromised for the sake of quantity. The happy balance is somewhere in the middle.

Final Thoughts

If you have started running split tests on your own, congratulations! That will help you make informed improvements that can lead to a huge increase in revenue.

When you fail, remember the words of Oli Gardner, renowned CRO expert who once said “The best thing about a failed (A/B) test is that it kicks you in the crotch, reddens your cheeks, and make you try harder next time”

Just avoid these 6 pitfalls and you should be fine!

Happy split testing!