What You Should Know About A/B Testing

What You Should Know About A/B Testing

Dennis van der Heijden
September 5, 2014

It’s a known fact that, in the virtual domain, A/B testing is indeed a vital tool in improving conversion rates and generating profits. Since A/B testing makes it possible for web developers and entrepreneurs to run a series of tests on their web page before implementation, it also poses fewer mistakes if done with expertise. If you are not a pro in this area, then you have definitely clicked the right page.

In Josh Krafchin’s recent webinar, he talks about the pitfalls and advantages of A/B testing. Below are some of the most frequent issues that circulate regarding A/B tests. We and Josh (of Clever Zebo) If you are amongst the thousands of people who are curious about the dos and don’ts and the ins and outs of A/B tests, prepare your mind because you are definitely in for a treat.

A/B Testing: Establishing Clear Goals and Use Specific Variables

[Tweet “A/B testers do not get the most solid results because of lack of clarity. #abtesting “]

A/B testers do not get the most solid results because of one very simple factor—lack of clarity. Once you put yourself out there, you must first make sure that you have already created clear goals. Otherwise, you will only gain poor and vague results that will not help you improve conversion rates and gain profits.

Therefore, be specific on the variables you would use and test one design at a time. It may be quite tempting to run multiple treatments all at once, but unfortunately, you will only waste time in doing so because you would not be able to gain the best results and weigh the pros and cons in the most enlightened perspective.

Counting Visits for A/B Testing


Since the main essence of A/B testing is getting good conversion rates, a lot of people often wonder the number of visits a site should gain to get good results. The answer is quite simple. To get the best results, you must first ensure that your site has already reached a high statistical significance.

Luckily, there are many tools you can use that can help you indicate if your site has already gained the relevant statistical significance it needs. Conversely, if you do not have such tools at hand, you can always look for free calculators that could help you out in measuring statistical significance.

[Tweet “You must 1st ensure that your site has reached a high statistical significance. #abtesting “]

What is Null Hypothesis For in A/B Testing?

Null hypothesis is a major factor in reaching statistical significance. Once you give enough time for A/B testing, you would be able to either prove or disprove the null hypothesis.

Therefore, the more efforts and the longer the time you exert for A/B testing, the more you would be able to analyse the null hypothesis and gain the best results for your site to run well.

Optimizing conversion rates is not exciting; it’s boring, repetitive, detailed, but necessary work, much like general management. – Bryan Eisenberg

A/B Testing and SEO: Friend or Foe?

If you have heard of the notion that A/B testing has a negative effect on SEO because it could be categorized under the duplicate content, there is no reason for you to be worried because this is a complete misconception.

In reality, A/B testing could greatly improve your site’s ranking because it focuses on maintaining the quality and functionality of your site.

[Tweet “#ABtesting improves your site’s ranking (#SEO) due it maintains your site’s quality.”]

Where to Run A/B Tests

What You Should Know A/B Testing

Contrary to popular belief, A/B tests are not restricted to websites and landing pages alone. A/B tests are also best for email, call-to-action, and PPC campaigns.

The variables you need to include for email testing are personalization features, subject line, sender name and many others. A/B tests for paid search ad campaigns, on the other hand, can be used for keywords, body text, headlines and link texts.

Lastly, A/B testing is implemented for changing texts for CTA’s, as well as its shape, color and placement.

Testing Credibility

In the event that you see loopholes in the result of your A/B test and you have gained essential information that proves that there are discrepancies, what you can do best is run the same test all over again. This will enable you to test the credibility of the results. This will also make sure that one test is not merely an anomaly or that the results were somehow different on that specific test.

Starting Up A/B testing

Jumpstarting A/B test can be done by using software tools like our Convert Experiment. Even Google provides a free tool called Content Experiments in Google Analytics. This could be a helpful way for you to start A/B testing. Indeed, A/B testing is the tool designed for the wise, analytical thinkers. With A/B testing around, you are assured that your site would generate more profits.

If you want to find out more about A/B testing and its complications, check out Josh’s entire webinar here. If you found value on this article, would you please share it? Thanks!

Originally published September 05, 2014 - Updated April 15, 2019
Dennis van der Heijden
Co-founder and CEO of Convert.com, Dennis is a passionate community builder and out of the box thinker. He spends his time innovating to make Convert Experiences better. Learn about his journey as an entrepreneur and leader on the SaaS Club podcast.
Guest Post Form

We have brought thought leaders, influencers, visionaries and veterans to our tribe. Now it’s your turn. If you have something worthwhile to share with a large community of savvy testers, go ahead and pitch your post idea. We’re listening.



[hclightbox id='5' text='Anchor text']