Why an A/B Test Winner May Damage Your Profits (and how to avoid it)

Why an A/B Test Winner May Damage Your Profits (and how to avoid it)

Pawel Ogonowski
June 30, 2015

Not long ago I attended a Digital Analytics Association Symposium in San Francisco where Krista Seiden, Analytics Advocate from Google, delivered a talk on Building a Culture of Optimization. She told a story about how she once rushed to her boss’s office to declare a winner from an A/B test – and you would probably expect me to be surprised when she then revealed that it ultimately turned out to be a total flop due to inaccurate data. But honestly, I wasn’t.

I have seen so many conversion rate optimization efforts on ecommerce websites, which fail to deliver good results due to data inaccuracies, that it has stopped shocking me. Do you want to be sure that you are not going to be tricked by the next A/B test results that you analyze? Then I have got a bunch of tips for you.

Do the Homework — Ensure Data Quality

ab testing

Are you 100% sure that your digital analytics tools are collecting accurate enough data for you to act on? Do you completely believe in the figures that your A/B testing software is showing you? Well, I don’t. Unless I validate them.

At Mavenec we always follow 3 steps when we are building an A/B testing ecosystem:

1. Audit of data in digital analytics tools.

Google Analytics is by far the most popular digital analytics platform. According to our internal statistics, in 8 out of 10 Google Analytics implementations that we audit, we find at least one data collection error which can make the data unreliable. Check your implementation thoroughly before you analyze the data (and of course integrate your A/B testing tool with Google Analytics!)

2. Compare the transaction data in the digital analytics tool with your financial records.

We all know that digital analytics tools are never 100% percent accurate and will never, ever show you complete data. But does that make them useless? Not really. If these tools collect around 80-90% of the data with consistent accuracy, then we can still act on it.

Typically, anything between 10-20% of transaction data gets lost due to users opting out of tracking, javascript errors and so on. I was horrified when (for the first time) I had a client whose Google Analytics reported only 60% of the transactions. How could anyone act on such data?

If you want to avoid this kind of mistake (and you certainly do) compare the figures between your financial records and Google Analytics (or Webtrends, AT Internet, Adobe Analytics or any other analytics tool that you use) in order to be sure that you are acting on reliable figures.

[Tweet “Are you 100% sure that your digital analytics tools are collecting accurate enough data for you?”]

3. Pass the data from the test to your accounting / CRM system.

I just love using Google Analytics data to analyze segments after the test is conducted. Sadly, hardly ever am I able to find there all of the data that I need to evaluate a test.

When analyzing A/B tests on e-commerce websites I would usually like to know some additional metrics that are missing – with the average margin on every challenger as a great example. Another is the return rate, which may vary between challengers. Typically, you won’t find those in digital analytics tools.

An easy fix to this problem is to pass the information to your accounting / CRM system about the version of the website that the user has seen alongside the transaction.

Following these 3 steps will ensure that you have the right data in place and that you can rely on it. But the right data is only half of the success. What else could go wrong you may ask? Well, you may have the right data in place, but your analysis could mislead you if you rely too much on one metric of success.

Do not Focus Solely on the Conversion Rate

Way too many A/B tests on e-commerce websites focus only on the conversion rate. The conversion rate is just a part of the story. By generating more transactions, it’s possible you could still generate lower revenue, if for example the average order value goes down. So, what other metrics should you take into account when assessing your next A/B test?

Ab testing

1. Average order value

The first metric – aside from the conversion rate – that you should take a look at is the average order value. If you manage to lift the conversion rate but impede the average order value you may end up with more work to be done in order to ship the orders but with the same level of revenue.

Use the Two Sample T-test from this calculator in order to determine whether the average order value is significantly different between two competing versions.

2. Profit Margin

Not all of the products an online merchant sells are equal and certainly, not every one of them produces the same money. Never, ever should you forget about the profit margin.

Checking this metric is extremely useful if you conduct tests on free delivery. There have been numerous examples where slashing delivery costs to zero skyrocketed the conversion rate and boosted the average order value. But this strategy also affects the profit margin.

Analyze this additional metric cautiously, to be sure that this strategy earns its keep (hopefully if you have followed one of the steps from the previous list and integrated your test data with your financial systems you can calculate it without much more hassle).

3. Return rate

Pushing users to complete transactions with psychological tricks may sometimes backfire. These users may not be satisfied with their purchases and you will then have to cope with a jump in the number of returned orders. This will create additional costs and hamper your profits. Always check if the return rate hasn’t gone up.

[Tweet “Compare your financial & your #analytics records to be sure you’re acting on reliable figures.”]

Before determining your next big winner in an A/B test on an e-commerce website, take note of the two steps mentioned above. Firstly, make sure that you are collecting the right data that you can rely on. Secondly, pair the conversion rate with additional metrics. Forgetting about these steps may lead you into proclaiming a winner, which boosts your conversion rate and sales, but leaves you with less money in your pocket.

Do you want to come up with great testing hypotheses for your next A/B tests? I bet you do! If I bet right, you should get our Ultimate 115-point Ecommerce Optimization Checklist which we use with our clients to dramatically increase their conversion rates and sales.

Originally published June 30, 2015 - Updated September 06, 2019
Pawel Ogonowski
The Ecommerce Optimization Guy who helps 7 & 8-figure online retailers get guaranteed revenue uplifts in an IT-free way. To get more of Pawel's content subscribe to Growcode's Ecommerce Optimization Blog and Bite-size Ecommerce Optimization on YouTube. Connect with Pawel on LinkedIn, Twitter and Instagram.
Guest Post Form

We have brought thought leaders, influencers, visionaries and veterans to our tribe. Now it’s your turn. If you have something worthwhile to share with a large community of savvy testers, go ahead and pitch your post idea. We’re listening.



[hclightbox id='5' text='Anchor text']