Back to the Basics: Freeze Your Design While Your Testing

Back to the Basics: Freeze Your Design While Your Testing

Always be up to Date subscribe to updates

Subscribe to our mailing list

* indicates required First Name Original Signup Source Email Address * Whitepaper - 99 Tip Blog Weekly Webinar Whitepaper - 18 Kick-Ass Visual Analysis Tools Whitepaper - Top 10 E-Commerce Abandonment Recovery Tools Persona eCommerce Marketing Managers Employee of Online Marketing Agency Undefined
- June 12, 2014

Every once in a while, I realize that there are some basic tenets of testing that are worth revisiting. One I am going to explore today is the idea of making changes to the pages you are testing, while the test is running. With only a couple tiny exceptions, this is widely considered a huge NO NO.

It all goes back to where AB Testing on the web came from, something most of us learned in elementary school science class called “the experimental method.” The experimental method involves manipulating one variable to determine if changes in one variable cause changes in another variable. This method relies on controlled methods, random assignment and the manipulation of variables to test a hypothesis.

The most important part of this definition is “manipulating one variable.” Using a carefully designed experiment that changes only one variable, any change in the outcome (other variable) can be attributed to the single change.

What happens if you change more than one thing at once? Confusion. What we testing geeks call confounding of the results. One can no longer reliably link a given change to a given result. (Note- this is not the case with Multivariate experiments.) So back to my original point. When a split test is running, it is essential that all possible variables be held constant (frozen), until enough data has been collected to reach a point of statistical certainty.

And what happens when you make changes to the pages that are already running in a test? Its worse than changing multiple things at once. Your results will be utterly useless, unless you have hit statistical validity before making the change. In this case, you are basically completing one experiment and launching a second.

  • 12 Jun, 2014
  • Posted by Dennis van der Heijden
  • 1 Tags
  • 0 Comments

Written by Dennis van der Heijden

Co-founder and CEO of Convert.com passionate in building communities that care. Trying to make that happen inside and outside Convert. I love working with my team to make our A/B testing software better for agencies and e-commerce clients.

CATEGORIES Articles Blogs

Anchor text

Subscribe to our mailing list

* indicates required First Name Original Signup Source Email Address * Whitepaper - 99 Tip Blog Weekly Webinar Whitepaper - 18 Kick-Ass Visual Analysis Tools Whitepaper - Top 10 E-Commerce Abandonment Recovery Tools Persona eCommerce Marketing Managers Employee of Online Marketing Agency Undefined

Subscribe to our mailing list

* indicates required First Name Original Signup Source Email Address * Whitepaper - 99 Tip Blog Weekly Webinar Whitepaper - 18 Kick-Ass Visual Analysis Tools Whitepaper - Top 10 E-Commerce Abandonment Recovery Tools Persona eCommerce Marketing Managers Employee of Online Marketing Agency Undefined