Back to the Basics: Freeze Your Design While Your Testing

Every once in a while, I realize that there are some basic tenets of testing that are worth revisiting. One I am going to explore today is the idea of making changes to the pages you are testing, while the test is running. With only a couple tiny exceptions, this is widely considered a huge NO NO.

It all goes back to where AB Testing on the web came from, something most of us learned in elementary school science class called “the experimental method.” The experimental method involves manipulating one variable to determine if changes in one variable cause changes in another variable. This method relies on controlled methods, random assignment and the manipulation of variables to test a hypothesis.

The most important part of this definition is “manipulating one variable.” Using a carefully designed experiment that changes only one variable, any change in the outcome (other variable) can be attributed to the single change.

What happens if you change more than one thing at once? Confusion. What we testing geeks call confounding of the results. One can no longer reliably link a given change to a given result. (Note- this is not the case with Multivariate experiments.) So back to my original point. When a split test is running, it is essential that all possible variables be held constant (frozen), until enough data has been collected to reach a point of statistical certainty.

And what happens when you make changes to the pages that are already running in a test? Its worse than changing multiple things at once. Your results will be utterly useless, unless you have hit statistical validity before making the change. In this case, you are basically completing one experiment and launching a second.