Time your A/B tests to get best results

Time your A/B tests to get best results

Akamai_Technologies

Every time you run a test, think about things you can do to reduce noise- or uncontrollable effects on your data. One thing that is often overlooked is fluctuations caused by the day of the week. Think about it. Many sites enjoy significantly more traffic and more sales on weekdays, while others perform best on the weekends.

<a title=Web site visitors tend to be very different on the weekends and evening, than mid day during the work week. I have seen a lot of activity and responses come from senior management of very large companies. Rarely during the day. Weekends are when these time-stressed executives get to surf.

Say you run a test starting on Monday. By Thursday, you could have a substantial amount of data, pointing to a clear “winner.” The inexperienced tester would begin celebrating prematurely. Along comes the weekend, and what you thought was the winner could end up falling behind. How to avoid this? Always run your tests by the calendar week (yes you can schedule tests in Convert’s A/B testing software). This essentially removes the weekday/weekend fluctuations from affecting your end results.

Originally published August 12, 2014 - Updated April 15, 2019
Dennis van der Heijden
Co-founder and CEO of Convert.com passionate in building communities that care. Trying to make that happen inside and outside Convert. I love working with my team to make our A/B testing software better for agencies and e-commerce clients.
Guest Post Form

We have brought thought leaders, influencers, visionaries and veterans to our tribe. Now it’s your turn. If you have something worthwhile to share with a large community of savvy testers, go ahead and pitch your post idea. We’re listening.

Fill out our guest post form

CATEGORIES Blog

COMMENTS
Avatar
Dennis, True, but I would argue that if the winner on the weekdays turns out to be the looser on the weekend, it is a good case for segmentation. Plus whatever your test changed will likely tell you a lot about your weekday visitors vs your weekend visitors.
    Avatar
    Hi Jared, indeed it helps to take a look after the experiment what worked mobile/non-mobile, country, weekend/weekday and source. Most interesting details will be in those segments (for most) Dennis

Leave a Reply

Your email address will not be published. Required fields are marked *

[hclightbox id='5' text='Anchor text']