Running A/B tests that users can see are not helpful as we already proven in 2012 when we detected an 18% drop in conversion rate when users were able to see the changes happen right in front of their eyes. Today I noticed that Qualaroo was testing, since I noticed a strange blink on the customer logo section. When I did a hard-refresh I noticed Intuit and Groupon were replaced with Uber and Shopify logos. We know some A/B testing software tools make changes slower but it’s important to point out that this should be part of your Q&A.
See here the video on what happens at the website and why you should be doing good quality checks before putting the A/B tests live. In short use great A/B testing software that does not have this problem.