Tuesday, 10 July 2012

Unequal visits in Google Experiments

Having run my new Google Experiment for a couple of weeks I have noticed that the distribution of traffic between the default and test page has become hugely unequal.
After a bit of trawling around the web I discovered that this was most likely caused by Google applying a Multi-armed bandit test approach (here)

Google states "a multi-armed bandit is a type of experiment where:
the goal is to find the best, or most profitable action, and
you learn about payoff probabilities as the experiment progresses."

"Once per day, we take a fresh look at your experiment to see how each of the variations has performed, and we adjust the fraction of traffic that each variation will receive going forward. A variation that appears to be doing well gets more traffic, and a variation that is clearly under performing get less."

So essentially Google are no longer retaining an equal traffic share for each variation in the test. If a variation is perceived to be doing worse than another or the default it gets less number of visitors. Equally if a test variant is doing better than another variant or the default it gets a greater share of the traffic. This is fine and will be rewarding for people who want a quick win during experiementation time but it's harder to swallow for those that have been doing conventional A/B or MVT testing up until now. In the past Google have let you adjust test weighting manually and it would be good to offer an overide option whereby the Multi Arm Bandit approach can be disabled by those who want clarity of their tests.