Netflix a/b testing

Last week I came across an item on the Netflix Tech Blog that showed a slide and presentation of their plans for a/b testing their new web interface running on Node.js

It was all very worthy stuff and I wasn't entirely sure either what they were doing or trying to achieve by this testing; the body of the content spent most of the time waxing lyrical about not having to touch they're underlying system platform. I guess this is a big deal for Netflix! Anyway as an avid Netflix user I believe I have witnessed their testing strategy firsthand but in a rather perplexing fashion. Below is a screengrab of what I'm talking about. Under documentaries recommendations I frequently see at least two listings of 'The Long Way Down'. These are for the same programm/episode etc but one is shown in some weird Instagram effect the second in a monochrome  colour with a variation in image.

Now this is either a mistake or an entirely new way of a/b testing. I've never seen test variations (if that's what they are) presented side by side before. If this is a legitimate test is it not a rather crude means of promoting an episode to the end user? What is the end goal to see whether fans of Instagram filters opt for one creative over another? Is the 'Long Way Down' just such unmissable viewing entertainment that it warrants mentioning A LOT! Answers on a postcard. Happy testing : )

What is MVT testing?

In it's simplest form MVT or multivariate testing is where you test alternative experiences of an existing webpage against the current page design or user journey. Unlike AB or Split testing where you test one webpage design against another, in MVT you test a combination of page elements like alternative page copy, headings, images, buttons and so on all at once to see which specific combination of alternative designs work best in driving page visitors or customers towards an end goal of your choosing.

All testing, regardless of whether its an AB or MVT test requires two things. Time and Traffic.  By that I mean you need to give a test enough time to establish what's called statistical significance, to basically reach a stable conclusion, and you will also require a volume of website traffic to churn through your test experiences. Typically you need less traffic to run through an AB test which normally only has a couple of alternative page designs to get through. In an MVT test you will typically have a higher number of test combinations to get through and therefore will require enough traffic to 'feed' your test.

In all test situations you continue to send a proportion of traffic to your existing page or default page to benchmark your alternative experiences against it. This is where you are calculationg any uplift in conversion, be that click through rate or the number of people getting to and passing through your check out process.

Testing outcomes can be positive or negative. A positive outcome might be that you test a red check out button versus a blue check out button and the red button delivers 8% more purchases. A negative outcome might be that the same button delivers a -5% drop in purchases. However, either way you have a valuable learning that tells you what changes to make to your site or inform further testing.

The Gutenberg rule - revisited

In a previous post about the Gutenberg rule I mentioned that we'd achieved some good AB and MVT testing results using this design principle. The principle basically works off the theory that humans subconsciously scan a print or web page from top left to bottom right and then loop back up the page. So allowing this principle to manage your page layout seems to have become the norm in web design placing buttons and other key calls to action in the 'fertile' areas, typically bottom right.

Increasingly though I notice that more and more people are breaking from this practice whether through test learnings or just asthetic decisions. An example being Aviva.co.uk .As you can see below their landing page for car insurance aligns much of it's content to the left of the page. Having become comfortable with placing the onward journey point at the bottom right this just jars. Having said that it's important to challenge the norms to see what resonates with the customer. I may have to test this layout myself to check we're not missing a trick.

In the meantime here's a mock-up of an alternative design I would test based on established findings.

Happy testing : )

Page fold - browser size tool revisited

Problem: Recently Google have moved their Browser Size Tool into Google Analytics. For me and many other sites this means it simply no longer works as the 'In-page' reporting showing heatmaps and so on doesn't work or execute with our lovely CMS.Boo!
Solution: There's a free alternative. Yay! Called whereisthefold.com.

This is what it looks like on this blog.

a/b testing tools comparison by popularity

As part of my role I have to routinely compare A/B testing and MVT testing tools, suppliers and solutions. Many different sites list these suppliers but I've tried to use a few public metrics to determine how popular these tools are with the general public. I call it 'Reach ranking' in that suppliers are ordered 1). by their Alexa.com rankings, 2). then their average monthly searches for their test platform as determined by Google Trends and finally 3). how many clients they list (if at all) on their website.

It's also interesting to see so many new companies and start-ups there are in the CRO business since I last did this audit back in 2010 and also how many providers are no longer around. It's a fierce and competitive business....
So here's what I've found as at August 2014. Sources: Alexa.com, Google Adwords and Own Vendor sites
updated 13th Oct 2014

Reach ranking	Vendor	Alexa volumes YTD	Avg. monthly searches for product	No. of publically listed clients
1	SiteSpect	381,175	98,600	55
2	adlucent	324,719	27,000	0
3	Google Experiments	10,806,000	802,000	0
4	Conversion multiplier	3,987,218	73,600	0
5	Global Maxer	3,210,851	51,000	0
6	Conductrics	2,333,087	14,700	0
7	Qubit	1,506,519	25,880	131
8	Avenso	1,020,946	1,030	32
9	Clickthroo	974,000	47,819	0
10	Adobe Test and Target	945,671	78,000	0
11	Visual Website Optimizer	908,000	13,102	0
12	Accenture	860,000	13,240	0
13	Webtrends	763,000	19,600	0
14	Hi Conversion	722,004	47,000	44
15	Get Smart Content	673,173	3,740	14
16	Convert	672,000	126,000	0
17	Taplytics	283,016	12,789	6
18	Optimizely	239,000	58,898	25
19	Maxymiser	149,697	6,980	78
20	Site Tuners	121,678	13,000	72
21	Autonomy	120,271	26,000	0
22	AB Tasty	93,689	27,000	31
23	Monetate	36,501	21,230	39
24	Unbounce	22,230	12,230	0
25	Hubspot	2,700	541	0
26	Genetify	0	6,830	0
27	Vanity	0	6,400	0

Happy testing : )