Wednesday, 27 August 2014

Netflix a/b testing

Last week I came across an item on the Netflix Tech Blog that showed a slide and presentation of their plans for a/b testing their new web interface running on Node.js

It was all very worthy stuff and I wasn't entirely sure either what they were doing or trying to achieve by this testing; the body of the content spent most of the time waxing lyrical about not having to touch they're underlying system platform. I guess this is a big deal for Netflix! Anyway as an avid Netflix user I believe I have witnessed their testing strategy firsthand but in a rather perplexing fashion. Below is a screengrab of what I'm talking about. Under documentaries recommendations I frequently see at least two listings of 'The Long Way Down'. These are for the same programm/episode etc but one is shown in some weird Instagram effect the second in a monochrome  colour with a variation in image.

Now this is either a mistake or an entirely new way of a/b testing. I've never seen test variations (if that's what they are) presented side by side before. If this is a legitimate test is it not a rather crude means of promoting an episode to the end user? What is the end goal to see whether fans of Instagram filters opt for one creative over another? Is the 'Long Way Down' just such unmissable viewing entertainment that it warrants mentioning A LOT! Answers on a postcard. Happy testing : )

Thursday, 21 August 2014

What is MVT testing?

In it's simplest form MVT or multivariate testing is where you test alternative experiences of an existing webpage against the current page design or user journey. Unlike AB or Split testing where you test one webpage design against another, in MVT you test a combination of page elements like alternative page copy, headings, images, buttons and so on all at once to see which specific combination of alternative designs work best in driving page visitors or customers towards an end goal of your choosing.

All testing, regardless of whether its an AB or MVT test requires two things. Time and Traffic.  By that I mean you need to give a test enough time to establish what's called statistical significance, to basically reach a stable conclusion, and you will also require a volume of website traffic to churn through your test experiences. Typically you need less traffic to run through an AB test which normally only has a couple of alternative page designs to get through. In an MVT test you will typically have a higher number of test combinations to get through and therefore will require enough traffic to 'feed' your test.

In all test situations you continue to send a proportion of traffic to your existing page or default page to benchmark your alternative experiences against it. This is where you are calculationg any uplift in conversion, be that click through rate or the number of people getting to and passing through your check out process.

Testing outcomes can be positive or negative. A positive outcome might be that you test a red check out button versus a blue check out button and the red button delivers 8% more purchases. A negative outcome might be that the same button delivers a -5% drop in purchases. However, either way you have a valuable learning that tells you what changes to make to your site or inform further testing.

Monday, 18 August 2014

The Gutenberg rule - revisited

In a previous post about the Gutenberg rule I mentioned that we'd achieved some good AB and MVT testing results using this design principle. The principle basically works off the theory that humans subconsciously scan a print or web page from top left to bottom right and then loop back up the page. So allowing this principle to manage your page layout seems to have become the norm in web design placing buttons and other key calls to action in the 'fertile' areas, typically bottom right.

Increasingly though I notice that more and more people are breaking from this practice whether through test learnings or just asthetic decisions. An example being .As you can see below their landing page for car insurance aligns much of it's content to the left of the page. Having become comfortable with placing the onward journey point at the bottom right this just jars. Having said that it's important to challenge the norms to see what resonates with the customer. I may have to test this layout myself to check we're not missing a trick.
In the meantime here's a mock-up of an alternative design I would test based on established findings.

Happy testing : )

Friday, 15 August 2014

Page fold - browser size tool revisited

Problem: Recently Google have moved their Browser Size Tool into Google Analytics. For me and many other sites this means it simply no longer works as the 'In-page' reporting showing heatmaps and so on doesn't work or execute with our lovely CMS.Boo!
Solution: There's a free alternative. Yay! Called

This is what it looks like on this blog.

Thursday, 14 August 2014

a/b testing tools comparison by popularity

As part of my role I have to routinely compare A/B testing and MVT testing tools, suppliers and solutions. Many different sites list these suppliers but I've tried to use a few public metrics to determine how popular these tools are with the general public. I call it 'Reach ranking' in that suppliers are ordered 1). by their rankings, 2). then their average monthly searches for their test platform as determined by Google Trends and finally 3). how many clients they list (if at all) on their website.

It's also interesting to see so many new companies and start-ups there are in the CRO business since I last did this audit back in 2010 and also how many providers are no longer around. It's a fierce and competitive business....
So here's what I've found as at August 2014. Sources:, Google Adwords and Own Vendor sites
updated 13th Oct 2014
Reach ranking Vendor Alexa volumes YTD Avg. monthly searches for product No. of publically listed clients
1 SiteSpect 381,175  98,600 55
2 adlucent 324,719  27,000 0
3 Google Experiments 10,806,000 802,000 0
4 Conversion multiplier 3,987,218 73,600 0
5 Global Maxer 3,210,851 51,000 0
6 Conductrics 2,333,087 14,700 0
7 Qubit 1,506,519 25,880 131
8 Avenso 1,020,946 1,030 32
9 Clickthroo 974,000 47,819 0
10 Adobe Test and Target 945,671 78,000 0
11 Visual Website Optimizer 908,000 13,102 0
12 Accenture 860,000 13,240 0
13 Webtrends 763,000 19,600 0
14 Hi Conversion 722,004 47,000 44
15 Get Smart Content 673,173 3,740 14
16 Convert 672,000 126,000 0
17 Taplytics 283,016 12,789 6
18 Optimizely 239,000 58,898 25
19 Maxymiser 149,697 6,980 78
20 Site Tuners  121,678 13,000 72
21 Autonomy 120,271 26,000 0
22 AB Tasty 93,689 27,000 31
23 Monetate 36,501 21,230 39
24 Unbounce 22,230 12,230 0
25 Hubspot 2,700 541 0
26 Genetify 0 6,830 0
27 Vanity 0 6,400 0

Happy testing : )