CRO statistical significance and beyond

How do you know when your AB test, split test, multivariate test or any other conversion rate optimization task is complete? When do you know you've found a winning experience that you can confidently apply to your web experience and expect the conversions to come rolling in?

Well the key thing is statistical significance of course, where essentially the volume of visitors to your test have voted with their feet, you've separated signal from noise and so on...

BUT, and it's a big but, have you allowed your test to play out through key trading times?

In my current company  we have a number of trading factors that mirror many other eCommerce businesses:

1. Day of week

2. Visits by hour

Once a CRO test has lived through several of these, plus a number of other factors specific to our industry (seasonality)  we can safely say the new experience has done it's tour of duty and is fit to implement.

Reaching a statistically significant result in itself is therefore not enough. Conversion optimization experts must ensure that their test audience takes into account critical factors that impact their tests.  This is important because one variation may appeal to one season or segment more than others and ultimately misguide the result.

Discount code testing on checkout

Been running ANOTHER personalisation test using VWO again on the checkout. This time targeting people who qualify for a discount code and presenting that code either just ahead of the promo code box or actually pre-loading the promo code box with an applicable discount code.

Result? 27% conversion when you put the promo code pre-loaded into the box versus 5.26% conversion when no code provided. 

Remove header & footer from checkout

I'm going for a different post style in a bid to post more frequently. I'm going to tell you what I'm testing and the test result. That's it....

We have an online checkout (basket etc). We tested removing the header with the primary navigation and the footer with its various site links. Why? Because these areas represent leakage points for customers and can also be distracting. Removing these escape points helps focus the user on the task at hand, ie, checking out!

Result = 22.7% conversion versus 10.29% conversion rate.

Web session recording comes of age

I think I first looked at website session recording or session capture as it's also known  back in 2007 when a company called Speed Trap were in the business of web analytics. It was really a side offering to their main suite of analysis tools that didn't look dissimilar to Google Anayltics does today. Back then it didn't work. It created movie files of a very narrow subset of user sessions that you could, in theory replay at your leisure to highlight user struggle and experience. Looking back it was too far ahead of it's time. Some 8 years later, and having seen a few session recording tool contenders* I've just witnessed what we'd always looked for in such a tool. Visual Website Optimizer (VWO) from Wingify now offers a Session Recorder as part of it's already very agile testing toolkit.

*For balance let me just list the other tools I have used for session recording ;  Tealeaf, Click tale and SessionCam.

With VWO, basically you tell it what page to watch and within minutes (not hours and days) you have a selection of user sessions to replay based on multiple devices and platforms. I was immediately hooked. No jerkiness, no blank screens, no style sheet dropping just a slick, smooth replay of the users every mouse move and click. I've already waxed lyrical about VWOs heat map tool, which integrates nicely with this tool. It's fairly addictive and I found myself replaying 24 sessions in one sitting!!

The payoff?

Having watched those sessions on a checkout page it became obvious we immediately had an issue. Every single visitor without exception was missing a credit card type field and missing the validation messages associated with this. So I fired up VWOs AB testing tool and created an alternative version of the checkout page with this card field in a more obvious place. Immediately we achieved a xx% improvement in sales performance.

Below are screenshots of the session replay in action. The red lines are the mouse moves. The blue bubbles are the left clicks.

Heat maps with Visual Website Optimizer

I love heat maps I really do! These are tools that map where people click and sometimes scroll on a web page, traditionally reflecting a higher heat intensity by colour etc. Trouble is, in the past the tools I've had access to have fallen short of user expectation. Either they simply haven't worked on our content or have been hugely misleading in terms of what they have captured and finally the time lag in gathering data and reporting back.

But no more. When we procured Visual Website Optimizer (VWO) from Wingify as a testing platform back in March of this year, not only did we get a very competent testing platform  it also had a whole set of very useful kit including heatmaps.

So running a split test or MVT test will tell you which experience is best for achieving your online goals but it might not tell you exactly how people are interacting with your test experience. Below is an example of a recent test I've been running on our basket checkout. This personalization test added promo code copy, wrapped around the promo code field entry for people looking for specific holiday dates. As you can see the promo code box and call to action is heavily used at this point in the journey but it also shows people scanning and interacting with the appropriate copy. Conversely heatmaps can obviously show you when people are not interacting with your beloved copy.

Using heat maps in conjunction with testing and conventional analytics really gives you a three dimensional view of exactly what behavior you may or may not be driving.

Heatmap

Heatmap showing fertile and fallow areas of our product pages. People tend to click on only one image within an image set. In this case the 360 degree call to action is an outright winner.