CRO statistical significance and beyond

How do you know when your AB test, split test, multivariate test or any other conversion rate optimization task is complete? When do you know you've found a winning experience that you can confidently apply to your web experience and expect the conversions to come rolling in?

Well the key thing is statistical significance of course, where essentially the volume of visitors to your test have voted with their feet, you've separated signal from noise and so on...

BUT, and it's a big but, have you allowed your test to play out through key trading times?

In my current company  we have a number of trading factors that mirror many other eCommerce businesses:

1. Day of week

2. Visits by hour

Once a CRO test has lived through several of these, plus a number of other factors specific to our industry (seasonality)  we can safely say the new experience has done it's tour of duty and is fit to implement.

Reaching a statistically significant result in itself is therefore not enough. Conversion optimization experts must ensure that their test audience takes into account critical factors that impact their tests.  This is important because one variation may appeal to one season or segment more than others and ultimately misguide the result.

Striking while the iron is hot

Probably the holy grail of any web optimisation exercise is to be serving the most optimised page content at the most effective time possible. For us in banking this opportunity presents itself once a year in the form of the ISA Season*.

We were keen to improve the conversion of ISA landing pages in preparation for this busy sales time for the bank. This test was for an ISA savings product page and we were looking to test header copy, images and body copy combinations in order to improve conversion. The winning design would be implemented in time for the peak volumes during this period.

The Hypothesis for this test is as follows:

By Multivariate testing combinations of header copy, images and body copy on the ISA product page we expected to see a 10% improvement in number of visitors starting an application for a product and improve out visitor to submitted application ratio by 5%.



This test was run twice (Between 13/03/2009-06/04/2009 (Pre Tax Year) and between 06/01/2009-13/01/2009 (Post tax year)). We were forced to conclude the second test early due to the product being withdrawn but in both instances the winning design was the ‘Right hand side tabbed content’ although we did have alternative images winning in each test. In the final test we saw the lead design resulting in a 16.14% increase in conversion (17.13% in the first test) over the default design with a statistical significance level of 98.65%.



* End of the old tax year and the start of the new one and something many in the financial services industry call "ISA Season"

Seasonal shift

The established wisdom of web page optimization suggests that images, and particularly images of people help build a connection between the visitor and your content/product/brand, See earlier post about quick wins regarding this.

We have been testing this theory through ongoing multivariate (multivariable) testing on several product pages on the site. However we've inadvertently created a rod for our own backs.

The original test was started back in November and we gave it an autumnal theme. The test variant with these images originally performed well but a gradual errosion in performance occured as the test and time wore on. The test was still running as we entered the Christmas period and suddenly the autumn theme looked out of place so the drop in visitor appeal was understandable. To confirm this we revisited the variant and replaced the autumn images with winter based ones. Again an uplift in performance was immediatly acheived. The test is still running and currently producing a respectable 8.95% uplift in conversion; but it's still wintery outside with the occasional snow flurries to reinforce that, so maybe this explains the appeal. Now as time moves on it will be interesting to see if this begins to errode in performance too.

So we have a situation where we know a particular content works but it will require constant maintenance to keep on top of the shifting seasonal themes. We'll need to come up with a sustainable image or theme that will work as well despite the time of year. We still have an MVT test on the roadmap for March/April that plays on the Spring theme but hopefully this test will be it's last incarnation. All of the past and proposed images are below, we've nicknamed this variant 'Shallow Grave' due to the half-buried nature of the models in shot.
AUTUMN > WINTER > SPRING.....