CRO statistical significance and beyond

How do you know when your AB test, split test, multivariate test or any other conversion rate optimization task is complete? When do you know you've found a winning experience that you can confidently apply to your web experience and expect the conversions to come rolling in?

Well the key thing is statistical significance of course, where essentially the volume of visitors to your test have voted with their feet, you've separated signal from noise and so on...

BUT, and it's a big but, have you allowed your test to play out through key trading times?

In my current company  we have a number of trading factors that mirror many other eCommerce businesses:

1. Day of week

2. Visits by hour

Once a CRO test has lived through several of these, plus a number of other factors specific to our industry (seasonality)  we can safely say the new experience has done it's tour of duty and is fit to implement.

Reaching a statistically significant result in itself is therefore not enough. Conversion optimization experts must ensure that their test audience takes into account critical factors that impact their tests.  This is important because one variation may appeal to one season or segment more than others and ultimately misguide the result.

New company, new job, new kit

I haven't posted for a while and the reason is I've moved jobs in the last few months, moving from finance and banking to the wonderful world or travel and tourism. I am still in the website optimization game but now I get to play with a whole new set of challenges, tests and testing kit.

So, to kick off I've recently procured Visual Website Optimizer for the new company as it's testing platform of choice. I've already got a couple of optimization exercises set-up and I'm already very impressed with this nifty offering.

I'm not going to do a direct comparison between other kit and testing services I've used before. A full listing of testing tools sorted by popularity is available here.

What's impressive about VWO:

• It's affordable, the Enterprise solution doesn't run into 4 or 5 figures per month like some solutions out there. You're not tied into a contract and can flex your usage  upwards & downwards. Flexibility is the key word here.
• At enterprise level you get a plethora of optimisation toolkit to play with; AB testing, Split testing, MVT testing, Heat maps, Click maps, Personalization, Website Reviews, Onsite feedback, AB testing of native iOS apps and more. This is not a stingy offering :)

• Test progress reporting is simple, straight-forward and intuitive. I'd forgotten what it was like to get excited about running a test and wanting to constantly check progress. The summary report acts as a heartbeat for any test you have out there allowing you to dig down for more detail if you need it with the Detailed Reports interface but not overwhelming you with countless filters and segments that end up raising more questions than answers.

I still need to really get under the skin of this tool but so far the price, functionality and customer service has really impressed. I will do a follow-up on this post looking at the personalisation offering in VWO as that looks pretty interesting. Happy testing.

Optimizely Experience London event 9th Oct 2014

I attended this very popular event today where we were given several case studies from various clients including The Student Room*, and Haymarket.

Optimizely looks a very capable testing tool with a leaning towards a self managed service or buying Optimizely Partner consultancy.

Key takeaways for me today were:

1. Optimizing for specific audiences. Split out and target sources of business and then focus on improving them individually, i.e spend 20% effort to gain an 80% return.

2. Use mobile wrappers to do proof of concept testing before spending big on a native app development. 

3. The Optimizely native app testing tool is amazing and I want it! An engaging demo of changing the Flappy Bird App on the fly. This was the big one for me today. Effortless AB and MVT testing of an iOS native app. To me this tool is what sets Optimizely apart from the pack. A very trivial demo using a popular game to show the power to change a user experience within an app with just a few clicks? This wasn't creating a test before the app was compiled, this was after the app was live.

In summary a good day to meet others operating with the same issues as me in Conversion Rate Optimization. Lots of Hippos in organisations still making subjective decisions about web design. An expectation of a successful test everytime by the business.

* Love this. The Student Room drag unwitting students off the streets of Brighton and ask them their opinion on their website and then give them a tenner for their troubles. This kind of 'street' qualitative testing I like.