1st Direct Bank throw it out there

First Direct have recently launched First Direct Labs. This is a new section of their site which shows you what new ideas and concepts they are testing for their on-line experience. Current tests being; QR Code functionality, redesign concepts, and mobile apps.Visitors are encouraged to rate these concepts and designs as well as make suggestions of their own for new tests. The missing link here is that this feedback is obviously a channel for qualitative testing of these ideas and for my mind seems to be their only current means of gauging how effective these new ideas will be. Obviously I'm not privy to their whole web testing strategy but I would hope there is more in their testing toolbox than just this focus group approach. Either way First Direct are to be commended for divulging part if not all of their testing strategy, it's a safe thing to do as experience has shown that competitors can rarely benefit from implementing test finding vicariously without first doing their own comprehensive testing. I'll address vicarious testing in greater detail in a later post.

Continuing with my ongoing retrospective theme it's worth pointing out that I haven't been averse to going public with test ideas and designs in the past as seen in this post from 2009 where I ask the general public to rate our page designs following a non-conclusive round of MVT testing.

Testing During a Traffic Spike revisited

In a post I wrote back in 2009 I talked of the benefits of combining MVT testing with campaign activity in a post called Riding the tsunami. Coming late to the party, but nonetheless getting there in the end are Get Elastic ,a very good web optimisation site with some valuable testing ideas and concepts, endorsing this very same approach after a bit of soul searching with an article titled "Should You Avoid Testing During a Traffic Spike?" Definitely worth a read. 


I think fundamentally the message remains the same, it's okay to be MVT testing during a campaign if you're trying to optimize that campaign and not the long-term web experience of your visitor.  As ever testing results are usually as a result of the following variables:


 product benefit + moment in time + market position + customer experience

The law of diminishing returns

Once you've optimised a page using either multi variant testing (MVT) and/or split testing (A/B Testing) and managed to achieve a respectable uplift in sales conversion, when it comes to revisiting that page with further testing, likely as not you're entering the realm of Diminishing Returns.

This is historically an economics term but it also applies to web optimisation testing. This is where subsequent testing or optimisation activities prove to be less rewarding in terms of finding web content that works than the original or earlier rounds of testing.

This has been recently illustrated last week when a colleague produced a new version of an optimised landing page. He wisely wanted to test that it could perform as well as the existing page or even better. The original page was the result of several rounds of previous optimisation testing and was already proving to be very good at converting visitors to submit an on-line application. The image below sees the ongoing split test as conducted in Google Website Optimizer . The original or default page is proving hard to beat, the new page (variant 1) is bettering the original but it's not pulling away in a massive uplift as you might see in the first or second rounds of testing.

It's important to realise that while you should always look to be frequently testing your pages, previously tested or otherwise, as part of a continued programme of testing, we should realise that the big headline results of earlier testing should start to decline test on test . This is a sign of successful testing, indicative that you're starting to get things right from the visitor conversion perspective.

As a very rough guide I would say the following is true for a successful roadmap of testing; Let's call it the ARSSS  approach (sorry I'm such a child!):

1. Analyse your site metrics, establish user journeys, understand what's going on.
2. Rationalise your site. Remove unnecessary  pages and clicks. Remove obvious leakage points in your sales funnel.
3. Start MVT testing. Use this to get under the skin of the user experience. Do as many rounds of testing as it takes to answer your questions and hopefully start to improve your conversion. In essense you're starting to narrow and hone your sales funnel.
4. Start split testing. Once you know what works on a page element by element basis through MVT you can start to use A/B testing to start look & feel testing entire pages.
5. Segment Users. Once you've done all of the above start to get into User Segmentation, ie. start to group your customers into segments based on behaviour (I'll be writing a more in-depth post on this in the future).

Happy testing!

net.finance 2011

Last week I spoke on the merits of creating a testing culture at the financial services web conference Net.Finance in Chicago on behalf of Maxymiser. See their blog here for details Max Blog. It was a great trip and an extremely exhausting week due to my own crazy schedule but I absolutely loved every second of it.

Key take-aways from the conference? 

• Multivariate testing is yet to really take off in the states, especially in the financial sector. It's ripe for massive growth on an unprecedented scale.
• US marketeers are really excited about 'Mobile Payments' and see it as a potential bank killer.The question remains which big names will forge alliances to make it finally happen.
• The buzz word is 'mobile' and has been for a while, but everyone, including the main players are waiting to see who goes really big on mobile first and will then quickly follow suit. The US web sector are scared and have been stung before by 'novelties' that have failed to bear fruit, so there's a vast amount of tangible caution in the average US eCommerce department; dollars are quite rightly spent sparingly and wisely. 
• There are a lot of companies offering what is perceived as real 'added value' but is really not worth the huge initial $ outlay when you start to dig deep on their technical claims. Qualitative testing and research continues to commit multiple crimes in the name of informed user feedback and fall vastly short of continuous multivariate testing by a country mile. 

Would be interesting to see if anything changes in the US market in the next 12 months and especially to see if mobile becomes the hunting ground of the web marketeer as predicted.

Calculating Uplift

This is fairly basic but a common query none-the-less:

Q. How do you calculate conversion uplift?
A. Winning% - Old% / Old% x 100 = UPLIFT%

Simple!