Optimisation Maverick Blog

Bigger. Stronger, faster, better - best test ever

2012-02-17T12:26:00.001-08:00

I launched a Google Otimizer Ab test at the start of the week to see if I could drive an uplift in Mobile Banking registrations. Essentially the newly launched landing page for mobile banking had a CTA for the registration form but it was available on only 1 tab of a 4 tab page design. Using GWO I added the same CTA to the footer the page so that it was available across all tabs. Within just a few dayherhe new design was beating the old design hands down with a +200% uplift in people clicking through to the registration form with a 99% confidence index. I'll add an image of the design soon.

Maxymiser - Multiple KPIs

2011-10-29T11:44:00.000-07:00

A natty little feature of the Maxymiser reporting interface is the ability to report on multiple KPIs within the same report. Below is a screen grab of a conversion report which reports on both the 'Application submit rate' referred to as KPI 1, whilst also showing the performance of the secondary KPI 2 of 'Click to Apply rate' for the same test combinations. In the past you would have to flick between separate reports for each KPI, whereas this is obviously no longer the case.

To enable this feature you simply select multiple KPI or Actions in the report filter drop-down menu within the Maxymiser console. See illustration below.

Charlotte 29th Oct 2011

Google Analytics - Map Overlay

2011-10-22T03:39:00.000-07:00

Lessons learnt from Retail* #1

I know Google Analytics (GA) has had it's Map Overlay function for sometime now, a report in GA which shows geographically where your online traffic comes from. Recently though I've had a degree of exposure to the retail sector as opposed to just the financial sector. With this particular brand it was interesting to see how the online traffic so accurately matched up with the branch locations. See image below, on the left is GA map overlay report for the brand and on the right you see its store locations. An incredibly obvious relationship appears between online activity and store location illustrating in retail how branch locality drives online brand awareness.

This kind of thing just doesn't happen in the financial sector where online traffic bears no immediate correlation with the offline world and the companies branches.
Obviously though when there is a match there's a geotargeting opportunity if you chose to act upon that information. This is obviously taken to a higher level with augmented reality applications in the mobile world where there are great marketing opportunities when GPS technology meets the highstreet retail opportunity. I will cover this in more detail in a later post.

*I've been privileged to get access to the web analytics of a retail brand lately. In a brief series I will cover insights I've found between the retail world and the financial sector where I specialize as an online optimization expert

Tracking uplift post ab testing

2011-08-26T03:19:00.000-07:00

Once you've ran either an A/B test or a Multivariate (MVT) test on your website and you're told by your testing tool of choice that your test has achieved statistical conclusion how do you continue to measure performance, and more importantly, should you continue to monitor performance?

I know it sounds reckless, to continue to track uplift is the responsible thing to do right? But in reality there's an important aspect of optimisation testing that needs to be taken into consideration here. Testing outcome is usually the result of the following variables:

product benefit + moment in time + market position + customer experience

Can you continue to accurately measure and account for each of these factors after you've conducted a test? The honest answer is probably no. Also I don't know many people with the personal bandwidth to monitor every single test once it's finished. If I wanted to double my workload I certainly would!

The important thing to do is ensure is that your test has been given enough testing time and traffic volume in the first place before you conclude the test as this post reasonably states Don't Fool Yourself with A/B Testing. If you've done that you should have a reasonable level of confidence in it's future performance.

If you still want to track uplift after testing I would suggest the following are available options for you:

1. Set up a Google Analytics Goal. This gives you the ability to track the performance of a specific customer journey within your normal web analytic. Yes you have to use Google for this one, but any web metrics tool worth it's salt will have the same functionality.

2. Leave your test running. This to me is the fail-safe option. Once you have a test winner up-weight this in favour of the default content but leave a small percentage of your traffic going to the default as a benchmark for continued performance. I usually leave 5% going to the default for a period of time where possible to ensure I've made the right decision.

3. Run a Follow-up experiment. This is a great feature in Google Optimizer but you can do the same in any other testing tool if you have the resource to do it and there's lingering doubt about the original test outcome.

4. Bespoke tracking. On the pages I optimise I append tracking values to the application form that when submitted to a sales database can be used to tie back sales to a specific landing page. Using this approach you can monitor conversion rate performance before, during and after a test. I cant recommend this approach enough and is entirely dependent upon your on-line application forms particular design as to whether you can implement it.

That's about it really. If I think of any other methods for ongoing tracking I'll add them in.

Happy testing!

1st Direct Bank throw it out there

2011-08-16T06:35:00.000-07:00

First Direct have recently launched First Direct Labs. This is a new section of their site which shows you what new ideas and concepts they are testing for their on-line experience. Current tests being; QR Code functionality, redesign concepts, and mobile apps.Visitors are encouraged to rate these concepts and designs as well as make suggestions of their own for new tests. The missing link here is that this feedback is obviously a channel for qualitative testing of these ideas and for my mind seems to be their only current means of gauging how effective these new ideas will be. Obviously I'm not privy to their whole web testing strategy but I would hope there is more in their testing toolbox than just this focus group approach. Either way First Direct are to be commended for divulging part if not all of their testing strategy, it's a safe thing to do as experience has shown that competitors can rarely benefit from implementing test finding vicariously without first doing their own comprehensive testing. I'll address vicarious testing in greater detail in a later post.

Continuing with my ongoing retrospective theme it's worth pointing out that I haven't been averse to going public with test ideas and designs in the past as seen in this post from 2009 where I ask the general public to rate our page designs following a non-conclusive round of MVT testing.

Testing During a Traffic Spike revisited

2011-08-15T07:18:00.000-07:00

In a post I wrote back in 2009 I talked of the benefits of combining MVT testing with campaign activity in a post called Riding the tsunami. Coming late to the party, but nonetheless getting there in the end are Get Elastic ,a very good web optimisation site with some valuable testing ideas and concepts, endorsing this very same approach after a bit of soul searching with an article titled "Should You Avoid Testing During a Traffic Spike?" Definitely worth a read.

I think fundamentally the message remains the same, it's okay to be MVT testing during a campaign if you're trying to optimize that campaign and not the long-term web experience of your visitor. As ever testing results are usually as a result of the following variables:

product benefit + moment in time + market position + customer experience

The law of diminishing returns

2011-08-01T06:36:00.000-07:00

Once you've optimised a page using either multi variant testing (MVT) and/or split testing (A/B Testing) and managed to achieve a respectable uplift in sales conversion, when it comes to revisiting that page with further testing, likely as not you're entering the realm of Diminishing Returns.

This is historically an economics term but it also applies to web optimisation testing. This is where subsequent testing or optimisation activities prove to be less rewarding in terms of finding web content that works than the original or earlier rounds of testing.

This has been recently illustrated last week when a colleague produced a new version of an optimised landing page. He wisely wanted to test that it could perform as well as the existing page or even better. The original page was the result of several rounds of previous optimisation testing and was already proving to be very good at converting visitors to submit an on-line application. The image below sees the ongoing split test as conducted in Google Website Optimizer . The original or default page is proving hard to beat, the new page (variant 1) is bettering the original but it's not pulling away in a massive uplift as you might see in the first or second rounds of testing.

It's important to realise that while you should always look to be frequently testing your pages, previously tested or otherwise, as part of a continued programme of testing, we should realise that the big headline results of earlier testing should start to decline test on test . This is a sign of successful testing, indicative that you're starting to get things right from the visitor conversion perspective.

As a very rough guide I would say the following is true for a successful roadmap of testing; Let's call it the ARSSS approach (sorry I'm such a child!):

Analyse your site metrics, establish user journeys, understand what's going on.
Rationalise your site. Remove unnecessary pages and clicks. Remove obvious leakage points in your sales funnel.
Start MVT testing. Use this to get under the skin of the user experience. Do as many rounds of testing as it takes to answer your questions and hopefully start to improve your conversion. In essense you're starting to narrow and hone your sales funnel.
Start split testing. Once you know what works on a page element by element basis through MVT you can start to use A/B testing to start look & feel testing entire pages.
Segment Users. Once you've done all of the above start to get into User Segmentation, ie. start to group your customers into segments based on behaviour (I'll be writing a more in-depth post on this in the future).

Happy testing!

net.finance 2011

2011-05-31T14:07:00.000-07:00

Last week I spoke on the merits of creating a testing culture at the financial services web conference Net.Finance in Chicago on behalf of Maxymiser. See their blog here for details Max Blog. It was a great trip and an extremely exhausting week due to my own crazy schedule but I absolutely loved every second of it.

Key take-aways from the conference?

Multivariate testing is yet to really take off in the states, especially in the financial sector. It's ripe for massive growth on an unprecedented scale.
US marketeers are really excited about 'Mobile Payments' and see it as a potential bank killer.The question remains which big names will forge alliances to make it finally happen.
The buzz word is 'mobile' and has been for a while, but everyone, including the main players are waiting to see who goes really big on mobile first and will then quickly follow suit. The US web sector are scared and have been stung before by 'novelties' that have failed to bear fruit, so there's a vast amount of tangible caution in the average US eCommerce department; dollars are quite rightly spent sparingly and wisely.
There are a lot of companies offering what is perceived as real 'added value' but is really not worth the huge initial $ outlay when you start to dig deep on their technical claims. Qualitative testing and research continues to commit multiple crimes in the name of informed user feedback and fall vastly short of continuous multivariate testing by a country mile.

Would be interesting to see if anything changes in the US market in the next 12 months and especially to see if mobile becomes the hunting ground of the web marketeer as predicted.

Calculating Uplift

2011-05-25T05:13:00.000-07:00

This is fairly basic but a common query none-the-less:

Q. How do you calculate conversion uplift?
A. Winning% - Old% / Old% x 100 = UPLIFT%

Simple!

Above the fold - The Google Browser Size Tool

2011-04-21T03:12:00.000-07:00

Here's one I'd stumbled across a couple of years back now and had subsequently forgotten about but Lord knows why because it's so useful and so simple in it's conception. Google Labs has a Browser Size tool that let's you overlay a summary of browser size (based on visits to the Google homepage) over any web page. For a while people had spoken about there being no page fold when it came to web design. Well that's poppycock, if multivariate testing and UX testing has taught us anything it's that if you have content which people have to scroll down to a lot of people will either not bother or just not realize the content is there in the first place. In testing I've found that if you cant get away from a lengthy page you need to make the design imply that its worth scrolling to the content or try and bring everything back above the fold through tabbed design etc.

The conception of Short Wave Testing

2011-03-28T13:31:00.000-07:00

Right, well this is really work in progress. I think I've invented a new form of multivariate testing on the web. And for clarity this has nothing at all to do with Short Wave Radio. However, a couple of points to start off with; A) I'm not entirely sure it hasn't been done before and B) It's a valid test methodology.

Well hang it this blog is all about being a testing 'Maverick' so here goes nothing....

First off, let's not get confused by Iterative Wave Testing as used by Optimost. I think I'm right in saying that's where you test the same variants over a sustained period in 'waves' of testing to ensure what you have is validated and statistically significant. All very worthy, good stuff.

What I've been experimenting with is trying a set of test variants in one brief wave of testing and then ditching or culling any negative or lesser performing variants in favor of an entirely new variant in a new wave of testing that sees the positive or successful variants carried forward from the last wave. The whole process is repeated for as many waves as it takes to get a robust set of variants that out-perform everything else pitted against them. The only qualifying criteria for a variant to be carried forward to the next wave of testing is that they either continue to outperform the original default design or better the performance of anything that has gone before them, i.e; anything that has been previously removed.

I hope this simple (ish) diagram illustrates how this short wave testing works. Below we have 4 test areas in a web page and we have 4 phases of testing. As we can see in Test Area 1, Variant A is successful enough never to be culled from the test and ultimately becomes the winner for Test Area 1. Test Area 2 shows an initially unsuccessful Variant A that is culled after the first phase of testing and replaced with a new variant B which goes on to be the winning variant of Test Area 2. Test Area 3 has a different story, in the end it takes 4 different variants over 4 phases of testing to find a variant that is positive enough to be declared a winner. And Test Area 4 arrives at a winner on the third phase of testing with variant C.

Now I'm aware that this form of testing is both labour intensive and resource-heavy in it's undertaking. I was able to do this kind of testing because I was both motivated enough to dedicate resource to it and had enough ideas in the locker that I wanted to test for each test area and test wave. I used Google Optimizer to do it and coded the variants myself and the outcome has been, well staggering. A sustained uplift in the region of 18% for product purchase has been achieved (a personal best BTW) and to me I am reasonably confident in the results because the final variants I had, had reported consistently the same uplift over 9 separate waves of testing.
What I'm hoping for now is the counter-argument from my testing peers (drop me a line at farmerfudge@googlemail.com). I'm aware of the shortcomings of this approach but want others to have their say on this kind of testing methodology. Here's my bonfire, feel free to piddle all over it : ) Happy Testing!

UPDATE: One thing worth noting with this testing approach is that if it goes right your conversion rate for the test variants should improve for each wave where you attain, keep or build on positive performing variants but at the same time you will also see a diminishing uplift for each wave. This is because you are continually testing against improved and stronger performing variants in the test segment. Ultimately though you should still see a good uplift against the underlying original default design.

No need to shout about it

2011-03-25T02:25:00.000-07:00

I've been running an MVT test on a comparison page and recently introduced an 'Ends Soon' label next to the product call to action. Initially the presence of this message was negative. I resized the image by half making it much smaller and the conversion results are much improved, illustrating that sometimes people just don't want to be shouted at : )

Update: Although this 'hurry message' didn't work well with this particular page which was a product comparison page the same image used on an already optimized product page using Maxymiser has led to a 44% uplift in product application submit rate.

An Offline Call To Action

2011-02-10T06:40:00.000-08:00

A recent MVT test using Google Website Optimizer answered the question.

"Exactly what impact does having an off-line Call To Action next to an on-line have?"

In this test I would measure the impact upon the click to apply rate on a landing page where using MVT I would serve up a link to a pop-up window which would show both a telephone sales number and a branch locator to a section of the page visitors.

During the test period, in addition to monitoring the test console results I monitored the Google Analytics report for the pop-up window.

Here's the summary of results:

675 Visitors saw the default (no offline CTA)

248 of which click Apply = 36.7% Conversion rate

678 Visitors saw the offline CTA variant

205 of which click Apply = 30.2% Conversion rate

The offline CTA variant is down –17.7% in Conversion rate

against the default page

The offline CTA pop-up received 569 Unique Views in the

test period. Therefore 83.9% of people who see an offline

CTA will click it.

The Gutenberg Rule

2011-01-10T09:09:00.000-08:00

Recently a couple of people have reminded me about how we'd used this design principle during MVT testing and yielded some good results and insights as a result. So I thought I'd commit some learnings to a post on the subject .

The Gutenberg Rule is a design philosophy named after the designer of the printing press Johannes Gutenberg. This principle suggests that people read content top to bottom and left to right. You can therefore split a page into four quadrants, the “Primary Optical Area” in top-left, the “Strong Fallow Area” in top-right, the “Weak Fallow Area” in the bottom-left and a “Terminal Area” in bottom-right. Splitting a web page into four quadrants as illustrated below we tested the various positions of a product offer by rotating it through these 4 positions (in more than 1 test).

This testing confirmed that

position 1 yielded the highest uplift
position 2 the second highest
position 3 the third most profitable position
position 4 the least uplift

Additionally, just below position 2 proves to be the ideal location to place a Call To Action in numerous optimisation exercises

Horizontal Positioning

Extending on from this principle it's also worth noting that horizontal positioning is of equal significance, born out with the following test example. On a landing page we rotated three product benefits through a horizontal layout as follows and monitored the effects on click to apply rate. Swapping the 'Great rate' benefit to second position after a cash back offer yielded a 3.24% uplift.

Again, swapping the Overdraft benefit with the Cash back tile yielded an even greater uplift of 3.69%. I guess you could call this the "Gutenberg Horizontal Positioning rule".

So in conclusion, positioning of message and offers can be absolutely crucial to the success or failure of a web design based upon some highly established design principles.

Good old button testing

2010-10-28T07:45:00.000-07:00

Everyone, at some point, does some optimisation testing of buttons designs. Some people think it's a trivial exercise to undertake when there's bigger fish to fry. Well I disagree, button design testing is exactly the kind of thing you can be doing quickly and easily with Google Optimiser or similar. We've done loads of testing in the past on buttons, testing colours, sizes, Apply text and so on, but I read an interesting article from Get Elastic on how unusual button designs can give you an easy uplift in conversion. So I tried over the course of a couple of months on a landing page testing all the designs you see here. No.1 was the default design, and the winner was...No.3 the 'boxed arrow' design with a 32% uplift in click to apply rate. The arrow-based designs were in the upper end of the winning designs overall, but the, *cough* phallus-based designs stole an early lead but didn't win out overall. Give it a go on your website, it's quick easy and surprisingly fun.

Google Optimizer Follow-Up experiments

2010-10-21T06:56:00.000-07:00

First off - What is a follow-up experiment?

Google says:

"

When the results of an experiment suggest a winning combination, you can choose to stop that experiment and run another where the only two combinations are the original and the winning combination. The winning combination will get most of the traffic while the original gets the remaining. This way, you can effectively install the winner and check to see how it performs against the original to verify your previous results."

And why should I run a follow-up experiment?

"Running a follow-up experiment will give you two benefits. First, it will enable you to verify the results of your original experiment by running a winning combination alongside the original. Second, it will maximize conversions, by delivering the winning combination to the majority of your users. We encourage you to run follow-up experiments to get the best, most confident results for any changes you make to your site."

But what happens when a follow-up experiment delivers contradictory results?

The screenshot below shows the original MVT test results....

I commenced a follow-up test running the the winning variant from this test in a head to head with the original default. And this is what happened...

The blue line is the original design beating the first test winning variant. This has happened time & again with my follow-up experiments. Then I noticed something. When you set up a follow-up experiment it's easy to overlook the weightings setting or the 'choose the percentage of visitors that will see your selected combination' option of a follow-up test. By default it's set to 95% for your selected combination.

Now I cant offer any explanation but from previous testing with other tools such as Maxymiser we've seen when you up-weight a particular variant in a test in favour of another, invariably it's conversion performance goes down, sometimes radically so. I recommend only doing a 50-50 weighting at anytime in any follow-up experiment because for whatever reason an unequal weighting seems to skew performance. Just be aware of this possibility and you'll be fine : )

If anyone can offer me a scientific explanation for this behaviour I'm all ears!

By the way, below shows the test after the weightings are reset to a 50/50 split. Bit different from the original follow-up experment no?

Give a Huq?

2010-10-14T07:07:00.000-07:00

It's good to see some examples of AB testing that aren't just about which web page works best. And here's another example of Magazine cover split testing. This months issue of Company magazine is running with two variations of it's cover, one with presenter Fearne Cotton, the other with presenter Konnie Huq (who's married to this guy by the way). As you can see both covers are the same bar mention of the featured presenter and the hero shot; even the poses are almost identical. I've mentioned magazine cover testing before in a previous article here but I havn't seen any recent evidence of the Press engaging in this kind of marketing test in recent years. Would be (mildly) interesting to see who wins this particular test.

Google Trends for MVT Terminology

2010-09-27T03:19:00.000-07:00

Out of curiousity I've ran a couple of queries in Google Trends to see what are the more popular terms in the world of web optimisation. I queried 'ab test' versus 'split test' and found the latter to be less widely used. However I was more keen to see exactly what terms people were using for 'MVT', 'multivariate', versus the old skool 'multivariable'. I thought that the term MVT would be more widely used in this day & age, or indeed be on the increase. It turns out to be far less popular than multivariate and on a par with 'multivariable'. I think it would be handy if we all stuck to a single expression, my personal preference being 'MVT'! This would certainly help for job searchs too! This particular blog gets ranked well for the term 'MVT blog' but is off the radar for 'Multivariate blog' too, so there's another side-effect to of our mis-aligned industry terminolgy! : )

UPDATE 16th August 2011: Whilst Google Trends is still a good tool for examining terminology usage or just trending full stop a far better tool for this job is Google's Keyword Tool . Obvious really : )

Amazon cross-selling

2010-09-23T06:44:00.000-07:00

This is clever. Well I think so anyway. Amazon are cross-selling their new Mastercard credit card via the medium of your actual shopping basket. They give you a £10 credit if you apply for the card but take this one step further by applying this credit in a hypothetic, yet obvious way to your current check-out total. Automatically your eyes are drawn to the discounted price of your purchase. Nothing seems unusual as we're now finely accustomed to the concept of discount codes et al. Personally I think this is fairly slick marketing example.

Online polls

2010-08-25T02:39:00.000-07:00

I'm always looking for tools and services that compliment the core optimisation activity we do using MVT and AB testing. In the past I've tried getting the general public to rate a selection of test variants using an on-line survey (see earlier post here for details). To me testing is always going to be the ultimate customer feedback on what works for on-line sales conversion but there's no harm in garnering opinion of your winning web design and page content. I'm now piloting on-line polling on a few landing pages using a Polldaddy poll widget. This simply embeds a rating widget in the top corner of selected pages enabling visitors to either give the page a thumbs up or thumbs down . Obviously the shortcoming here is that you cant tell if people are giving the page design or page content a good or bad rating, but at this stage I am trying to get a very high level user response to the page as a whole. Ideally I would have this widget on every page on the site as a means of flagging up any under performing pages or highlighting pages that people have an affinity with.

Is Fatwire any good?

2010-07-07T05:35:00.000-07:00

Looking for information on Fatwire and MVT testing?

If you didn't know Fatwire is a content management system...

We have been doing MVT and AB testing on a Fatwire (FW) website for the last 10 months. FW dictates that your website is constructed through a series of templates which dont lend themselves to conventional mvt tagging; well not easily anyway.

We decided upon using Maxymiser as a testing solution primarily because their one touch solution could be worked into the header of the websites master template, thereby enabling us to conduct tests across site with zero (any I really mean zero) IT department input from then on. Putting Maxymiser (or any third party testing tool) tagging in the header means it needs to be in a site-wide 'asset' of FW, thus enabling you to conduct MVT or AB testing anywhere within your website. Doing this allowed Maxymiser to then render out alternative/test content on top of your 'restrictive' Fatwire content.

If you want to be consistent with your Fatwire templates, you really need to come up with test variants based on the content restrictions of your current, available templates. This is a high level choice to make. Personally because Maxymiser could render any designs I liked over the top of the current suite of Fatwire templates we had, I decided to go with page designs and content that didn't bear much,if any, resemblance to the Fatwire content beneath. This was because I wanted to see what gains could be made by testing alternative layouts and use any uplifts as leverage with our IT department in getting new templates built, becauseI could prove the benefits of doing so. Additionally while your running an MVT test over the top of the Fatwire you are effectively 'making hay while the sun shines'. By that I mean that even while your running a test, nine times out of ten, because your layouts are not restricted by the Fatwire template business rules, you're already going to benefit in customer conversion during the test period alone which you wouldn't have got otherwise. I will talk about using your testing tool as a CMS in a later post. It's a bit controversial : )

In all honesty if I had the choice I would not be working with Fatwire at all, especially not the very old version we have. It's effectively a step backward in terms of web design, it's highly restrictive in terms of design flexibility. The illustration I've taken to recently using if that a one man band building an e-commerce website in his bedroom using something like Microsoft Front-page, actually has the potential means to add 90% more functionality to his website than someone using a traditional content management system such as FW. Think about that. If you have some JavaScript based web service or dynamic content you want to have a go with using, just by way of a trial, with FW, how much heartache do you have to go through to get that tagging, or JavaScript included into your website body header or body? If it's not a major hassle to you or your IT department then fair play but you're probably not the norm.

Anyway here's my true feelings about good old FATWIRE laid bear through the medium of satirical animated video.....

Google Optimizer - Landing Page A/B Testing

2010-05-07T05:06:00.000-07:00

I launched a split test on one of our highly trafficed Current Account landing pages last week.

This basically saw the rather stale current champion page design challenged by a much more creative led design for the same page. The graphic above shows the readout for the visitor conversion rate which shows the creative led design as the outright winner (16.9% uplift in click to apply rate). However, I have tagged both pages so that we can identify which pages actually result in submitted current account applications, and the results show the original design leading over the creative led design. This is yet another example of how pretty design may compell people to click an apply button but if they havn't read the fine detail they're less likely to go through the entire end to end transactional process.

The original page design (champion) is shown below on the left and the creative led design (challenger) is shown below on the right.

An unbiased review of Google Website Optimizer

2010-04-30T07:37:00.000-07:00

I notice that a lot of reviews for Google Website Optimizer (GWO) have been done by what at first appear to be independent companies & individuals, but at second glance are actually affiliated or partnered to Google in some way. So this being the case, here's my unbiased opinion & experience of GWO for what it's worth. And please note: I am not pushing or offering any service here relating to Google.

Anyway, I've been itching to give GWO a trial for the past couple of years. Up until now our websites have not been conducive to implementing Google tags and besides which we've been using a managed testing service from Maxymiser. However a few things attract me towards GWO over a paid service.

First off, it's free. If you have the means to implement Google tracking tags into your web pages, why wouldn't you at least try a test or two?

Secondly, even if like our company your paying a third party company to build and run multivariate and AB tests for you on your behalf, no matter what contract you're on there's never enough time and resource to run with all testing concepts and ideas you might like. Which is why I've persisted with GWO for the past couple of months as it's relieved a bottleneck in our testing schedule being able to run quick & dirty tests on the fly alongside our more formal testing.

If your business is to be an optimization expert you need to try a variety of tools, especially the most commonly used one to add to your knowledge base.

We now have a sub domain now where we can do whatever we like on landing pages including testing and tagging with GWO. I've now conducted three landing page tests ranging from AB tests to MVT and just launched my fourth.

My first test was on a Personal Loans page where we tried a dozen different page designs and copy changes to see if we could get more people to apply for a loan. It was a fairly simplistic affair but has yielded a 14.77% uplift in visitor conversion by offering up a different page header and product introduction copy.
The second test was on a High Interest Current Account landing page (phase 1). This looked at simplifying overly complicated product information and has yielded a 20% uplift in conversion. I've just launched a phase 2 of this test where I champion/challenge the previous winner, a rather stodgy design against a more creative led design.
My third test was recreating a Bank Accounts comparison page from our main website but in a landing page environment. We then employed the usability web experts Foolproof to come up with an improved design for the page and test it (amongst other designs) against the default page. It's still running and yielding a 9% uplift in submitted current account applications to date.

Now the problems with GWO.

For whatever reason, aside from my most recent test, none of the tests have displayed progress data in the reporting interface of the GWO console! Fortunately in every test I have been able to tag each variant with a bespoke/unique tracking value which appears in our downstream sales database when someone submits an application . Because of this I have been able run successful tests with a reliable MI to go by.

Customer support for GWO is non-existent and you're reliable on forums to get any kind of useful information about troubleshooting. I have had to solve a number of glitches with the tests ran to date myself using significant time & effort!

Because I have web developer experience I am able to setup a test and build the page variants for my test also. But this is not the norm and I can imagine working with an internal or external agency to implement and update GWO test tags and content might make the whole experience unworkable.

Actually coding the test variants is badly handled in that they just give you a single unformatted text area to do your HTML editing in, so lot's of cutting and pasting from a conventional web editing tool is involved, such as in Dream Weaver or Visual Studio (even Notepad is better!).

The positives of GWO

I love the GWO console. Both thr reporting interface (below is the reporting interface showing a current MVT test in progress) and the test build interface are simplistic but highly functional in design. The step by step code implementation are really straight forward. You tell GWO what your original, test and conversion pages are and it tells you what code to paste into those pages to get the test going.

It allows you to weight test variants so that they are displayed to a set portion of your test audience, and the test follow-up feature is great too for validating the results of a concluded test.

If Google could come up with a solution that didn't involve placing tags on test and conversion pages then I think the technology could take off like a rocket.

In conclusion

So besides some teething problems with getting up to speed with GWO its been a good learning curve and I've been able to turn in some very reasonable test results to compliment my other testing.

If you want to see a previous article comparing GWO to other optimisation providers click here

Culling multivariate test variants in a Maxymiser test

2010-03-02T04:42:00.000-08:00

I've covered the topic of culling before on this blog here. Now I'll go through my method of identifying test variants to cull from a running MVT test in Maxymiser where you have multiple actions.

1. Below is a screengrab of an MVT test report in Maxymiser after a test has run for a week. On the left side are the bottom ranking variants for a 'click apply button' action and on the right are the bottom ranking variants for a 'submit application' action.

2. The conversion rate uplift is negative for these variants and are not adding anything to the test overall and so need to be removed or 'culled'.

3. Indentify page combinations (see Page ID field above) that are both negative in uplift and appear in the bottom ranking across both actions then select the 'remove page' option within the console.

4.Looking at the test report before and after the cull identifies the immediate effect on the 'chance to beat all' metric within the test.

The chance to beat all value moves from 27.86 % for the lead variant to 28.11%, this uplift shift also cascades downwards through the other variants left in the test.

This culling exercise would then be repeated at periodic intervals for the remainder of the test.

note: Caution should be taken when removing variants from the test. The number of generations and actions should be taken into consideration, whilst over-culling a test can bring it to an early and unproductive conclusion.

Optimisation providers compared

2010-02-09T03:03:00.000-08:00

We recently reviewed Optimisation providers and their tools as we needed to procure another years contract for such a tool and service. Here's a few advantages and disadvantages that we found with a few of the top names in the business at the moment.

Omniture

Advantages

Comes with segmentation engine by default
Has self optimising content feature once test has completed
No limits to the number of domains you can test on

Disadvantages

Requires multiple pieces of code on every test page - very IT resource heavy
Reporting only shows winning design not all results so test within test not possible
Poor reporting interface
Technology-wise they seem to have fallen behind the opposition providers.

Maxymiser

Advantages

Solely undertake optimisation work, not distracted by other functions such as web analytics etc
Minimal IT resource required as tests require a single line of Javascript at the bottom of the test page. Once implemented no further work is required.
Full end to end managed service including design resource.
Product has real time reporting
Manual control over tests if required,i.e; culling of test variants
They have an extensive serving infrastructure that can serve winning content until new templates can be developed.

Disadvantages

Initial quotes were high compared to competition
Sometimes sluggisn in terms of test turnaround time from past experience

Autonomy

Advantages

Offers all the functionality you'd expect from a top-end provider
Full managed service
Very strong test recommendations and results analysis
Appears to cope well with more technically challenging tests
Uses Wave testing approach which means end results are more accurate when implemented live
No limits to numbers of tests and number of domains covered with contract
Segmentation engine included
Minimal IT resource required as tests require a single line of Javascript at the bottom of the test page. Once implemented no further work is required (although complete clarification on this capability was ever obtained from the vendor at the time).

Disadvantages

Poor reporting interface
Solution requires code to be put in the header - this may cause issues for templated sites
3 month contract required to implement a Pilot
Perception of a limitation on doing radical page layout testing, more elemental testing favoured.

Sitespect

Advantages

Once Sitespect server implemented then no additional IT resource required
Unlimited control over what content we wish to test and no limits to the number of tests that can be run (limited only by our resource)
It can test all content including any online forms
Comes with a segmentation engine at no additional cost

Disadvantages

It is not a Managed Service so labour intensive from an eCommerce point of view
Developer know-how required within our team
User interface very complicated and requires full training
Results reporting not very user friendly

Google Optimizer

Advantages

It's free to use
Wide customer base and as a result there seems to be a wide knowledge base although forums are not the best means of resolving a critical testing issue in a timely manner!
Likely to improve over time due to the volume of people using it.
Integrated reporting with Google analytics, which is good if you like the GA interface.
Can be either self optimising by automatically removing under-performing variants or done manually, but once variants are out of a test that's it.
Can be adapted to do event tracking, ie. onclick events so in theory you dont need a conversion page to undertake a test which is dead handy.

Disadvantages

Fairly IT resource heavy in its implementation
Cant easily track across domains

Obviously not a managed service
Can only record an action on the test page and not all the way to application submit (hence missing a critcal metric for serious testers).
No consultancy service from Google so dependant on forums for problem solving
Feels like it's still in beta
Have been some noise about outages and slow reporting in terms of seeing test progress or even confirmation that your test is firing correctly.

OUTCOME OF TESTING TOOLS EVALUATION

In the end we choose Maxymiser mainly because we'd already worked with them before and knew what to expect more or less, going forward. Also because they had the ability to remove under-performing variants within a test and their reporting console is user friendly compared to the competition. They are keen and quick to improve their tools and service and welcome feedback.

UPDATE:

Since this article was originally posted we have also decided to pursue use of Google Optimizer in parallel to the chosen managed service as it's a free service and I personally have the developer know-how to code the test variants without IT input (ie, it's not for everyone in the testing world).
UPDATE:I've posted my feedback on this in a seperate article here .

Riding the tsunami - optimising an online marketing campaign

2009-12-18T12:35:00.001-08:00

In the last three months we've taken web optimisation in a new direction. To date we've been trying, (emphasis on 'trying') to follow the essential principles of mvt and split testing. Putting in test content, wait a period of time, wait for visitor volume to do it's work, seeing what works in getting more customers to convert, seeing what doesn't, etc, etc. This is all well and good and will always be the core of what we do, but recently I've decided to just go hell for leather on getting an uplift and cashing in on an already positive proposition.

Our bank pioneered the cash switching incentive. This is where you can switch current accounts and the bank (us) will give you £100 cashback for the pleasure. This has been an extremly good promotion for us and we will routinely serve this promotion on the website to drive current account applications, and boy does it work. I cant give you the monetary figures for obvious reasons, but there's a very good reason other highstreet banks are now offering this same incentive. Anyway, after seeing the sudden peak in traffic that you get to the site during the campaign I decided to perform an optimisation test during the promotion period on the customer journey or offer page. I basicaly came up with half a dozen alternative page designs that focus on the promotion and were ran directly against the default page design using our normal testing tool (Maxymiser). Because of the nature of the promotion period it is definitly not a conventional MVT or split test. I had no intention of seeing what would emerge as the outright winner over time I just wanted to serve the best performing content for that moment in time. So you run the 'test' and due to the massive traffic volumes you can just sit there and watch the customers come in and crank through the test variants very quickly. You soon see what works and what doesn't. I then engage in some aggressive culling of the negative performing variants.

So after an initial trial I already know what the best page design is for this particular campaign and will serve this content again during the next campain period, knowing it will be sucessful and do it's job. This means I can just challenge this 'champion' at my leisure. But in terms of results the first run gave me a 44% uplift over the default page design in getting people to apply for the product. Wow!

UPDATE: 15th August 2011.
Coming late to the party, but getting there in the end are Get Elastic ( a very good web optimisation site) endorsing this approach (after a bit of soul searching) with an article titled

Should You Avoid Testing During a Traffic Spike?

Optimization of a product page

2009-10-27T06:00:00.000-07:00

Like most highstreet banks we have a Loans product page, but we hadn't carried out any kind of optimization activity on this page, or this product for that matter. Generally speaking the thing that sells a loan is the rate (natch). But we have found that in terms of what visitors want to see on a Loans product page is the ability to use a loans calculator to see what their repayments are likely to be etc.
Therefore the following test was planned and conducted...

Objective – To get more people to use the Loans Calculator and more people to apply for the Loan product and submit the application form.

The test was ran continuously for 27 days between the dates of 21/08/2009 to 16/09/2009.

Test audience – 100% of traffic to Highstreet Bank Loans Hub page.

Hypothesis – By testing alternative page designs on the Loans Hub page we expected to see an improvement in the visitor to apply ratio by 3% and improve our visitor to submitted application ratio by 5%. We also hoped to see an improvement in visitors using the calculator by 3%

Test Results Summary

9.08% improvement in visitor to apply ratio
4.59% improvement in the submitted application ratio
4.12% improvement in visitors using the calculator
5.06% improvement in product accept rate

The existing page design was as follows (note the placement of the loans calculator)...

The winning page design was as follows...

Put into context
To put this improvement into context, if all of the 126439 visitors during the test period had been shown the new design :

126439 Visitors to winning design x 16.17% visitor conversion x 33.2% accept rate during test = 6787 accepted applications

126439 Visitors to default design x 15.17% visitor conversion x 31.6% accept rate during test = 6061 accepted applications

This equates to a 11.9% increase in accepted applications

Findings

Bringing the loan calculator and apply now button front and centre focuses a visitor on the primary calls to action; perform a calculation, click apply.

The static right-hand promo tile whilst previously used to host the loan calculator was now used to promote creative that played on 3 core themes; ‘cash’, ‘car’ and ‘home improvement’. We wanted to see which image and theme appealed to the most loans customers. In the past we have used a variety of images in the loans section that appealed to these areas, this test would go some way in answering which theme had the widest appeal.

We’re aware that images can be largely subjective in either turning people on or off but using a very basic image that illustrated a single concept provides a jump-off point for enhancing the ‘home improvement’ message relating to a personal loan application.

How to increase average spend - The 97p Shop

2009-09-08T05:40:00.001-07:00

Cross selling and add-on sales are nothing new. If you visit Amazon often enough you'll be more than aware of the best in class method of cross selling, they are the masters. Wish lists, gift lists, recommends, you name it, they have it. Anyway something caught my eye the other day when I was visiting wilkinsonplus.com. They've introduced the 97p shop which essentially offers a range of items for, yes you've guessed it 97p. They promote this shop right across the customer journey.

1.

2.

3.

By the time you've selected an item from their main range and you are at the checkout your already primed to top-up your purchase with an item from the seemingly low cost range. Most people wont blink at spending an extra 97p on the average transaction and this particular price promotion will easily fly under most shoppers cost radar. The growth and success of retailers like Pound Stretcher on the UK highstreet during this economic downturn also adds weight to this particular promotion. We've already adjusted our psychological perceptions of the value of a £1 and how far it should go. Wilkinsons are effectively taking on these established budget shops by playing them at their own game. A similar approach can be seen if you visit your local Tescos where they've introduced budget ranges that compete directly with the likes of Lidl and Aldi. If Wilkinson.com make a success of this campaign (as I suspect they might) they could end up effectively owning the 97p value in customers minds and will be an endemic part of their brand

Rate our page designs

2009-09-07T02:01:00.000-07:00

We've been conducting a multivariate test on our Personal Loans page. The results are okay but not conclusive. If you've got a minute we'd love it if you could rate the last 5 page designs left in the test.

~~Take our very quick survey~~
UPDATE: Sorry looks like Glowday.com went out of business and took our survey down with them : ( Thanks to everyone who completed the survey though, much appreciated.

Eye Tracking

2009-09-02T06:10:00.000-07:00

I recently had the opportunity to witness Eye Tracking research on our websites. This was undertaken at a company called Foolproof in London. Eye tracking is the process of measuring either the point of gaze ("where we are looking") or the motion of an eye relative to the head. Eye tracking relating to website usage is a method of capturing eye movement relating to web page design, content, layout and it's usage. Eye trackers are used in research on the visual system, in psychology, in cognitive linguistics and in product design.

With regards to this particular session you get to sit in a viewing suite, on comfy sofas, while people next door are monitored whilst they conduct a customer journey on your site. Their eye movements are tracked and you can see their monitor in front of you projected onto a big screen. The image below is of this event captured rather badly on my mobile camera (apologies for the terrible quality) Bottom left is the interviewee on a video feed and on the right is a projection of their monitor with eye tracking overlaid in realtime.

It's all very interesting. You do get a real insite into what people genuinely experience in the real world. There were many learnings to be had from this session. Too many to repeat here. One key one for me however, was how users view our '3 Box Wonder page' design. This is a page design we arrived at as an outcome of extensive multivariate tests (see image below). It's an unpopular design here in ecommerce but it performs consistently well in getting customers to convert. The eye tracking finding suggested that the reason it may be successful is that the negative points relating to the product are contained within the 3 stacked boxes to the right. These boxes happen to look like Google adword ads so are almost subconsiously ignored by the visitor. This is not something we had intended by this design but it certainly goes some way to explain why this design is so good at what it does! It also helps to prove the value of eye tracking too.

3Box Wonder...

Twitter tools (Part Two) - revised

2009-08-13T08:31:00.000-07:00

Following on from my last Twitter related post about a few tools that caught my eye I'm adding a few more to the list (maybe not all 100% useful but show an interesting use of the Twitter API none-the-less):

1. http://tweetingtoohard.com/ - This site describes itself as a place 'Where self-important tweets get the recognition they deserve'. It's a listing of the most egotistical tweets/tweeters out there. Disturbing but fun.

2. http://tweleted.com/ - Thought that tweet had been deleted? Think again. Handy for looking up Gail Porters (@Gailporter) mobile number too.

3. http://twitturly.com/ - This site tracks and ranks what URLs people are talking about on Twitter. Looking up individual Tweeters URL postings is quite useful too.

4. http://dossy.org/twitter/karma/ - Similar to my previous recommendation of http://friendorfollow.com/ for graphically representing who's following who but in a much slicker interface.

5. http://twistori.com/ - A failrly basic, but slightly compulsive site for displaying live feeds, grouped by six simple concepts; love, hate, believe, feel and wish.

Read part one of this post (more Twitter tools)

Behind the paywall - Can you really charge for News content?

2009-05-14T02:54:00.000-07:00

As MediaGuardian, New York Times and News Corp as a whole started making noises again about charging for their content it begs the question of whether this idea is at all workable? There seem to be too many barriers to this ever actually happening?

1. The BIGGY. Most web users can get the same information a click away for free and carry the mindset that the majority of web content is free. How can you change this ingrained expectation and behaviour?

2. The obvious scenario perhaps. Website X throws up a Paywall* and starts charging for access to it's cherished content. User Y subscribes to content, copies content and immediately disseminates content for free.

3. Advertising revenue. Advertisers would run to the hills. Why pay for placement on a site that will invariably attract/get less visitors than before? The exception to this has been FT.com.
Whilst they successfully charge their users high premiums for access to content they can expect to charge equally high premiums to advertisers. Those that do pay for digital content are a valuable audience, both in terms of the additional profile data from registrations and the simple fact that they are obviously willing to pay. But, and it's a big but, on the web they are still the exception not the rule.

4. The BBC. The BBC is as ever proving to be the eccentric case in this mix. They already have a Paywall in the sense that they charge the UK public licence fee £142.50 annually for it's service through it's Public Service Broadcasting remit. The online content continues to be funded through this and while this is the case how can other news sources possibly compete?

5. Supply outstrips demand. There are still more news sources than consumers on the web. A survival of the fittest invariably kills off the weaker brands over time but the biggest growth in news feed comes from the blogging community and social media.

6. Credit Crunch. As we continue to be gripped by the economic downturn how many of us really value our daily news injection from Website X, Y or Z over other more 'essential' financial obligations?

* Paywall - A website that restricts access to certain content only to paid subscribers.

UPDATE: 20th April 2011
This article in the Guardian today has probably given us a sneak peak of the future of Paywall content. Slovakia's media throw up a universal paywall :

Page Combination Removal Feature in Maxymiser

2009-05-13T06:37:00.000-07:00

We use Maxymiser as our Multivariate/AB testing tool of choice. Maxymiser allows you to see how individual test variants are performing in it's console. In a previous post about culling variants I talked about how we like to actively remove under-performing variants from our tests. Well this has been a contentious issue, whether or not it's the right thing to do in a test scenario (see post for full discussion). However Maxymiser have recently added a new feature to their test console that allows you to remove under-performing page combinations* from your test. Doing this also allows you to immediately and clearly see the impact of performing such an action and its immediate effect on the remaining page combinations.

Below is a screenshot of our Maxymiser console displaying an active test before we remove an underperforming page combination. P8 page combination is highlighted as an under-performing page combination. It has an uplift of minus 1.76% and a 'Chance to Beat All' value of 25.23%.

Now after we exclude the P8 page combination from the test you can see what the results look like. The lead page combination P2 leaps from a 39% 'Chance to Beat All' to 51%, hence speeding up your test. The overall uplift value also moves from 3.19% to 3.59%.

So the introduction of the page exclusion feature actually allows you to experiment more with 'What If' scenarios. We like to think that this enhancement was made to Maxymiser as a direct response to our need but it's probably not the case!

* A page combination is a collection of multivariate test variations

My Twitter toolset - a few of the most original & useful tools out there

2009-05-12T05:13:00.000-07:00

There are many listings of the plethoria of tools that have been developed for using directly or indirectly Twitter and it's many facets and features. I've decided to put together a list of what I think are the most useful of the ones I've seen & use regularly. Here's my top eleven (everyone does 10):

1. http://twittersnooze.com/ TwitterSnooze - I love this. You can pick people you are following who, shall we say, are a little over enthusiastic on the tweet front. You can 'rest' them from your news feed while not unfollowing them altogether. Genius really.

2. http://twitterholic.com/ Twitterholic - gives you the current list of who's most popular on Twitter and also let's you find out your own ranking. Rather surprisingly @stephenfry is ranked somewhere in the mid sixties at time of writing.

3. http://www.hashtags.org: “#hashtags “or 'Hash tags' – a word prefixed with the # symbol – are used by Twitterers to make it easier for people to follow key topics on Twitter. For instance, users who were tweeting about the recent Swine Flu breakout added #swineflu to their posts. hashtags.org provides an at-a-glance running ticker of popular hash tags, to help Twitter users keep up to speed with hot topics.

4. http://friendorfollow.com/ Friend or Follow - allows you to see who your following but who's not following you back (how rude). You can then manually unfollow each if you so desire or use the next tool in the Twitter arsenal...

5. http://socialtoo.com/twitter SocialToo - This tool allows you to do a host of different things automatically in Twitter. Some services, however you will have to pay for. Amongst it's features are the ability to automatically follow people who follow you and automatically unfollow people who don't follow you. I've signed up for this particular paid service as I think it's actually the most powerful Twitter application I've seen so far. You have to kick it off periodically and the script can take an eon to run but it does what it says on the tin, you'll log into your account after it's run and suddenly see you are following an equal number of people to the number who are following you.

6. http://tweeteffect.com/index.php Tweet Effect - This tool helps you to find out which of your Twitter updates made people follow or leave you. Now it's certainly an interesting incite into the way people view you on Twitter but it has to be taken with a pinch of salt. It basically suggests that for every tweet you gain or lose followers. Twitters not that straight forward in reality, people sometimes choose to follow/unfollow an the basis of other reasons not as the result of a single tweet. That being said it's still a good barometer for your news feed.

7. http://www.twitpic.com/ TwitPic - Basically this site lets you share photos on Twitter. You have a Twitter account, you therefore have an implied TwitPic account. If you send a photo from your phone or whatever to your Twitter account it gets hosted on this site. Settings allow you to either automatically post the image to your Twitter feed or manually do it at a later date.

8. http://bit.ly/ Bit.ly - a site where you can pump in your stupidly long URL and get it shortened to a much more Twitter-friendly (below the 140 character limit) length. Or alternatively there's http://tinyurl.com/ the original URL shortner

9. http://www.twitalyzer.com/twitalyzer/index.asp Twitalyzer - a tool to evaluate the activity of any Twitter user and report on relative influence, signal-to-noise ratio, generosity, velocity, clout, and other measures of success in social media. This one's okay but again I have my reservations. Yes they're using some interesting metrics to measure success but it seems to me that success in Twitter will always be measured by the number of followers you have, it doesn't matter which way you cook your stats!

10. www.tweetdeck.com Tweetdeck - Doesn't really need any introduction but ...it's an extremely popular desktop app that makes it easy to break feeds into manageable chunks, and even categorise replies. Requires a computer running Adobe Air, but works on both PCs and Macs.

11. http://www.twittergadget.com/ TwitterGadget - is just one of dozens of Twitter clients for iGoogle. It allows you to do almost everything you can do in Twitter but from your iGoogle homepage. You've got to be a fan of iGoogle to buy into this way of working, which I am!

Read Part Two of this post (more Twitter tools)

Striking while the iron is hot

2009-05-07T02:03:00.000-07:00

Probably the holy grail of any web optimisation exercise is to be serving the most optimised page content at the most effective time possible. For us in banking this opportunity presents itself once a year in the form of the ISA Season*.

We were keen to improve the conversion of ISA landing pages in preparation for this busy sales time for the bank. This test was for an ISA savings product page and we were looking to test header copy, images and body copy combinations in order to improve conversion. The winning design would be implemented in time for the peak volumes during this period.

The Hypothesis for this test is as follows:

By Multivariate testing combinations of header copy, images and body copy on the ISA product page we expected to see a 10% improvement in number of visitors starting an application for a product and improve out visitor to submitted application ratio by 5%.

This test was run twice (Between 13/03/2009-06/04/2009 (Pre Tax Year) and between 06/01/2009-13/01/2009 (Post tax year)). We were forced to conclude the second test early due to the product being withdrawn but in both instances the winning design was the ‘Right hand side tabbed content’ although we did have alternative images winning in each test. In the final test we saw the lead design resulting in a 16.14% increase in conversion (17.13% in the first test) over the default design with a statistical significance level of 98.65%.

* End of the old tax year and the start of the new one and something many in the financial services industry call "ISA Season"

Omniture Summit 09 London - Key Takeaways

2009-04-22T07:45:00.000-07:00

I attended the Omniture Summit 2009 in London yesterday, travelling down to the big smoke from Leicester. We paid £178 a head to attend this shindig attended by around 800 other like-minded individuals. Although we dont currently use any Omniture tools (we're strictly Maxymiser right now and frankly dont have that kind of money) we thought we'd see if we could get some optimzation ideas from our fellow analyst & marketeers at the event.

So my takeaway learning's for the day were:

1. You're justified in letting brand design and message go out the window if there's a decent uplift in conversion to be had as identified by your testing.
2. BT segment their customer by IP range. If the customers IP is within the BT ISP range then they're an 'Existing Customer', if they're not they are a potential 'Switcher'.

3. AutoScout24 use Google Adwords to segment they're customers. Those that search by brand and those that don't. If you arrive at they're site via a keyword you're part of their optimization test.
They've learnt that those who search by brand prefer a highly branded landing page, while those that don't use a brand term prefer a lesser branded landing page.

Breakout talk - 'Listen to your audience: Test Your Content & Learn Where to Take Segment-specific Action'. Speakers: Sam Calvert, Head of Online Sales, British Telecom, Sebastian Wetteraurer, Teamleader Web & Data Analytics AutoScout24.

4. What worked well in AB Testing :

Multiple calls to action
Empty pages convert better than fuller pages
Google users display different behaviour to non-Google users (although they didn't specify what behaviour)
Experiments need at least 2 to 3 weeks to run

Breakout talk - 'Why Should A/B Testing Be in My Marketing Strategy for This Year?'. Speakers: Nicolas Meriel, Team Lead Test&Target Consulting Services, Omniture. Gerard Lindeboom, Senior Web Analyst ABN AMRO.

5. Realise that emotion is a big factor in website design.
6. Usability labs are invaluable in understanding your customer experience.
7. Look after your customer. Delight and surprise them and they'll tell the world.
8. Aim to make navigation perpetual or persistent.
9. Familiarise yourself with customer language, i.e. do they search for a 'settee' or a 'sofa'?
10. Male & female search methodologies are different.
11. Relevancy is key so dynamic landing pages are essential.
12. Promote best products and offers in prime areas of your site. Products need to continually prove themselves for home page placement.
13. Optimise for any seasonal or opportunist event.
14. Keep up-to-date with search terms.
15. Abandonment - understand where you are losing customers and use testing to fix.
16. Establish a realistic conversion target.

Breakout talk - 'Flex Your Merchandising Muscles: Using Omniture Merchandising to turn shoppers into buyers. Speakers: Alison Lancaster, Marketing Director Harrods Direct, Chris Moffatt, eMerchandising Consultant, Omniture.

The optimisation of a landing page test (part one)

2009-03-30T07:27:00.000-07:00

We have an AB test running on a product page and landing page simultaneously to see which location would convert better. The background to this is that we once used landing pages a lot, and then the industry as a whole (banking) decided to move over to dropping people straight onto a product page. It became the established wisdom that product pages were better than landing pages for optimising your traffic. This test sought to revisit this hypothesis. We decided to use a specific traffic source, identify them on page load, split the traffic 50/50, 50% remaining on the product page, 50% redirected to a bespoke landing page.

As well as testing the different locations we wanted to test a few other concepts of optimisation opportunities we'd heard about:

1. Do bespoke welcome messages help in converting users to apply for the product?
2. Do the presence of primary navigation in landing pages actually help visitor conversion?
3. Does a 'band wagon' message help uplift; i.e., "join the thousands of other visitors in choosing this product" etc.

When we started the test we soon found out that the traffic source chosen to use as the test audience was a lot smaller than anticipated. Within a few days we had to adjust the 50/50 weighting and send 100% of the traffic to the landing page with a view to retesting whether product pages work better than landing pages at a future date. If we'd left it as it was we would still have been running the test 2 years later, so low was this traffic to the page from that specific traffic source.

As the test is still running we're still unable to answer question 1. 'Do bespoke welcome messages help in converting users to apply for the product?'. However, within the first 2 weeks we were able to answer questions 2 & 3. The primary navigation in the landing page performed very badly, allowing users to leak out from the test page and not convert. The welcome message using the band wagon theme also failed to convince any users to apply and also performed very badly. Both the presence of the primary navigation and the band wagon message are shown in the test page combination below (as taken from our Maxymiser test console). Upon conclusion of this test we will re-run the winning test variants on the product page to see again which page gets the biggest uplift in conversion, this will be covered by Part two of this post.

What is Statistical Significance?

2009-03-26T04:44:00.000-07:00

I've sort of overlooked this topic since establishing this blog but for subject completeness shall we say, I think I should now mention the role of statistical significance in optimisation testing.

One of the biggest headaches to running an AB test or Multivariate test on your website is knowing when your test is complete, or heading towards conclusion at least. Essentially how do you determine signal from noise?

Many 3rd party tools give you the metrics to determine a tests conclusiveness, for example the Maxymiser testing tool displays a 'Chance to beat all' metric for each page combination or test variant within your test.
But more importantly, what underpins these tests is the concept of statistical significance. Essentially a test result is deemed significant if it is unlikely to have occurred through pure chance. A statistically significant difference means that there is statistical evidence that there is indeed a difference.

Establishing statistical significance between two sets of results allows us to be confident that we have results that can be relied upon.

As an example, you have an AB test that has two different page designs. Analysing the data shows there are two results:

Page 1 - 1,529 generations with 118 responses or actions - giving a conversion rate of 7.72%.
Page 2 - 1,434 generations with 106 responses or actions - giving a conversion rate of 7.39%.

Looking at the two results which do you think is the better? Is page 1 better because it as a higher conversion rate that page 2? Using statistics and firing those 2 results through a basic Statistical Significance calculator (I'm using this one Google's Optimizer test duration calculator) tells us that the two results are 0.335218 standard deviations apart and are therefore not statistically significant. This suggests that it is highly likely that it is noise causing the difference in conversion rates, so plough on with your testing. If a 95% statistical significance is acheived you can safely say the test is onclusive with a clear winner. This is also indicative of a strong signal and gives you a result based upon a wholly statistical basis as opposed to human interpretation.

The HiPPO factor

2009-03-24T03:49:00.000-07:00

Just learnt a great new acronym today 'HiPPO' - Highest Paid Person's Opinion.
Never been more apt than when you're doing an optimization exercise on your website
and the 'HiPPO' blows your proposed new design out of the water because they dont like it based upon purely subjective thinking.

Found this on Dave Chaffney's site in an interview with Avinash Kaushik, author of Web Analytics - An Hour A Day.

The reptilian brain & your inner buy button

2009-03-18T04:10:00.000-07:00

When we do AB testing or multivariate testing you are generally experimenting with a persons unconscious response to your test content. Basically does a red apply button work better than an orange one for instance. Marketing experts refer to this as connecting to or appealing to the primitive or 'reptilian brain'. When someone arrives on a landing page and is making a quick intuitive based decision you are generally dealing with the inner, more reactive brain, not the higher conscious brain. Your not asking for any in-depth analysis of your page content you just measure what works best for getting a person to perform a desired action in your test. It seems for the majority of the time the reptilian brain is at the driving wheel so we need to talk directly to the driver.

This is considered to be neuromarketing in it's most basic form. The study of how we respond to adverts and products at a neurological level. In research in this area volunteers have their brain activity monitored via brain scanners whilst being exposed to marketing media to measure their response. The ability to understand how the inner workings of the brain processes images and messages and reaches decisions potentially gives marketers a new tool to fine-tune ads and marketing campaigns, bolster and extend brands and design better products. Marketers’ use of neuroscience technologies has alarmed some consumer groups who fear that it could lead to the discovery of an 'inner buy button', which, when pressed, would turn us into automated shoppers. Such fears spring from the increase in marketing-related problems such as pathological gambling, obesity and Type 2 diabetes.

The power of the testimonial [TEST 5]

2009-02-13T02:57:00.000-08:00

Been running a multivariate test on a current account product page again. This test has been running for over 3 weeks and not performing too well it could be said. Below is the readout from our Maxymiser console for this test.

So we sat down and thought how we might turn this under-performance around. I had been hoping to try and use a customer testimonial in a test for a while so I suggested that this might be the time & place to do just that. On the product page we had a test element that displays a strapline immediatly under the product title. So we took the worst performing variant and changed it to read a short & snappy customer testimonial instead (see before & after below).

Now we re-commenced the test. We didn't go for a complete restart, we just take the test results from the new start date of when we added the testimonial variant into the mix. And a few days in and the difference is incredible, the top performing page combination by a wide margin contains our new testimonial, this is identified as V_013F on the report below. So just goes to show, the test is still running at time of writing but we'll see if this performance degrades over time and whether this new test variant ends up in the winning page combo. At the moment though a 26% uplift in conversion shouldn't be sniffed at, and it's another thing we can tick-off as being tested.

UPDATE: 24th March 2009

I thought I'd better update this post today to take account of what's happened to this test. So the test has been running now for 63 days. Below is a screenshot of the report console for today. As you can see the V_013F variant (which is the testimonial variant) is placed in the top 3 winning page combinations. On the embedded graph I've pinpointed where we introduced this customer testimonial variant into the test. It seems that after a brief lift the test settles back down into a level pattern acheiving a roundabout 11.5% uplift in conversion, no longer retaining the 26% uplift first attained. The Maxymiser 'Chance to beat' metric also hovers around the 46% level for the best page combination (P10) so the test seems in a state of flux and not really progressing towards statistical significance. I should also point out that we've culled a few negatively performing variants from the test.

Seasonal shift

2009-01-08T07:28:00.000-08:00

The established wisdom of web page optimization suggests that images, and particularly images of people help build a connection between the visitor and your content/product/brand, See earlier post about quick wins regarding this.

We have been testing this theory through ongoing multivariate (multivariable) testing on several product pages on the site. However we've inadvertently created a rod for our own backs.

The original test was started back in November and we gave it an autumnal theme. The test variant with these images originally performed well but a gradual errosion in performance occured as the test and time wore on. The test was still running as we entered the Christmas period and suddenly the autumn theme looked out of place so the drop in visitor appeal was understandable. To confirm this we revisited the variant and replaced the autumn images with winter based ones. Again an uplift in performance was immediatly acheived. The test is still running and currently producing a respectable 8.95% uplift in conversion; but it's still wintery outside with the occasional snow flurries to reinforce that, so maybe this explains the appeal. Now as time moves on it will be interesting to see if this begins to errode in performance too.

So we have a situation where we know a particular content works but it will require constant maintenance to keep on top of the shifting seasonal themes. We'll need to come up with a sustainable image or theme that will work as well despite the time of year. We still have an MVT test on the roadmap for March/April that plays on the Spring theme but hopefully this test will be it's last incarnation. All of the past and proposed images are below, we've nicknamed this variant 'Shallow Grave' due to the half-buried nature of the models in shot.
AUTUMN > WINTER > SPRING.....

Emphasis of key words [TEST 3]

2009-01-07T02:34:00.000-08:00

A recent finding of our optimisation testing on our product pages is that if you embolden key words that emphasize a positive aspect or feature of your product you can more or less guarantee an uplift in conversion. The image below is taken from a recent winning test variant of a multivariate test we conducted on one of our savings product pages.

This winning page design resulted in a 4.32% improvement in conversion of visitors clicking on the ‘Apply Now’ button and more importantly it resulted in an 8.21% improvement in the conversion of visitors submitting the product application form. We're now rolling this keyword emphasis out across our other product pages.

Exclude ineffective campaign traffic from your test audience [TEST 2]

2008-11-14T08:19:00.000-08:00

We've recently conducted another multivariate/multivariable test on a key landing page we have on the website. We were looking to optimise this product page and then use this optimised page designs to test against various landing page designs in the future. This test also gave us an indication of what sort of improvements we could expect by testing large changes in page layout within the existing product page template.
As usual we looked to increase throughput of the product page by 5% and improve out visitor to application ratio by 3% (this is always the target for each of our tests).
So we ran a multivariate test on the one of our product pages testing the following elements:
· Product description title
· Bullet point heading
· Bullet point wording
· Apply online button copy
The image below shows on the left the winning, optimised page after testing and on the right the default, existing page design.

We ran the test continuously for 34 days. For the last week of the test we had a situation where we had 3 page designs all showing similar improvements in conversion. As the difference between the 3 were minor it was decided that rather than wait another month or two for one combination reach a 95% significance we would end the product page test prematurely. The winning combination had achieved a 57.42% confidence rate.
It should be noted that campaign activity has significantly impacted on the results for the default design in this test. The issue was that the majority of the 383k visits from campaign activity close the browser instantly (we looked at samples and on one site a large window displayed our product page over the homepage for a mobile phone site, preventing the visitor from seeing the phone site offers). When the browser was closed quickly (before the test variant content was served) then testing tool records the session as 'Default'. The by-product of this is conversion of the Default looks worse that it actually is.
Anyway, the winning page design resulted in a 3.56% conversion rate of visitors submitting an application. Due to issues with campaign activity (discussed later), the testing report console showed that the default design converted at a 0.95% rate but after some further analysis (excluding campaign activity) we believe a fairer value to be around 2.68%. This is an improvement of 32% compared to the default design.
So our findings were:
Buttons
A small red button was the best performing in the test and should be used as the default apply button throughout the site.
Key features
The two best performing body copy designs both highlight headline key features of the product above the full detailed bullet points.
Page headings
Page headings are one of the most important elements the conversion of a page and should not be changed without testing.
Campaign activity
From now on we will always exclude all Campaign activity from all future tests.

SEO and MVT - a marriage made in hell?

2008-10-24T12:09:00.001-07:00

There seems to be a bit of noise surrounding the impact of multivariate testing activity (MVT) upon Search Engine Optimisation (SEO). Basically the idea runs that if your rendering test variant content on a page during a test that doesn't always contain the keywords that promote or sustain your websites search rankings then the impact can be a negative effect on your SEO activity.

It seems it depends entirely upon your choice of test tool and experiment implementation methodology. With the exception of SiteSpect most MVT testing tools employ JavaScript in some means to render the test on the page which means that search engines robots and crawlers don't get to index the test content, they fall through into your default content. Apparently the good people at Google et al want their indexers to be part of the test.

From our own experience of conducting MVT, which incidentally involves using a tool that employs the JavaScript method for content rendering, we've understandably had a degree of pressure from our SEO colleagues to minimise impact on their area during the course of the test. So we've arrived at a process that helps everyone, more or less. During the early part of our MVT tests we exclude all our search traffic, both paid and unpaid. We do this by looking for specific URL parameters that identify search traffic during the test page load event. If the visitor is classed as search traffic they fall through the test and are presented with the default (non-test) page content, as are the search engine indexing agents. We monitor the test variant performances and after a week or so (given enough traffic) we cull out the variants that perform in an adversely negative way (see early post regarding this culling procedure). Then, when we have a test situation that in theory is performing in a wholly positive way, when conversions are up over the original default, at that point we stop excluding the search traffic and bring them back into the test audience. This way we get more traffic volume to feed the test and the SEO benefits from an optimised content.

I think the concept we should always be focusing on overall in regards to SEO and MVT is that if during testing you can impact on SEO rankings in the short-term, in the long-term the site will benefit from the optimised content and an uplift in conversion overall.

UPDATE: 2nd Augst 2011 ~ Here's a useful statement from Google on the use of Google Optimizer and the affect on the SEO ranking

Cover story - what magazines can teach us

2008-10-20T11:57:00.000-07:00

So what do magazines like Heat, OK, Glamour and the National Enquirer have to do with Optimization and multivariate testing?

Well as far as those glossies are concerned and Web Optimization we're being told as marketeers to adopt the same design model these magazines use to get countless people to buy them week after week. These publications obviously have to compete in a fierce market where the cover is the singular entity that can make the difference between a huge circulation or being just another magazine on the shelf. When constructing your sales message or call to action we should endevour to recreate that strong headline message or cover story that makes a customer buy, purchase, convert. According to industry analysts 80% of the purchasing decision is based upon what's on the front cover of a magazine. These guys are the experts, they know the value of catching the eye and conveying the message in a single shot, so surely our web pages should aspire to do the same whenever we seek to acheive the biggest bang for our buck?

Now in terms of magazines and multivariate testing the National Enquirer, a tabloid magazine, famous for running, amongst other things, items on alien abduction and conspiracy theories, has a massive circulation of around 2.5 million readers in the US.
Faced with a downturn in sales in recent times the publishers decided to enlist a company to undertake an MVT experiment. This company developed several versions of the same magazine cover and distributed them in a limited number of outlets, available only in a couple of states and then monitored which one had the biggest sales. Despite a large cost involved with running the different covers the Enquirer reported a 20% increase in sales as a direct result of the experiment. I think this illustrates the reach and scope of multivariate testing these days showing that's not just confined to the web.

Quick wins for web optimization

2008-10-14T11:53:00.000-07:00

If I had to summarise the key points that I have heard again and again from experienced MVT testers, be it web based or published literature I would say the following seem to be touted regularly as quick win changes you can make on your site to gain conversion uplift. From personal experience some work and some just dont. As you will see we're yet to put them all into practice:

1. If you have a call to action - make it BIG.

We have an MVT test running currently that has around 8 different variants of apply buttons. Some are big and some are small in a variety of colours. Currently a large version of a newly designed button is beating the rest of the pack with a 18% uplift in conversion (It's orange by the way). Optimization experts generally recommend making your button big and red to draw as much attention to it as possible. We had one such variant and it performed so badly that we have culled it from the test. At least we've tested it though and put that theory to bed.

2. Have more than one call to action on page.

This should be a no brainer really, but when your running a test you need to be able to track people clicking on such buttons during the course of the test. If you have multiple buttons this can be tricky and from my own experience there are limitations as to how many 'actions' you'll want to feasibly track during the testing period.

3. Have a testimonial of your product on the page.

We havn't been able to test the use of this one yet in an MVT situation yet but we have tests scheduled that will exhaustively test this theory in the future. We have however A/B tested the use of testimonials within the customer journey and there was no real uplift to speak of.

4. Use photos of people in your page.

We've had an unofficial embargo on images on our site until recently due to a perceived impact on SEO rankings. This embargo has lifted and we can now start to use images with avengence once more. We've got a Homepage MVT test scheduled which will pit images against text based variants and I'll let you know how that pans out.

5. Simplify your copy.

If you have loads of copy it's daunting, off-putting and boring. Find ways to either reduce your copy or orgainise it in such a way as to make it look less overwhelming. Try tabbed content, or use expand and collapse containers on a page. The key winning element of our first MVT test was to drastically reduce and simplify the copy on the page by moving some to a pop-up window. We've also actually been aided by recently having to comply with the Plain English campaign. Initially it's a headache getting your content to comply with their rules, but once you've done it it can make a world of difference to the way a message scans to the end user.

I'll personally try to continue working these concepts into each of the tests we
undertake as we move forward. Where they don't work I'll hopefully learn from that and carry that forward to the next test. The key thing is to evolve your experiments as you go, bearing in mind what works for one person may not work for the other. Such simple changes can however be useful if there's a tangible pressure to gain results and an uplift in sales.

Go LARGE

2008-10-07T11:51:00.000-07:00

If there's a big learning we've made since starting multivariate testing on our website it's that if you're going to test different content, you're alternative content needs to be as radically different as possible to your existing content to get anything out of a test.
We currently have a MVT test running on a product page on the site. It has 4 MaxyBoxes* , each MaxyBox has around 3-6 alternative variants being served within it. The test has been going around 3 weeks now and for a couple of reasons which I wont go into now, we've had to restart the test. However, the alternative content is not too different from the existing content. We basically have text that has subtle changes in it from the original. Essentially the same message just re-jigged to see if the way we word a benefit or feature of a product has any uplift in conversion. Looking at the reports for this test the test variants performances are fairly uniform with no clear winner emerging from the pack and our mistake of keeping the changes between each variant subtle is clear to see. I think from this test alone we can safely say that subtle changes just dont work in an MVT test. You would need months and months of relatively high traffic volumes to get a result, or what we call statistical significance.
The reason this current test ended up with the 'softer', or subtle variants is that this test was planned when we thought that MVT testing could yield results by making the small changes that would maybe add up to a bigger result.
So the big message is that if you really want to get results quickly from your tests you have to go large. You have to go with the bolder ideas that represent a shift from the existing content.
*MaxyBoxes are areas of the page, usually Divs that are tagged for serving multivariate test variant content from our testing tool Maxymiser.

Culling your test variants

2008-09-21T11:43:00.000-07:00

As an Optimization team, we are new to the whole multivariate testing business. We are more than aware that we dont always do things by the book. So when we undertake a multivariate test we dont always adhere to the basic principles of testing.

The biggest rule we tend to ignore is 'Do not tamper with your test'. The trouble is we always have to have an eye on the bottom line. So we are constantly asking ourselves whether what we're undertaking in terms of testing is not impacting upon our sales in a negative way? Are we actually reducing the conversion rate on the website?

My colleague generally monitors what's going on downstream during a test and looks at the basic application submittion rates for our products. If he notices a downturn in conversion rate during the course of a test we get a bit nervous.

Thankfully our multivariate testing tool, Maxymiser allows us to look at how individual variants are performing. If after a period (usually around one week into a test in our case) we start to see a downturn we'll start to examine closely which variants we can 'cull' from the test.
Once we highlight the under-performers we then downweight* them out of the test entirely. This is beneficial for two reasons:

1. You minimise negative impact on conversion and sales.

2. You reduce the number of page combinations in the test.

The lower the number of page combinations the quicker your test period. This is great for us because of the second rule of testing that we frequently ignore 'Allocate enough time for testing'. Basically speaking we run tests for a far shorter period than is recommended.
Most tests, given enough visitor traffic to your site run anywhere from 4 to 10 weeks, or even longer. We tend to have ran tests from 2 to 6 weeks. Our excuse for this is that there is so much other activity going on on the website at any given time by other people that we have a very narrow window in which to test and get a result.

Another key thing when planning your MVT test is knowing how much traffic you get to your site and whether you've got enough traffic to run all your page combinations and see an outright winner at the end of your test. So far we've been reasonably lucky in that we've had enough traffic to run the tests for a relatively short period and still acheive a winner.

Obviously ignoring key testing rules and principles is not recommended. But if lke our Optimzation team, you're stuck between a rock and a hard place, and there's a certain need to get some kind of testing done our early experiences have shown that you can bend the rules to get some kind of learning or outcome in a short space of time.
* In multivariate testing, each variant is usually allocated a weighting. For example, if you give a variant a 50 weighting in the test console it will be served 50% of the time, while the default content is served the other 50% of the time.

Our 1st multivariate test - Killer Questions [TEST 1]

2008-09-09T11:34:00.000-07:00

We have an application form that is used for various products, in this case we focused on the one used for an online current account.
The first page of this app form is what we call the 'Killer Questions' page. This is where you set the scene for the applicant in terms of the criteria they have to meet to continue with their application and what bits of information they're going to need to have to hand to complete the application process.
Typically we could expect around a 35-40% drop-off rate for this page. This could of course be because people simply dont meet the criteria and depart the app process. Alternatively it could be that the page design, which was by nature quite weighty in terms of the volume of info needed to be conveyed to the user at that time. Based upon the hypothesis that the content may be at fault we decided this was the best candidate for hosting our first multivariate test. Here we would test different content on the killer questions page to see if an uplift in number of people commencing the application could be acheived. We looked to gain a 5% uplift overall.
The Killer Questions page looked rather like this.
The 2 boxes are 'MaxyBoxes'.
To effectively carry out multivariate testing on your website you're going to need a tool to host & serve the alternative page content you want to test and additionally a means of reporting the progress and outcome of your test. We opted for Maxymiser to acheive all this. The MaxyBoxes you see there represent the areas of the page that will have variant content served to the visitor.
So basically in this particular test we wanted to see what would happen when we served different security images in the top-right of the page and different page content or layout in the second MaxyBox on the page (the larger box).
We split the traffic 50-50. Half being served the default content, the other half a combination of new test variants we'd come up with. Coming up with alternative content is a whole new learning experience in itself as no one in our Optimization team is a copywriter or creative type but I'll go into this in more detail in a later post.
After several weeks of multivariate testing we ended up with a winner...

You may notice that this has a different security image to the original, has simplified page copy, all the heavy and intimidating legal wording was removed to a pop-up link.
This page design resulted in 8% more people passing through the killer questions page compared to the default design. It also resulted in 5% more people going on to submit the application. When accept rate* was taken into account the new page design resulted in a 20% increase in productivity over the default design.
This test, although it's our first ever, will probably turn out to be our most important. This is because it was at the very last point in the visitors journey that we can influence whether they buy into the product or not. It's make or break time when they're on the doorstep of the application form, the point at which they choose to purchase or run away.
* note: You can apply for current account - but you might not necessarily be accepted as a customer, hence an accept rate.

Optimisation Maverick Blog

Bigger. Stronger, faster, better - best test ever

Popular posts

Maxymiser - Multiple KPIs

Google Analytics - Map Overlay

Tracking uplift post ab testing

1st Direct Bank throw it out there

Testing During a Traffic Spike revisited

The law of diminishing returns

net.finance 2011

Calculating Uplift

Above the fold - The Google Browser Size Tool

The conception of Short Wave Testing

No need to shout about it

An Offline Call To Action

The Gutenberg Rule

Good old button testing

Google Optimizer Follow-Up experiments

First off - What is a follow-up experiment?

Google says:

"

And why should I run a follow-up experiment?

But what happens when a follow-up experiment delivers contradictory results?

Give a Huq?

Google Trends for MVT Terminology

Amazon cross-selling

Online polls

Is Fatwire any good?

Google Optimizer - Landing Page A/B Testing

An unbiased review of Google Website Optimizer

Culling multivariate test variants in a Maxymiser test

Optimisation providers compared

Riding the tsunami - optimising an online marketing campaign

Should You Avoid Testing During a Traffic Spike?

Optimization of a product page

How to increase average spend - The 97p Shop

Rate our page designs

Eye Tracking

Twitter tools (Part Two) - revised

Behind the paywall - Can you really charge for News content?

Page Combination Removal Feature in Maxymiser

My Twitter toolset - a few of the most original & useful tools out there

Striking while the iron is hot

Omniture Summit 09 London - Key Takeaways

The optimisation of a landing page test (part one)

What is Statistical Significance?

The HiPPO factor

The reptilian brain & your inner buy button

The power of the testimonial [TEST 5]

Seasonal shift

Emphasis of key words [TEST 3]

Exclude ineffective campaign traffic from your test audience [TEST 2]

SEO and MVT - a marriage made in hell?

Cover story - what magazines can teach us

Quick wins for web optimization

Go LARGE

Culling your test variants

Our 1st multivariate test - Killer Questions [TEST 1]