Been running a multivariate test on a current account product page again. This test has been running for over 3 weeks and not performing too well it could be said. Below is the readout from our Maxymiser console for this test.
So we sat down and thought how we might turn this under-performance around. I had been hoping to try and use a customer testimonial in a test for a while so I suggested that this might be the time & place to do just that. On the product page we had a test element that displays a strapline immediatly under the product title. So we took the worst performing variant and changed it to read a short & snappy customer testimonial instead (see before & after below).
Now we re-commenced the test. We didn't go for a complete restart, we just take the test results from the new start date of when we added the testimonial variant into the mix. And a few days in and the difference is incredible, the top performing page combination by a wide margin contains our new testimonial, this is identified as V_013F on the report below. So just goes to show, the test is still running at time of writing but we'll see if this performance degrades over time and whether this new test variant ends up in the winning page combo. At the moment though a 26% uplift in conversion shouldn't be sniffed at, and it's another thing we can tick-off as being tested.
So we sat down and thought how we might turn this under-performance around. I had been hoping to try and use a customer testimonial in a test for a while so I suggested that this might be the time & place to do just that. On the product page we had a test element that displays a strapline immediatly under the product title. So we took the worst performing variant and changed it to read a short & snappy customer testimonial instead (see before & after below).
Now we re-commenced the test. We didn't go for a complete restart, we just take the test results from the new start date of when we added the testimonial variant into the mix. And a few days in and the difference is incredible, the top performing page combination by a wide margin contains our new testimonial, this is identified as V_013F on the report below. So just goes to show, the test is still running at time of writing but we'll see if this performance degrades over time and whether this new test variant ends up in the winning page combo. At the moment though a 26% uplift in conversion shouldn't be sniffed at, and it's another thing we can tick-off as being tested.
UPDATE: 24th March 2009
I thought I'd better update this post today to take account of what's happened to this test. So the test has been running now for 63 days. Below is a screenshot of the report console for today. As you can see the V_013F variant (which is the testimonial variant) is placed in the top 3 winning page combinations. On the embedded graph I've pinpointed where we introduced this customer testimonial variant into the test. It seems that after a brief lift the test settles back down into a level pattern acheiving a roundabout 11.5% uplift in conversion, no longer retaining the 26% uplift first attained. The Maxymiser 'Chance to beat' metric also hovers around the 46% level for the best page combination (P10) so the test seems in a state of flux and not really progressing towards statistical significance. I should also point out that we've culled a few negatively performing variants from the test.
