If there's a big learning we've made since starting multivariate testing on our website it's that if you're going to test different content, you're alternative content needs to be as radically different as possible to your existing content to get anything out of a test.
We currently have a MVT test running on a product page on the site. It has 4 MaxyBoxes* , each MaxyBox has around 3-6 alternative variants being served within it. The test has been going around 3 weeks now and for a couple of reasons which I wont go into now, we've had to restart the test. However, the alternative content is not too different from the existing content. We basically have text that has subtle changes in it from the original. Essentially the same message just re-jigged to see if the way we word a benefit or feature of a product has any uplift in conversion. Looking at the reports for this test the test variants performances are fairly uniform with no clear winner emerging from the pack and our mistake of keeping the changes between each variant subtle is clear to see. I think from this test alone we can safely say that subtle changes just dont work in an MVT test. You would need months and months of relatively high traffic volumes to get a result, or what we call statistical significance.
The reason this current test ended up with the 'softer', or subtle variants is that this test was planned when we thought that MVT testing could yield results by making the small changes that would maybe add up to a bigger result.
So the big message is that if you really want to get results quickly from your tests you have to go large. You have to go with the bolder ideas that represent a shift from the existing content.
*MaxyBoxes are areas of the page, usually Divs that are tagged for serving multivariate test variant content from our testing tool Maxymiser.
We currently have a MVT test running on a product page on the site. It has 4 MaxyBoxes* , each MaxyBox has around 3-6 alternative variants being served within it. The test has been going around 3 weeks now and for a couple of reasons which I wont go into now, we've had to restart the test. However, the alternative content is not too different from the existing content. We basically have text that has subtle changes in it from the original. Essentially the same message just re-jigged to see if the way we word a benefit or feature of a product has any uplift in conversion. Looking at the reports for this test the test variants performances are fairly uniform with no clear winner emerging from the pack and our mistake of keeping the changes between each variant subtle is clear to see. I think from this test alone we can safely say that subtle changes just dont work in an MVT test. You would need months and months of relatively high traffic volumes to get a result, or what we call statistical significance.
The reason this current test ended up with the 'softer', or subtle variants is that this test was planned when we thought that MVT testing could yield results by making the small changes that would maybe add up to a bigger result.
So the big message is that if you really want to get results quickly from your tests you have to go large. You have to go with the bolder ideas that represent a shift from the existing content.
*MaxyBoxes are areas of the page, usually Divs that are tagged for serving multivariate test variant content from our testing tool Maxymiser.
