As usual we looked to increase throughput of the product page by 5% and improve out visitor to application ratio by 3% (this is always the target for each of our tests).
So we ran a multivariate test on the one of our product pages testing the following elements:
· Product description title
· Bullet point heading
· Bullet point wording
· Apply online button copy
The image below shows on the left the winning, optimised page after testing and on the right the default, existing page design.
We ran the test continuously for 34 days. For the last week of the test we had a situation where we had 3 page designs all showing similar improvements in conversion. As the difference between the 3 were minor it was decided that rather than wait another month or two for one combination reach a 95% significance we would end the product page test prematurely. The winning combination had achieved a 57.42% confidence rate.
It should be noted that campaign activity has significantly impacted on the results for the default design in this test. The issue was that the majority of the 383k visits from campaign activity close the browser instantly (we looked at samples and on one site a large window displayed our product page over the homepage for a mobile phone site, preventing the visitor from seeing the phone site offers). When the browser was closed quickly (before the test variant content was served) then testing tool records the session as 'Default'. The by-product of this is conversion of the Default looks worse that it actually is.
Anyway, the winning page design resulted in a 3.56% conversion rate of visitors submitting an application. Due to issues with campaign activity (discussed later), the testing report console showed that the default design converted at a 0.95% rate but after some further analysis (excluding campaign activity) we believe a fairer value to be around 2.68%. This is an improvement of 32% compared to the default design.
So our findings were:
Buttons
A small red button was the best performing in the test and should be used as the default apply button throughout the site.
Key features
The two best performing body copy designs both highlight headline key features of the product above the full detailed bullet points.
Page headings
Page headings are one of the most important elements the conversion of a page and should not be changed without testing.
Campaign activity
From now on we will always exclude all Campaign activity from all future tests.
