The optimization of a landing page test

We have an AB test running on a product page and landing page simultaneously to see which location would convert better. The background to this is that we once used landing pages a lot, and then the industry as a whole (banking) decided to move over to dropping people straight onto a product page. It became the established wisdom that product pages were better than landing pages for optimising your traffic. This test sought to revisit this hypothesis. We decided to use a specific traffic source, identify them on page load, split the traffic 50/50, 50% remaining on the product page, 50% redirected to a bespoke landing page.

As well as testing the different locations we wanted to test a few other concepts of optimisation opportunities we'd heard about:

1. Do bespoke welcome messages help in converting users to apply for the product?
2. Do the presence of primary navigation in landing pages actually help visitor conversion?
3. Does a 'band wagon' message help uplift; i.e., "join the thousands of other visitors in choosing this product" etc.

When we started the test we soon found out that the traffic source chosen to use as the test audience was a lot smaller than anticipated. Within a few days we had to adjust the 50/50 weighting and send 100% of the traffic to the landing page with a view to retesting whether product pages work better than landing pages at a future date. If we'd left it as it was we would still have been running the test 2 years later, so low was this traffic to the page from that specific traffic source.

As the test is still running we're still unable to answer question 1. 'Do bespoke welcome messages help in converting users to apply for the product?'. However, within the first 2 weeks we were able to answer questions 2 & 3. The primary navigation in the landing page performed very badly, allowing users to leak out from the test page and not convert. The welcome message using the band wagon theme also failed to convince any users to apply and also performed very badly. Both the presence of the primary navigation and the band wagon message are shown in the test page combination below (as taken from our Maxymiser test console). Upon conclusion of this test we will re-run the winning test variants on the product page to see again which page gets the biggest uplift in conversion, this will be covered by Part two of this post.