Maxymisers Visual Campaign Builder (VCB) test tool

Updated: 19th August 2014.

If you've been using Maxymiser for your MVT testing in the past you may already appreciate this very competent testing tool. Recently the Maxymiser team have developed a new feature that I've been very keen to get my hands on, their Visual Campaign Builder, aka, VCB.

What is it?

Well essentially it's a way to build and launch your own tests very quickly using the MaxTest solution (and leverage the MaxSegment solution too if you have that) using a WYSIWYG on page wizard. You navigate to the test page, create a new test campaign, create new variants, drag and drop assets, change copy and content, assign an existing Action for action tracking and then publish it.

I've trailed two tests to date, one where I'm introducing a new banner to a page and another where I've embedded a YouTube video. Reporting is exactly the same as conventional testing whereby you log into the Maxymiser report console.

Where is it?

Log into the Maxymiser console and navigate to the Campaigns page, you will see a button for the VCB in top right side...

Once it's enabled for your site, when you click the VCB link it should the option to create or edit an existing campaign...

And how do I use it?

Well you should of course get the Maxymiser team to talk you through the first couple of set ups for VCB testing but here's the easy process in summary...

1. Create Campaign.
2. Give it a name.
3. Choose Audience. Here you can target visitors by device, browser, geography, behavior or just choose to test with all visitors.
4. Select the Page(s) you wish to test on by page URL. Cut & Paste the URL into the the URL box provided and add to campaign. At this point you may want to 'Include any query string parameters' or 'Include both HTTP and HTTPS protocols'
5. The next step is the Content creation step for your test. Here you can either add or remove content or create alternative experiences for your test variants. Tip: This is the tricky part of the whole endeavor and I strongly recommend you test how your variants are looking in all browsers and devices you wish to test against.
6. Next step is to select the Actions. Give it a primary action from your drop-down. If you don't have an appropriate action to track on your test you will need to speak to Maxymiser. This can be a stumbling block for getting a test straight out there but if you've got a suite of test actions you've used before there's potential to re-purpose those for a VCB test as long as it doesn't conflict with any other tests you may be running at the same time.
7. Review & Publish. From here you're more or less going through the normal test publishing process, for example, adding change comments and going via the publishing console to send your test live. Tip: You may get a warning about new pages being mapped and might take an empty publish first before you can publish your VCB test.

That is it in it's simplest form.

Good luck and happy testing : )

Maxymiser - Multiple KPIs

A natty little feature of the Maxymiser reporting interface is the ability to report on multiple KPIs within the same report. Below is a screen grab of a conversion report which reports on both the 'Application submit rate' referred to as KPI 1, whilst also showing the performance of the secondary KPI 2 of 'Click to Apply rate' for the same test combinations. In the past you would have to flick between separate reports for each KPI, whereas this is obviously no longer the case. 

To enable this feature you simply select multiple KPI or Actions in the report filter drop-down menu within the Maxymiser console. See illustration below.

Charlotte 29th Oct 2011

net.finance 2011

Last week I spoke on the merits of creating a testing culture at the financial services web conference Net.Finance in Chicago on behalf of Maxymiser. See their blog here for details Max Blog. It was a great trip and an extremely exhausting week due to my own crazy schedule but I absolutely loved every second of it.

Key take-aways from the conference? 

• Multivariate testing is yet to really take off in the states, especially in the financial sector. It's ripe for massive growth on an unprecedented scale.
• US marketeers are really excited about 'Mobile Payments' and see it as a potential bank killer.The question remains which big names will forge alliances to make it finally happen.
• The buzz word is 'mobile' and has been for a while, but everyone, including the main players are waiting to see who goes really big on mobile first and will then quickly follow suit. The US web sector are scared and have been stung before by 'novelties' that have failed to bear fruit, so there's a vast amount of tangible caution in the average US eCommerce department; dollars are quite rightly spent sparingly and wisely. 
• There are a lot of companies offering what is perceived as real 'added value' but is really not worth the huge initial $ outlay when you start to dig deep on their technical claims. Qualitative testing and research continues to commit multiple crimes in the name of informed user feedback and fall vastly short of continuous multivariate testing by a country mile. 

Would be interesting to see if anything changes in the US market in the next 12 months and especially to see if mobile becomes the hunting ground of the web marketeer as predicted.

Use of Awards endorsement in web pages

In the past we've included any awards the company or product have attained based upon the assumption that they can only be a positive thing to have on the web page. Well I finally got around to measuring exactly what effect the use of an award logo had when added to a page. Basically I had concluded an earlier optimisation exercise on a landing page using Maxymiser and tweaked the winning page combination to show an award logo instead of the image of two people. I let the adapted test run for 3 weeks and the winning combination fell from a 9.96% uplift to minus 4.05% thus showing that the use of award imagery in this context was ineffective. Click on the image below to enlarge for a summary of what happened.

Culling multivariate test variants in a Maxymiser test

I've covered the topic of culling before on this blog here. Now I'll go through my method of identifying test variants to cull from a running MVT test in Maxymiser where you have multiple actions.

1. Below is a screengrab of an MVT test report in Maxymiser after a test has run for a week. On the left side are the bottom ranking variants for a 'click apply button' action and on the right are the bottom ranking variants for a 'submit application' action.

2. The conversion rate uplift is negative for these variants and are not adding anything to the test overall and so need to be removed or 'culled'.

3. Indentify page combinations (see Page ID field above) that are both negative in uplift and appear in the bottom ranking across both actions then select the 'remove page' option within the console.

4.Looking at the test report before and after the cull identifies the immediate effect on the 'chance to beat all' metric within the test.

The chance to beat all value moves from 27.86 % for the lead variant to 28.11%, this uplift shift also cascades downwards through the other variants left in the test.

This culling exercise would then be repeated at periodic intervals for the remainder of the test.

note: Caution should be taken when removing variants from the test. The number of generations and actions should be taken into consideration, whilst over-culling a test can bring it to an early and unproductive conclusion.

Page Combination Removal Feature in Maxymiser

We use Maxymiser as our Multivariate/AB testing tool of choice. Maxymiser allows you to see how individual test variants are performing in it's console. In a previous post about culling variants I talked about how we like to actively remove under-performing variants from our tests. Well this has been a contentious issue, whether or not it's the right thing to do in a test scenario (see post for full discussion). However Maxymiser have recently added a new feature to their test console that allows you to remove under-performing page combinations* from your test. Doing this also allows you to immediately and clearly see the impact of performing such an action and its immediate effect on the remaining page combinations.

Below is a screenshot of our Maxymiser console displaying an active test before we remove an underperforming page combination. P8 page combination is highlighted as an under-performing page combination. It has an uplift of minus 1.76% and a 'Chance to Beat All' value of 25.23%.



Now after we exclude the P8 page combination from the test you can see what the results look like. The lead page combination P2 leaps from a 39% 'Chance to Beat All' to 51%, hence speeding up your test. The overall uplift value also moves from 3.19% to 3.59%.



So the introduction of the page exclusion feature actually allows you to experiment more with 'What If' scenarios. We like to think that this enhancement was made to Maxymiser as a direct response to our need but it's probably not the case!


* A page combination is a collection of multivariate test variations

What is Statistical Significance?

I've sort of overlooked this topic since establishing this blog but for subject completeness shall we say, I think I should now mention the role of statistical significance in optimisation testing.

One of the biggest headaches to running an AB test or Multivariate test on your website is knowing when your test is complete, or heading towards conclusion at least. Essentially how do you determine signal from noise?

Many 3rd party tools give you the metrics to determine a tests conclusiveness, for example the Maxymiser testing tool displays a 'Chance to beat all' metric for each page combination or test variant within your test.
But more importantly, what underpins these tests is the concept of statistical significance. Essentially a test result is deemed significant if it is unlikely to have occurred through pure chance. A statistically significant difference means that there is statistical evidence that there is indeed a difference.

Establishing statistical significance between two sets of results allows us to be confident that we have results that can be relied upon.

As an example, you have an AB test that has two different page designs. Analysing the data shows there are two results:

Page 1 - 1,529 generations with 118 responses or actions - giving a conversion rate of 7.72%.
Page 2 - 1,434 generations with 106 responses or actions - giving a conversion rate of 7.39%.

Looking at the two results which do you think is the better? Is page 1 better because it as a higher conversion rate that page 2? Using statistics and firing those 2 results through a basic Statistical Significance calculator (I'm using this one Google's Optimizer test duration calculator) tells us that the two results are 0.335218 standard deviations apart and are therefore not statistically significant. This suggests that it is highly likely that it is noise causing the difference in conversion rates, so plough on with your testing. If a 95% statistical significance is acheived you can safely say the test is onclusive with a clear winner. This is also indicative of a strong signal and gives you a result based upon a wholly statistical basis as opposed to human interpretation.