Culling multivariate test variants in a Maxymiser test

I've covered the topic of culling before on this blog here. Now I'll go through my method of identifying test variants to cull from a running MVT test in Maxymiser where you have multiple actions.

1. Below is a screengrab of an MVT test report in Maxymiser after a test has run for a week. On the left side are the bottom ranking variants for a 'click apply button' action and on the right are the bottom ranking variants for a 'submit application' action.

2. The conversion rate uplift is negative for these variants and are not adding anything to the test overall and so need to be removed or 'culled'.

3. Indentify page combinations (see Page ID field above) that are both negative in uplift and appear in the bottom ranking across both actions then select the 'remove page' option within the console.

4.Looking at the test report before and after the cull identifies the immediate effect on the 'chance to beat all' metric within the test.

The chance to beat all value moves from 27.86 % for the lead variant to 28.11%, this uplift shift also cascades downwards through the other variants left in the test.

This culling exercise would then be repeated at periodic intervals for the remainder of the test.

note: Caution should be taken when removing variants from the test. The number of generations and actions should be taken into consideration, whilst over-culling a test can bring it to an early and unproductive conclusion.