Optimisation providers compared

We recently reviewed Optimisation providers and their tools as we needed to procure another years contract for such a tool and service. Here's a few advantages and disadvantages that we found with a few of the top names in the business at the moment.

Omniture

Advantages

• Comes with segmentation engine by default
• Has self optimising content feature once test has completed
• No limits to the number of domains you can test on

Disadvantages

• Requires multiple pieces of code on every test page - very IT resource heavy
• Reporting only shows winning design not all results so test within test not possible
• Poor reporting interface
• Technology-wise they seem to have fallen behind the opposition providers.

Maxymiser

Advantages

• Solely undertake optimisation work, not distracted by other functions such as web analytics etc
• Minimal IT resource required as tests require a single line of Javascript at the bottom of the test page. Once implemented no further work is required.
• Full end to end managed service including design resource.
• Product has real time reporting
• Manual control over tests if required,i.e; culling of test variants
• They have an extensive serving infrastructure that can serve winning content until new templates can be developed.

Disadvantages

• Initial quotes were high compared to competition
• Sometimes sluggisn in terms of test turnaround time from past experience

Autonomy

Advantages

• Offers all the functionality you'd expect from a top-end provider
• Full managed service
• Very strong test recommendations and results analysis
• Appears to cope well with more technically challenging tests
• Uses Wave testing approach which means end results are more accurate when implemented live
• No limits to numbers of tests and number of domains covered with contract
• Segmentation engine included
• Minimal IT resource required as tests require a single line of Javascript at the bottom of the test page. Once implemented no further work is required (although complete clarification on this capability was ever obtained from the vendor at the time).

Disadvantages

• Poor reporting interface
• Solution requires code to be put in the header - this may cause issues for templated sites
• 3 month contract required to implement a Pilot
• Perception of a limitation on doing radical page layout testing, more elemental testing favoured.

 
Sitespect

Advantages

• Once Sitespect server implemented then no additional IT resource required
• Unlimited control over what content we wish to test and no limits to the number of tests that can be run (limited only by our resource)
• It can test all content including any online forms
• Comes with a segmentation engine at no additional cost

Disadvantages

• It is not a Managed Service so labour intensive from an eCommerce point of view
• Developer know-how required within our team
• User interface very complicated and requires full training
• Results reporting not very user friendly

Google Optimizer


Advantages

• It's free to use
• Wide customer base and as a result there seems to be a wide knowledge base although forums are not the best means of resolving a critical testing issue in a timely manner!
• Likely to improve over time due to the volume of people using it.
• Integrated reporting with Google analytics, which is good if you like the GA interface.
• Can be either self optimising by automatically removing under-performing variants or done manually, but once variants are out of a test that's it. 
• Can be adapted to do event tracking, ie. onclick events so in theory you dont need a conversion page to undertake a test which is dead handy.

Disadvantages

• Fairly IT resource heavy in its implementation
• Cant easily track across domains

• Obviously not a managed service
• Can only record an action on the test page and not all the way to application submit (hence missing a critcal metric for serious testers).
• No consultancy service from Google so dependant on forums for problem solving 
• Feels like it's still in beta
• Have been some noise about outages and slow reporting in terms of seeing test progress or even confirmation that your test is firing correctly.

 OUTCOME OF TESTING TOOLS EVALUATION

In the end we choose Maxymiser mainly because we'd already worked with them before and knew what to expect more or less, going forward. Also because they had the ability to remove under-performing variants within a test and their reporting console is user friendly compared to the competition. They are keen and quick to improve their tools and service and welcome feedback.

UPDATE:

Since this article was originally posted we have also decided to pursue use of Google Optimizer in parallel to the chosen managed service as it's a free service and I personally have the developer know-how to code the test variants without IT input (ie, it's not for everyone in the testing world).
UPDATE:I've posted my feedback on this in a seperate article here .