An Introduction to Multivariate Testing

Why use Multivariate testing?

Let your visitors tell you what works best!

There is no formula for the perfect website so you need to test what combinations of page elements and design are most effective. Continual testing and optimisation is fundamentally important for site owners looking to improve usability, conversion ratios, and ultimately the return on investment from their website. Small changes can have an enormous tangible impact and multivariate tests help identify the most effective changes.

• ‘A-B Split testing’ – compare two entirely different versions of a page, (useful if low traffic, or for quicker results).
• ‘Multivariate testing’ – simultaneously measure the performance of several content variations on a page.

Analysis of the resulting traffic and user behaviour data from these tests provides a visible measure on the impact of a change, or combination of changes, to a page’s layout, design, and function.

Running these tests enables fine-tuning of landing pages, product pages, the buying process, ad copy, e-mails and so on.

How does it work - strategy?

1. Prepare multiple versions of the same page with differences in content, design, and / or layout.
2. Run the variations on the live site for a set duration with the site’s analytics service measuring how visitors respond.
3. The traffic data is then studied to assess which combination of changes was most successful.

In order to get the most out of multivariate testing it is important to properly plan and prepare. There are also some widely accepted ‘best practice’ guidelines:

• Test pages which receive sufficient traffic to generate meaningful results within your desired timeframe.
• For the testing to remain manageable it’s important to limit the number of variations for each test.
• The variations should also not be too subtle, but discernable changes that will impact upon the site user.
• Consider a phased approach, with continual testing.
• Ensure every combination is still acceptable within your sites overall design, usability, and accessibility standards.
• Measure range of conversion goals not just standard calls-to-action, e.g. time on page (bounce rates).
• Don’t interpret the results in isolation but within the context of conclusions from wider testing.

Industry Examples

• E-marketing specialists at Unical, (part of Rural Cellular Corp.) achieved an increase in click through of over 50% on their optimised banner advert identified through rigorous multivariate testing. – [Marketing Sherpa Case Study, Apr-2008]
• In a recent 4-way split test run by online clothing store Boden, on their homepage, results showed a purchase conversion ratio increase of 2.4%, and a vast 52% increase in e-mail sign-ups – [e-consultancy conversion keynote, Nov-2007]

Success Story – Pukka Herbs

Pukka Herbs have launched a full e-commerce website powered by WORKSsitebuilder.

Pukka promote Ayurvedic living and sell herbal teas and remedies that are organic, fair trade, sustainable, traceable, carbon neutral, and vegetarian. Because of the quality of their products, and the ethical policies the company rigorously follows, there are price competitiveness limitations. It was therefore vitally important to optimise the design of the product pages to make the unique selling points clear, establish trust and credibility, and maximise conversions.

The aim - refine the product pages to demonstrate quality, justify price, increase trust - to maximise conversions.
The test – variables included the layout, visibility of key information (customer service, terms, policies), product image, add-to-basket button, and the ‘Pukka Pledge’ (USPs)...
The results – the testing was run for 2 weeks collating data for 23,400 views of the specific product page, from which the winning combination of variables was identified. This achieved an increase in conversion ratio of 2.6%, which in revenue terms equates to a very significant increase in sales and a large return on a minimal investment.

Tools and Services

For further research the following specific multivariate / split testing tools and services may be of interest:

• Adlucent
• Google - ‘Website Optimizer’
• Memetrics
• Omniture – ‘Offermatica’
• Optimost
• SiteSpect
• TaguchiNow
• Vertster