Friday, November 2, 2012

multivariate testing 101

One step up from A/B testing is multivariate testing (also called multi-variable testing).

A/B tests are straightforward: try two scenarios, determine the better of the two content variations.
Multivariate testing can theoretically test an infinite number, though you need to keep in mind the need to maintain a statistically significant number of results to determine a performance difference between scenarios.  It's best carried out via a dynamically generated website: visitors are served proportionate but random variations of content to test a hypothesis.

Smashing Magazine artfully explains: "There are two principal approaches used to achieve multivariate testing on websites. One being Page Tagging; a process where the website creator inserts Javascript into the site to inject content variants and monitor visitor response. The second principal approach used does not require page tagging. By establishing a DNS-proxy or hosting within a website's own datacenter, it is possible to intercept and process all web traffic to and from the site undergoing testing, insert variants and monitor visitor response. In this case, all logic sits server rather than browser-side and after initial DNS changes are made, no further technical involvement is required from the website point of view."
To read more, check out: