Multivariate Testing 101: A Scientific Method Of Optimizing Design - Smashing Magazine

Do’s And Don’ts

I have observed hundreds of multivariate tests, and I have seen many people make the same mistakes. Here is some practical advice, direct from my experience.

Don’ts

• Don’t include a lot of sections in the test.
  Every section you add effectively doubles the number of combinations to test. For example, if you’re testing a headline and image, then there are a total of four combinations (2 × 2). If you add a button to the test, there are suddenly eight combinations to test (2 × 2 × 2). The more combinations, the more traffic you’ll need to get significant results.

Do’s

• Do preview all combinations.
  In multivariate testing, variations of a section (image, headline, button, etc.) are combined to create page variations. One of the combinations might be odd-looking or, worse, illogical or incompatible. For example, one combination might put together a headline that says “$15 off” and a button that says “Free subscription.” Those two messages are incompatible. Detect and remove incompatibilities at the preview stage.
• Do decide which sections are most worthy of inclusion in the test.
  In a multivariate test, not all sections will have an equal impact on the conversion rate. For example, if you include a headline, a call-to-action button and a footer, you might come to realize that footer variations have little impact, and that headline and call-to-action variations produce winning combinations. You get a powerful section-specific report. Below is a sample report from Visual Website Optimizer. Notice how the button has more impact (91%) than the headline (65%):

  

• Do estimate the traffic needed for significant results.
  Before testing, get a clear idea of how much traffic you’ll need in order to get statistically significant results. I’ve seen people add tens of sections to a page that gets just 100 visitors per day. Significant results from such a test would take months to accumulate. I suggest using a calculator, such as this A/B split and multivariate testing duration calculator, to estimate how much traffic your test will require. If it’s more than what’s acceptable, reduce some sections.

via smashingmagazine.com

View/comment on the original post at blog @cankoklu