App: How to Run a Multivariate Test (MVT Sample Size Calculator)

Two digital marketing and user experience optimization testing methodologies are A/B testing and multivariate testing (MVT). Both approaches aim to improve website performance but differ in complexity and scope. This article will define each method, compare strengths and weaknesses, and guide the implementation of multivariate testing.

If you’re looking for how to run an A/B test, we’ve published that article and the necessary calculators as well.

How to Run an A/B Test

Real-World Scenario: MVT Outperforming A/B Testing

Let’s consider a B2B software company that offers a project management tool. The company wants to optimize its demo request page to increase the number of demo sign-ups. They decide to test the following elements:

1. Headline
2. Image
3. Call-to-Action (CTA) Button

A/B Testing Approach

The company first conducts separate A/B tests for each element:

Based on these A/B tests, the company would implement the winning versions: the “Boost Team Productivity by 30%” headline, the software dashboard screenshot, and the “Request a Demo” CTA button. The combined effect might yield a 10% increase in demo sign-ups.

Multivariate Testing Approach

Now, let’s see how a multivariate test might yield different results. The company sets up an MVT with the following variations:

This creates eight possible combinations (2 x 2 x 2). After running the test, here are the results:

Headline	Image	CTA	Result
Control: “Streamline Your Project Management”	Control: Screenshot of software dashboard	Control: “Request a Demo”	Baseline
Control: “Streamline Your Project Management”	Control: Screenshot of software dashboard	Variation: “Start Free Trial”	2% increase
Control: “Streamline Your Project Management”	Variation: Image of diverse team collaborating	Control: “Request a Demo”	5% increase
Control: “Streamline Your Project Management”	Variation: Image of diverse team collaborating	Variation: “Start Free Trial”	8% increase
Variation: “Boost Team Productivity by 30%”	Control: Screenshot of software dashboard	Control: “Request a Demo”	7% increase
Variation: “Boost Team Productivity by 30%”	Control: Screenshot of software dashboard	Variation: “Start Free Trial”	10% increase
Variation: “Boost Team Productivity by 30%”	Variation: Image of diverse team collaborating	Control: “Request a Demo”	12% increase
Variation: “Boost Team Productivity by 30%”	Variation: Image of diverse team collaborating	Variation: “Start Free Trial”	18% increase

Analysis

The MVT reveals that the combination of Boost Team Productivity by 30% (H2), the image of a business team (I2), and Start Free Trial (C2) produces the best results, with an 18% increase in demo sign-ups. This outcome differs from what the A/B tests suggested in two key ways:

• The image of the business team performs better in combination with the productivity-focused headline despite losing in the individual A/B test. This suggests an interaction effect between the headline and image that wasn’t captured in the A/B tests.
• The Start Free Trial CTA works best in this combination, even though it lost in the individual A/B test.

The overall improvement (18%) is significantly higher than what might have been expected from simply combining the winning elements from the A/B tests (around 10%).

Explanation

The synergy between elements in the winning combination can be explained as follows:

• The Boost Team Productivity by 30% headline makes a strong, quantifiable promise that appeals to business decision-makers.
• The image of a business team reinforces the idea of improved productivity and teamwork, making the promise more tangible and relatable.
• The Start Free Trial CTA lowers the barrier to entry compared to Request a Demo, allowing potential customers to experience the productivity boost firsthand without scheduling a demo.

This combination effectively tells a cohesive story: here’s a significant productivity improvement (headline) that you can see benefiting your team (image), and you can start experiencing it right away without any commitment (CTA).

This scenario demonstrates how multivariate testing can uncover powerful combinations of elements that might be missed with A/B testing alone. By testing these elements together, the company discovered a synergistic effect that produced better results than optimizing each element individually. This underscores the value of MVT in identifying how different page elements work together to influence user behavior and drive conversions in a B2B context.

Complexity of Multivariate Testing

Multivariate testing is inherently more complex than A/B testing for several reasons:

• Multiple Variables: MVT tests several changes simultaneously, which increases the number of possible combinations exponentially.
• Larger Sample Size: Due to the increased number of variations, MVT requires a larger sample size to achieve statistical significance.
• Longer Duration: MVT tests typically run longer than A/B tests because of the more significant sample size requirement.
• More Complex Analysis: Interpreting MVT results can be challenging, as you need to understand how different elements interact with each other.
• Resource Intensive: Creating and managing multiple variations requires more time, effort, and often specialized tools.

Advantages of Multivariate Testing

Despite its complexity, multivariate testing offers several significant advantages:

• Holistic Optimization: MVT allows you to optimize multiple page elements simultaneously, providing a more comprehensive view of what drives performance.
• Interaction Effects: One key benefit of MVT is its ability to reveal how different elements work together. This can uncover synergies between elements that might not be apparent in isolated A/B tests.
• Efficient Testing: While individual MVT tests may take longer, they can potentially replace multiple sequential A/B tests, saving time in the long run.
• Nuanced Insights: MVT can provide more detailed insights into user preferences and behaviors, helping you fine-tune your design and content strategies.

Process for Multivariate Testing

Here’s a step-by-step process for conducting a multivariate test:

1. Identify Variables: Determine which elements on your page you want to test. Common elements include headlines, images, calls to action, and layout.
2. Create Variations: For each element, create alternative versions. Remember, the total number of combinations will be the product of the number of variations of each element.
3. Set Up the Test: Use a multivariate testing tool to set up your test. This involves creating the different combinations and setting rules for traffic allocation.
4. Determine Sample Size: Calculate the required sample size to achieve statistical significance. This will depend on the number of variations and your desired confidence level.

Multivariate Testing Sample Size Calendar

5. Run the Test: Launch your test and allow it to run until it reaches the required sample size or a predetermined time limit.
6. Analyze Results: Use your testing tool to analyze the performance of different combinations. Look for both winning combinations and insights about element interactions.
7. Implement and Iterate: Apply the winning combination to your live page and use the insights gained to inform future tests.

Tools for Multivariate Testing

Several tools can assist with multivariate testing:

• Adobe Target: Part of the Adobe Experience Cloud, it provides robust testing and personalization features.
• Optimizely: A comprehensive experimentation platform that supports advanced multivariate testing.
• Unbounce: While primarily known for landing pages, Unbounce also offers multivariate testing features.
• VWO (Visual Website Optimizer): Offers a user-friendly interface for setting up and analyzing multivariate tests.

While A/B testing is more straightforward and quicker to implement, multivariate testing offers a more comprehensive optimization approach. By understanding the strengths and limitations of each method, you can choose the right testing strategy for your specific needs and resources.