Even minor tweaks to your website can have major unintended consequences, tanking your conversion rate or creating a negative user experience. This is why great marketers don’t change websites and campaigns off hunches. They test each new element, comparing results against previous metrics to ensure all components work together.
The most common testing method is A/B testing, or split testing, where you test two versions with one differentiating factor. But marketers can also benefit from multivariate testing, in which they test multiple elements simultaneously.
Learn what multivariate testing is, its benefits and limitations, and how to run an effective multivariate test to increase conversions on your site.
What is multivariate testing?
Multivariate testing (MVT) is the practice of changing multiple elements on a website at once—such as headers, calls to action (CTAs), images, design layouts, and copy—to create the most effective version of a page. Using MVT can help you glean insights about more than just the performance of individual elements—it shows which combinations of those elements have the most influence on user engagement or conversions.
You can use multivariate testing to:
- Improve conversions on a landing page by testing elements like headers, CTAs, and copy
- Increase checkout clicks by testing different checkout button colors and their placement
- Encourage add to carts by testing variations of the images and pricing options that appear on product pages
- Boost subscriptions by testing different variations of sign-up forms, including form fields and placement
Although there are a few multivariate testing methods, the most common testing method is full factorial testing, in which you split all variations equally among all website traffic. For example, if you’re testing multiple elements of a product page design to understand which combination results in the most conversions, you would run the test as follows:
2 (product image) x 2 (product description) x 2 (CTA placement) = 8 variations split equally among all product page visitors
Multivariate testing vs. A/B testing
In A/B testing, also known as split testing, you test only two variations (A and B) of a particular element by directing traffic to one or the other, and see which performs better. This could look like testing two versions of a sign-up CTA or a homepage header.
In multivariate testing, you test multiple page elements simultaneously. This can mean testing different combinations of headlines, images, and copy on a homepage or product page at the same time.
The advantage of A/B testing is that it doesn’t require much traffic and can deliver reliable results quickly—usually in one to two weeks. Because of this, many companies run sequences of A/B tests continuously, even if they have many variants to test. However, multivariate testing allows for insight into more elements on a page and the relationship between those elements. For example, it can help show the relationship between your product image and header copy.
In an A/B test, you would test two copy versions on the same CTA. In a multivariate test, you would test the two versions of CTA copy, their placement, and color, all at once.
Benefits of multivariate testing
There are two main benefits of using multivariate testing over A/B testing.
Comprehensive, combination-informed insights
Testing many possible combinations of elements gives you a detailed understanding of how different page elements interact with each other and impact overall performance, allowing you to identify the best possible combination of elements. Because multivariate testing doesn’t limit the number of testing variations, you can be sure that you have identified the most optimal one once testing is complete.
Efficiency
Say your marketing team has mocked up three versions of your homepage. Each one has different headers, above-the-fold CTA copy, hero images, and messaging. Testing each element of each page with sequential A/B tests would take months. Instead, you can run one multivariate test to identify which specific combination results in the most clicks or conversions and have an answer in weeks.
Limitations of multivariate testing
Here are three limitations of multivariate testing:
Requires large sample sizes
Multivariate tests require larger sample sizes than A/B tests due to the increased number of variations. A good rule of thumb is that for every variant you introduce, double the amount of traffic or sessions in your sample size. This could look like the following:
2 variations: 10,000 sessions
3 variations: 20,000 sessions
4 variations: 40,000 sessions
…and so on.
This longer testing period is to accumulate enough website visitors to reach statistical significance.
Implementation can be complex
A/B testing is relatively easy, even for those with limited testing experience, as all you need to do is change one variable; however, setting up a multivariate test and analyzing the results can be complex and require a good understanding of statistics. Often, you need an advanced testing platform to manage multiple variables effectively.
Risk of overcomplicating analysis
Testing too many variables simultaneously can result in data overload, making it difficult to draw meaningful results. If the added complexity does not significantly improve performance, you might find that the increased complexity wasn’t worth the effort. Sometimes, it might be easier and faster to have fewer variables and run a couple of A/B tests compared to one long and drawn-out multivariate test.
How to run a multivariate test
- Define the problem
- Form a hypothesis
- Create variations
- Determine your sample size
- Choose a testing tool
- Collect data
- Analyze results
Executing multivariate tests isn’t very different from A/B testing, but it requires a few steps to ensure you’re running the test accurately due to the added complexity of extra variants.
1. Define the problem
Start by clearly identifying the number you’re looking to improve. Is your homepage receiving a lot of traffic but low click-throughs to product pages? Are customers adding products to their carts but dropping off before they buy? Setting an objective will help you identify which elements to focus your test on.
2. Form a hypothesis
Just like any good scientist, a marketer shouldn’t begin an experiment without a clear hypothesis. Formulate a suggested solution to the problem you’re looking to solve before starting any testing. A hypothesis will give you clear direction so you aren’t experimenting without focus. Fill in the blanks below to form your hypothesis:
Based on (research), I expect that (proposed solution(s)) will result in (expected outcome).
Here is an example:
Based on click events on the homepage, I expect that moving the “See All Products” CTA above the fold and changing the color from black to red will result in improved click rates to the All Products page.
3. Create variations
Now that you have your hypothesis, create variations to test it. This can include changing layouts, headers, CTAs, colors, and more. In the previous example, you would create the following test variations:
2 (black vs. red CTA color) x 2 (Below the fold vs. above the fold CTA) = 4 variations
- Black CTA below the fold
- Black CTA above the fold
- Red CTA below the fold
- Red CTA above the fold
4. Determine your sample size
Multivariate testing requires larger sample sizes than A/B testing to achieve statistically significant results. For each additional variation, double the amount of traffic you’ll need. In the example above, there are four variations. If your homepage usually receives 5,000 visits a week, you’ll need 40,000 visits in total as your sample size. The exact amount of traffic you need will depend on your website’s conversion rate—more conversions allow you to reach statistical significance with less traffic.
5. Choose a testing tool
Next, you will need a tool to run your test on. When choosing a tool, keep the following things in mind:
- User-friendliness and ease of use
- Ability to run multivariate tests
- Reporting and analytics, including built-in statistical significance calculations
- Flexibility and customization options
VWO, Convert, and Optimizely are three popular testing tools for ecommerce websites that integrate with Shopify.
6. Collect data
Start driving traffic to your testing pages and track key metrics related to your original objective. If you cannot gather enough traffic volume to achieve statistical significance, promote your pages on social media and increase paid ad spend to reach your sample size goal.
7. Analyze results
The last step is to analyze the test results and assess whether your hypothesis was correct. Review each variation carefully and look for patterns. Sometimes, the winning variation may not be the one you implement permanently—you may find that although one variation won, a different variation better aligns with your goals based on user behavior.
Multivariate testing FAQ
What are the different types of multivariate tests?
The different types of multivariate tests are full factorial, fractional factorial, and Taguchi testing. The most common one is full factorial testing.
What variables can you test in multivariate tests?
In multivariate tests, you can test combinations of elements on your website, including headlines, images, CTAs, colors, layouts, and copy to determine their impact on user behavior and overall performance.
What is an example of multivariate testing?
An example of multivariate testing is having multiple versions of a landing page with different headlines, CTAs button colors, and hero images to see which combination drives the highest conversion rate.
