Multivariate testing: Everything you need to know


Do you know what are the most important elements on your website and the optimal place to put them on your landing pages? There are a lot of options for different layout structures with the amounts of elements you have on your landing pages. Deciding what is the best layout composition can be done by multivariate testing.

This article will contain everything you need to know to prepare, set-up, and analyze multivariate tests. If you are into A/B testing and optimizing your website you will love this guide.


Chapter 1: The basics

Chapter 1: The basics

Chapter 2: Preparing

Chapter 2: Preparing

Chapter 3: Setting it up

Chapter 3: Setting it up

Chapter 4: Analyzing your tests

The basics

The basics

What is multivariate testing?

Multivariate testing visualized by stormtroopers sitting on a bike
Stormtroopers by Andrew Martin – 2020

Multivariate testing is an advanced form of A/B testing that will contain multiple variants that will all be tested at the same time. All combinations of these variants will be tested side by side to find the best combination possible. 

The number of combinations will exponentially increase with every element you test. Because every single combination needs to be tested. Just having 3 elements will give you 9 different combinations that will be testing.

The best combination needs to be found to validate the testing hypothesis. The best combination can then be implemented for maximum performance. With a normal a/b test you would have to run far more tests to find the optimal version because you are only testing one combination at a time. 

Benefits of multivariate testing?

Multivariate testing can eliminate the need for sequential ab testing because you can test so many combinations at once. This means you will spend a little bit more time to set up a test that will run for a longer time. This will result in a more time-efficient and effective testing cycle.

Another benefit is that you are testing are variations against each other at the same time. This means that you have a better chance of getting the best combination because testing the winning version against each other can make it so that you are losing on a really good version early on.

When analyzing your multivariate test you will be able to see the effect of every single element tested. This means you will learn a lot about the effectiveness of different element types that you can use for testing in other parts of your website.

You can also use it to test a bunch of smaller changes that would probably not yield a significant result on their own. This will give you more optimization opportunities that can have a big impact on your site’s performance. Imagine changing the color of a button for example. On its own, it would probably not have a big effect on your website performance. But combined with a different copy and interaction style might give your test a better chance of succeeding.

Downsides of multivariate testing?

Multivariate testing will need an insane amount of traffic on your website otherwise the tests would take to long the reach significance and justify a good return on investment. You might be testing 9 combinations at the same time and every one of them needs it’s own traffic te be successful.

Another downside of multivariate testing is the setup time. Because you have a lot more elements to set up a more complex test you will probably end up using a lot more time just setting it up properly.

If you then make on a mistake on any of your elements your whole test will need to be redone so checking your test once is started will be more important and will take a lot more time. But this is usually repaid if you do it well because you don’t have to do sequential testing and have a higher chance for a better performing change.

When do you use multivariate testing?

Multivariate testing option in Google Optimize
Testing options in Google Optimize – 2020

You can use multivariate testing when you have many different kinds of elements that you want to test because then you can test them at the same time.

Another reason would be if you are unsure what you want to test first on a landing page. Imagine many great ideas and you don’t know what might have the biggest impact. Multivariate testing can then be your crutch in this decision.

Multivariate testing is also a great way to set up a long test if you are going on vacation for example. This way you are effectively using the time that you are away by running an advanced test. Don’t do this if you can’t separate vacation and work because you might be tempted to check on the test every now and again. I know I would!

Example of multivariate testing?

Google had a really nice example to explain multivariate testing. Imagine looking at the homepage of the website for example. Above the fold of the website, you can see a call to action with another block element underneath it.

With these two locations, you want to test three hero images and two calls to action. Giving you a total of 6 combinations. 

Google visualizes multivariate testing very well
Google visualizes multivariate testing very well – 2020

This way you can test the best setup for your elements on a landing page.



How can you pre-test a multivariate test?

Preparing multivariate testing with your team
Preparing with your team by StockSnap – 2020

Before starting your test you need to analyze if you have enough traffic for completing the test because then you what the minimal detectable effect is for your test. This means the percentage increase or decrease needed to reach a significant result. This way you know that the changes you have made are probably responsible for the test results. 

This is fairly easy with a/b tests by using the CXL a/b test calculator. Just fill in the weekly traffic and conversions and see the minimal detectable effect you need for 6 weeks of testing. If you want to use this for multivariate testing you need to know the number of combinations you test will have. In our previous example, we saw 6 combinations meaning you need to divide your traffic by six before filling in the data.

You can get the amounts of combinations  you want to test by multiplying the number of variants per element location:

Elements location 1 * Elements location 2 * Elements location 3 = Combinations

You can add or remove locations to your liking. This is another example of how many locations and variables will increment the number of combinations drastically. Google Optimize will calculate these combinations for you automatically.

How to write a multivariate hypothesis?

Writing multivariate testing hypothesis
Writing by StartupStockPhotos – 2020

Multivariate tests require every element you test to have its own testing hypothesis because then you will be able to learn from all the elements you test. In the above example where there are 6 combinations because of 3 hero banners & 2 calls to actions, you will need 5 different hypotheses.

The way I write the hypothesis is taken from my frequently visited resource CXL. This article is described how Craig Sullivan uses this template for writing a strong hypothesis:

“We believe that doing [A] for people [B] will make outcome [C] happen. We’ll know this when we see data [D] and feedback [E].”

I use this for all of my tests and strongly recommend people to use the same format because that way you can always go back to see what you tested exactly.

What is a big enough sample size?

Sample size visualized with stadium chairs
Sample size by Ich bin dann mal raus hier. – 2020

This really depends on the amounts of combinations you are going to test with your multivariate test because we know that the more traffic a test has the more accurate it will be. For smaller changes, you will need a bigger sample size to accurately measure the effect. There are different ways of increasing your sample size for your multivariate tests.

On way is increasing the test duration because you will have a longer time to collect more data. Another way is by decreasing the number of combinations because every combination will divide the test into even more sections. The last way would by generating extra traffic to your test pages by online and e-mail marketing.

A good guideline is to have at least 500 conversions per testing hypothesis and checking the test while it’s running to see if it’s worth increasing the test time. Make sure you are always testing full weeks because weekdays have a lot of effect on user behavior. 

How long do you need to test for?

Day planner for multivariate testing
Day planner by Free-Photos

This depends on the number of combinations and traffic you have and what other important metrics. I recommend testing with a power of 80% because this is the industry standard for balancing out type 1 & type 2 errors and a confidence level of 95%. You can use the pre-test analysis in the ab test calculator from CXL to get the minimal detectable effect for each sequential testing week.

The power that you use for evaluating the test will determine the change for a missed result(type 2 error). Using a power of 80% will give you 20% of missing the result but will decrease the test time significantly so it’s definitely worth the risk.

When you use a confidence level of 95% you will have a 5% chance that the result is a result of change and note the change you made(type 1 error).

These two metrics will influence the minimal detectable effect(MDE). This is the minimum change you need to reliable meet your power value. The MDE will decrease when you are testing for a longer period of time with more traffic.

What do you normally get on your tests and what do you think is realistic for the changes you are making? You can also fill in the MDE with mockup data in the test analysis section to get an idea of the ROI if you reach the MDE with your power and confidence level.

I recommend testing at least two weeks and implement a platform like Tracedock if you are testing longer because cookies will be deleted after two weeks which will result into test all kinds of data pollution. Solutions like Tracedock will help you preserve this data but complying with the GDPR regulations.

How to make a multivariate testing plan?

Making a multivariate testing plan
Making a testing plan by Malachi Witt – 2020

Sometimes you will be able to implement the multivariate test yourself but it’s usually better when a developer sets up your test in a server-side environment. This means your developer will need a bunch of information to set up your test properly. I made a small list of information you can supply to a developer to help him set up your tests. Depending on his own interests you might be able to leave some of the information out.

  • Start and end date
  • Amount of combinations
  • Page types
  • Target group
  • Design of every element of every device. This means you need to make a lot of different design variations for every test you run.
  • Testing hypothesis per element
  • Traffic allocation(how much site traffic will the test receive? I recommend 100% because of the risk of data pollution)
  • Testname
  • Testing goal
  • Activation event(when is the test displayed? Page load or scroll for example)
Setting it up

Setting it up

What tools support multivariate testing?

Tools from – 2020

Most known A/B testing tools support multivariate testing. I made a small list of the top 10 known tools that are popular and if they support multivariate testing. Keep in mind that software might always add or remove these functionalities in the future so check it yourself before signing up!

ToolMultivariate testing
Google OptimizeYes
Google Tag ManagerNo
AB TastyYes
Adobe TargetYes

For the sake of simplicity and lack of time, I will not be making a guide in this article for every one of these tools because all of them already have a guide on them if they offer the functionality. I recommend using this Google Optimize guide if you are trying this for the first time.

Analyzing your tests

What are the multivariate analysis methods?

The goal of analyzing a multivariate test is to find the best performing combination. The way you analyze a multivariate test is by comparing every combination to the baseline. If you have nine combinations you will have 9 different results that can be compared to each other. The one with the highest change will be the winner. 

Screenshot of the multivariate testing results in Google Optimize
Screenshot of the multivariate test results in Google Optimize – 2020

You can use the CXL ab test calculator and analyze every combination and collect the results in a spreadsheet for future reference. Make sure you have a big enough sample size, duration & that it beats the original significantly before concluding anything with your data. I have my spreadsheet set up something like this and also group the hypothesis per combination. You can also add more of the CXL questions and your learnings per combination.

HypothesisCombinationsTrafficConversionsChangeSample size
Hypothesis 1
Hypothesis 2
Hypothesis 3
Combination 120000100100%Yes
Hypothesis 2
Hypothesis 3
Hypothesis 4
Combination 220000150200%Yes
Hypothesis 4
Hypothesis 5
Hypothesis 6
Combination 320000200300%Yes

In the calculator, you can see every combination as a testing variant, this is how it would look like in the CXL ab calculator. The numbers are just for the demonstration, unfortunately.

Multivariate combinations analyzed
Multivariate combinations analyzed – 2020

On a side note: You can actually share a filled-in version of your calculator so everyone can have a look at the data themselves. I love using it for my in house presentations:

Sample link to AB+ Test calculator by CXL


I hope you will find this information useful to set up your first multivariate test! By following the steps you should be well on your way of setting up such an advanced test that can be very beneficial for your company.

Multivariate testing is one of the bests ways of testing combinations of elements that would have otherwise need to be tested with one after another with suboptimal results. Go set up your first test today and feel free to share this post with your team. Have any questions? Feel free to send me a message or drop a comment down below!