Are you looking to hire someone that know’s a lot about conversion rate optimization & ab testing or you are about to apply to a new job? A lot of people claim they know what ab testing is because the concept is so simple. Experience has shown that there is a lot more to it because there are a lot of things to consider when running tests. I made a list of 10 ab testing interview questions that cover the most important aspects of running experiments within an organization. Let’s separate the amateurs from pros today!
AB testing interview questions #1: How do you increase conversion rate?
This question inspired by Peep laya from CXL in his article about creating a CRO process. It’s important to ask this question because you want to find out if the potential employee has a process. People with a CRO process will have higher success-rates and will be able to set up quality tests more frequently.
If they immediately answer with best practice strategies you probably want to find someone else because they are most likely amateurs and not very effective. You don’t want someone to try a bunch of tactics they found on a random blog because every test can take up to 4 weeks.
Try to found out how they choose what tests to run first and why they would do so because this requires them to have a process.
Question #2: What is the difference between a one-tailed and two-tailed test?
This question can give you some insight into someone’s knowledge of statistics. When running tests your result analysis must be valid otherwise they are implementing changes that have no proven positive effect on your conversion rate.
The answer you are looking for is that they know what kind of test tail they use depending on the testing hypothesis because using one-tailed tests can be cost-effective if you are testing in one direction.
A one-tailed is if you are testing something is smaller or bigger then a certain percentage and the other direction doesn’t matter. When you are testing if something is better then a 10% conversion rate for example. You want to know if it’s better and you don’t care if it’s worse because it reduces the costs anyway.
In conversion optimization on websites is rarely wise to select one-tailed testing because the other direction matters. You want to know what the result says about the other direction because it can have a bad effect on your revenue. To make sure that isn’t the case you are probably going to choose this one. But do know that this takes more time because you will need more sessions.
If you want to learn more about statistics in ab testing you can check out my other guide.
AB testing interview questions #3: What is the difference between qualitative research and quantitative research?
Another important aspect of AB testing is user research. There are different kinds of research methods available depending on what kind of answer you are trying to find. They need to know this difference because it will signal if they have any experience with user research.
You can ask them for a couple of examples per aspect to get a better view of the experience they have.
Qualitative research is direct and is usually more focused on smaller groups. Some examples would be focus groups, interviews, and co-design. These are examples that are quit attitudinal because they focus on what people say. But it can also be behavioral within a usability lab for example. Mostly it’s used to find insights and issues as inspiration for your testing hypothesis.
Quantitative research is usually done with a lot more participants and is mostly used to validate your testing hypothesis.
Examples of attitudinal quantitative research are intercept surveys and e-mail surveys because you are gathering a lot of information about what people say trough emailed questions. AB testing is an example of a behavioral research method because it’s focused on what people do depending on your goals for validating a testing hypothesis.
Want to know more? Check out this article about 8 UX research methods to find insights for A/B testing.
Question #4: What is the difference between client-side and server-side testing?
One thing to consider when you are AB testing is your tooling. One key difference between available tools is server-side and client-side testing. Asking this question will give you some insight into the potential employee’s knowledge of different tools. You could also ask them for a couple of example tools.
The main benefit of client-side testing is that it’s usually easier to implement with tools like Google Optimize with no start-up costs. This can also be done by non-developers and is mostly used to test smaller persuasive changes that don’t change the functionality of the page to much.
Server-side testing will require developers to implement these tools and can be used for bigger changes. The benefit to server-side testing is that is rendered a lot faster the client-side testing because it’s loaded before it goes to the user’s browser. This way the user will never be able to get the different variants using client-side tricks.
We notice that with a lot of client-side tools that anything you are testing above the fold will be invalid because it’s to slow with loading in. This way the user can see the change and the result will be affected. A solution here is to test under the fold or use a fast snipped as Google Optimize has.
A downside of server-side testing tools is that they are usually a lot more expensive.
AB testing interview questions #5: What’s the most important page metric?
If the potential employee answers anything but page speed he doesn’t know what he is doing. This is because any persuasion element or functional change will never have a positive effect on your conversion rate when it decreases the site speed.
Conversion rate compared to site speed by Cloudflare.
Site speed has a lot of effect on your conversion rate and this should always be one of the first things you should optimize when improving a website’s user experience.
You can also keep asking about the different kinds of page speed metrics available and what all of those metrics mean. There is interaction speed for example. The page might not be done loading but is ready for a user to interact with. This could arguably be even more important than the actual page speed.
You are trying to find out if the potential employee has any experience with marketing KPIs and how to use them. Because this is a very important part of AB testing. Most conversion optimizers are also performance marketeers.
Question #6: How do you present your insights in the company?
With this question, you are trying to find out if the potential employee has experience with sharing knowledge with their company and motivating other people.
You want this person to drive growth within your company and he or she will never be able to do that alone. It’s important this person has people skills and can transfer his motivation to other people.
Ask them to share their most awesome test with you and what they learned. The way the present it to you will tell you if you have a winner.
You can think of this person as an in the house coach that drives change and challenges people to think on their feet.
AB testing interview questions #7: Can you describe an ab test that gave you valuable insights?
Another important aspect of conversion optimization is learning. This can be done by testing a hypothesis or making mistakes in general. It will help you to improve the company and its process to accelerate your growth.
With this question, you are trying to find out if the potential employee reflects on their work when they make mistakes and learn from them. This is important because an employee that holds this skill will be able to improve his or her process. It shows if the candidate is confident enough to admit mistakes and show what he has learned from them. You want to know if the candidate logs and remembers his learning for use in the future and not just forget them as soon as he is done reflecting. Does this person find learning important enough? When the potential employee checks out with these three characteristics you will now if he is focused on growth.
Question #8: What is power in statistics?
In conversion optimization, it’s important to have a basic understanding of statistics because it will help you evaluate the statistical validity of your tests. Especially when you are running tests for bigger companies due to higher risks for revenue lost. A bad conversion rate optimizer can run your company to the ground.
This question is asked because you want to find out if he knows one of the most important factors in the analysis. This metric is very important because it’s value will decide that chance for missing out on a true average hypothesis. You don’t want to miss out on a chance that could net your company millions of dollars in extra revenue. Understanding and changing this metric will help you control this risk.
It’s also important because it helps optimizers assess the test duration needed for a specific hypothesis. This way he can decide quickly if a test hypothesis will even have a chance of reaching a significant result. As a result, saving your company a lot of money. This all depends on the level of risk you want to take off-course but he or she should be able to discuss this with you.
AB testing interview questions #9: What pages should you refrain from testing?
When choosing what to test the person needs to know how to prioritize what tests he will run first to maximize the chances to reach a significant test result. Testing costs a lot of time and money so making the wrong decisions about what to test next can cost you even more money. Especially when the tests are also being developed by an expensive developer.
You are trying to find out how your potential employee would prioritize the company’s tests. What things does this person consider and does he or she have a process for doing so? Ask them why their factors are important and what success-rate with testing they have had in the past. The average successful test rate according to VWO is about 13%. Any percentage higher than that will probably give you a good or lucky optimizer.
When specifically looking at pages you probably don’t want to test pages that change regularly. You want your tests to run with the least amount of variables possible because that will make your results more valid. A test that would have really good results with a particular homepage banner might not work the next week when the banner has changed. Make sure he or she focuses the effort on things that don’t change often or are going to change soon with a redesign for example.
Question #10: What is multi-variate testing?
There is more than one way of doing ab testing and one of those ways is using multi-variate testing. Which is running multiple ab-tests at the same time based on variables. This can be very effective when you are trying out different layouts or want to test for a longer time because you are going on vacation.
This question is designed to find out if the potential employee has experience with more advanced methods of testing. Most beginners will have probably never used this advanced technique of testing.
If they know what it is make sure you let them give you an example to prove their understanding of the topic. If you want to know more about multivariate testing you can check out my guide: Multivariate testing: Everything you need to know/
The main message of this article is that you want to find someone that takes calculated risks based on user research to maximize the chance for getting significant results that are statically valid. Good luck with your next interview on either side of the table! If you found this article interesting you can use one of the share buttons on the bottom of the post. Questions or remarks about this article can be left down in the comments below!