Why evaluate your ab test with google analytics?
You are using Google Optimize to run tests on your website because you want to improve your conversion rates. Why would you need to evaluate your ab test with Google Analytics? Imagine this: When you finally finished tests after waiting for two weeks something catches your eye in the results. It says the test has failed and you should start a new test because you are probably not going to reach significance anyway.
Don’t despair because there still might be hope for the research you have done. Evaluating your tests further in Google Analytics might be the answer to your problem because your test might not be successful when evaluating it in Google Optimize. But what happens when we evaluate the data in different segments. Maby your test will be significant for returning users for example.
This article is about evaluating your Google Optimize ab test with Google Analytics and using segments to find that golden nugget because just looking at the results in Google Optimize is not enough. In the end, you will be able the learn more from your tests and use the insights to implement personalizations on your website. Furthermore, you will be able to share a lot more data with your colleagues.
The sections that are described in this article are about Google Analytics segments and evaluating your ab test with Google Analytics using segments.
Google Analytics segments
When you have successfully implemented Google Optimize on your website you will have had to make a connection with Google Analytics to track your data. If you haven’t done this yet check out my other guide: How to use the Google Optimize a b testing tool? But how are you able to view the data that has been sent to Google Analytics? You can evaluate your test in Google Analytics using segments.
Choosing your segments
Before we start creating segments in Google Analytics we need to decide what segments we want to evaluate for our test. Here is a list of possible segmentations that I use when evaluating my ab test:
- Returning users
- New users
- Device category
- Location – (Depending on the locations of your audience)
This is just a fraction of the segmentations that are possible in Google Analytics. I don’t recommend making personalizations for segments that have to many variables. Let’s take interests for example.
There must be over 100 different interests that will just take to much of your time for little to no gain because the segments will be extremely small. Next to that, it takes a lot of work if you would need to implement different personalizations on this scale.
Pick the segments you want to analyze create your own list in a text editor. You will need this for the next step.
Naming your ab test in Google Analytics
Next up is naming your segments. I recommend making individual segments for the items on your list so you can easily use them to recheck older tests and on every dashboard. When I first starting making segments I made a horrible mistake. Giving your segments random names that don’t follow a strict pattern will make it so you can’t find your segments in the future quickly. You might become frustrated and just create a new segment and you’ll have even more of a mess.
You can come up with your own pattern but I use this one: Test number – Test type – Segment category – Segment – Variant. Here is what it looks like when I use this pattern for the list I made.
My segment list
You can also choose to only make the first two and use existing reports to filter out your specific needs. That’s all up to you.
Creating your ab test segment with Google Analytics
You will need to create segments for every test you make because every test has its own unique variable ID. Below are the steps to create a segment in your Google Analytics environment.
Step #1: Admin panel
Make sure you are logged into Google Analytics and look at the bottom left navigation. Click on the Admin button with the cogwheel.
Step #2: Segment view
Next look at the third column under view. Click on the “Segments” button under “Personal tools & assets”.
Step #3: Creating a new segment
In the next menu, u can see a list of all segments that you have made or already exist within this view. You will also see all shared segments that are connected to your account. We use this menu instead of making segments in a random report page because otherwise, you would need to reload the page data for every segment you make. This is obviously faster. Next click on the red “New segment” button to start creating a segment.
Step #4: Copying segment name
Furthermore, you will start adding your information to segment data in Analytics. On the top left, you can add your segment name. Copy the first item from your list that follows your naming convention.
Step #5: Segment conditions
On the left of the menu click on conditions. This is where you will add all your rules.
Step #6: Condition dimension
Click on the first drop-down field that now says “Ad Content”.
Step #7: Experiment ID with variant
You will be able to type in a search query to find the dimension you need. We are looking for the dimension “Experiment ID with Variant” because this is directly imported from Google Optimize. Click the dark green button with “Experiment ID with Variant” to add it.
Step #8: Getting your ID
In our previous post: How to use the Google Optimize a b testing tool? We discussed how to use the experiment ID in the browser application function the change the variant we are looking at. We will need to use that same experiment ID to create our segment. In Google Optimize, go to your test details and copy your experiment ID.
Step #9: Adding your ID
Going back to Google Analytics you will need to paste your variant ID and add .0 to it if you want to segment the control. If you want to add the variant you can add .1 or any other number depending on how many variants you have. In the example, I added the control. Copy my other settings if they are different from yours.
Step #10: Saving your segment
Now click on the blue “Save” button and you should have saved the first segment that you can use in Google Analytics.
Step #11: Repeat steps
Repeat steps 1 till 10 until you have created all the segments you want to analyze in Google Analytics. Again you can also choose to only make the control and variant and just use the different dashboards that Google Analytics has to offer.
Evaluating your ab test with Google Analytics
Now let’s get to the interesting part! We can use our new segments to start evaluating your ab test with Google Analytics.
Creating your report
First, we need to create the report that we can apply our segments to. For this tutorial, I will create a custom report because I can then choose precisely what metrics I want to analyze.
Step #1: Customisation
In Google Analytics click on the Customisation drop-down in the top-left.
Step #2: Viewing custom reports
In this drop-down menu click on “Custom Reports” to view all your custom reports.
Step #3: New custom report
Now click on the “New custom report” button on the top-left of the page.
Step #4: Custom report title
Fill your custom report title so you can easily find it in the future. I just called it AB testing. I didn’t make a naming convention here because I don’t make a lot of custom reports. But it might not be a bad idea if you are going to use this functionality frequently.
Step #4: Choosing metrics
Now you need the choose the metrics you are interested in. This really depends on the ab test you have done. In this example, I will choose the metrics that can be important for improving the e-commerce conversion rate. Click on “add metric” to get a drop-down with options.
Step #5: Search the metrics
Find the metrics e-commerce conversion rate, sessions & transactions and add them to your metric group selection. You can add by first searching for the metric in the search bar and then clicking on the button with your metric name.
Step #6: Default channel grouping
Now we need to add our dimensions the same way you added your metrics. I will use the dimension “Default Channel Grouping”.
Step #7: Saving your custom report
To create your custom report just press the grey “Save” button at the bottom of the page.
Adding your segments
When you have created your custom report you can always find it under Customisation > Custom reports in the Google Analytics menu. This is a report where we can add segments to.
Step #1: Adding your segment
On the top page click on the grey “Add Segment” outline to start adding our segments.
Step #2: Selected segment
First, we need to clear the standard segment. Click on the “selected” button under the “view segments” menu.
Step #3: Deselecting the standard segment
Then click on the blue “All Users: segment to deselect it from your current dashboard. The box will uncheck and the blue selection will disappear.
Step #4: Custom segment
Now on the left menu click on custom to view the custom segments you made in the previous steps.
Step #5: Control and variant segment
Now select your control and variant segment of the test, make sure you add them in this order because it will change the way you look at your data otherwise. To me, it makes more sense to always show your control first. The blue selection will be the first segment shown.
Step #6: Apply
And press the blue “Apply” button to show your segmented data in Google Analytics.
Step #7: Setting your date range
Make sure you select the correct date range that corresponds to your test planning otherwise you might miss out on data. You can find your test date range in Google Optimize.
Evaluating your data
Look at all that data! You can now already see some of the results of your test! You can choose to check out more of your segments to look at specific data or just be happy with the data you have now.
But when has your test been successful for each of your segments? Because having a higher conversion rate isn’t enough information to decide this fact. Luckily a lot of companies and institutions have made ab testing calculators that make it extremely easy to evaluate your test.
My personal favorite is the one made by CXL. Their calculator helps you to answer 4 specific questions post-test to decide if your test was successful or not. I will run through the answers with my test data as an example.
Does the test variant beat the original?
The variant beats the original because it had a lift of 14,29%.
This coincides with the minimal detectable effect that is needed for this sample test. In one week you need at least a lift of 12.32% for statistical significance. Imagine that you only had 9% in one week of testing. This tells you should keep collecting more data to successfully finish your test. Just look at this table if you need to decide to stop or continue testing. I would probably stop my test if I had only 2% in two weeks. Better of testing something else.
Does the test have the needed sample size?
The test does have the need sample size as shown on the calculator. For these results, we needed about 13.000 sessions per variant and we are well over that.
Does the test have the needed duration?
The test does have the needed duration as shown on the calculator because we needed 5 days of data. Our test data has 7 days of data. I always recommend testing in cycles of 7 days otherwise you might have polluted data. Every test will always act differently at the weekend for instance.
What is the monetary ROI of the test variant?
This one always makes people pay attention to. If you answered yes on the previous answers it will mean that implementing your test will probably have a return on investment. If you are an e-commerce website make sure to fill in your average order value for one non-control variant.
For this test, it would be €31175 extra a month because of 1247 transactions with an order value of €25. These numbers really depend on the size of your business of course. The return on investment will be much lower on a website with less traffic and it will also be a lot harder to reach statistical significance. I recommend having a minimum of 10000 visitors a month before it’s even worth testing.
You can check out my test data live at this link.
Checking for significance
To make sure your test is successful you need to check it’s statistical significance. This basically means that the results of the test are most likely because of the changes you made. I will make a whole article about statistical significance in the future. Luckily you can just scroll down in the ab test calculator and look at the Bayesian and Z Test results. They will show you the probability of a variant is a winner and if the test was statically significant.
Definitely check out conversionxl.com if you want to know more about growth and testing in general.
Deciding your next steps
Depending on your results you need to decide if you want to implement the test on your website or if you want to keep the test running because you couldn’t answer yes to ab test calculator questions. You can also adjust the confidence level or statistical power of the calculator the get significance for lower levels and make higher risk implementations. I would recommend just keeping the standard settings if you want to play it safe.
I hope this guide helped you to evaluate your Google Optimize ab test with Google Analytics. You should now be able to create segments in Google Analytics to dive deeper into the test data of Google Optimize. Next to that, you have some first time experience with evaluating your test for statistical significance and return on investment. If you have any questions feel free to ask them under this topic and I will answer all of them respectively. Good luck with your future tests!
Are your tests statistically valid?
This statistics course for A/B testing will help you avoid costly testing mistakes.