Categories
Guides

How to evaluate your Google Optimize ab test with Google Analytics?

Why evaluate your ab test with google analytics?

You are using Google Optimize to run tests on your website because you want to improve your conversion rates. Why would you need to evaluate your ab test with Google Analytics? Imagine this: When you finally finished tests after waiting for two weeks something catches your eye in the results. It says the test has failed and you should start a new test because you are probably not going to reach significance anyway. 

Don’t despair because there still might be hope for the research you have done. Evaluating your tests further in Google Analytics might be the answer to your problem because your test might not be successful when evaluating it in Google Optimize. But what happens when we evaluate the data in different segments. Maby your test will be significant for returning users for example. 

Diving deeper

This article is about evaluating your Google Optimize ab test with Google Analytics and using segments to find that golden nugget because just looking at the results in Google Optimize is not enough. In the end, you will be able the learn more from your tests and use the insights to implement personalizations on your website. Furthermore, you will be able to share a lot more data with your colleagues.

The sections that are described in this article are about Google Analytics segments and evaluating your ab test with Google Analytics using segments.

Do you want to become really good at using Google Analytics? Then check out this guide on Udemy.

Udemy logo

Google Analytics for Beginners – Zero to Hero

  • Effectively navigate Google Analytics reports
  • Demystify the common Digital Analytics terms and Google Analytics terms
  • Get Hands on Training with a Google Analytics eCommerce Demo Account
  • Track the Success of Your Marketing Campaigns in Google Analytics
  • Understand Account Hierarchy and Structure in Google Analytics
  • Learn how to setup Goals in Google Analytics
  • Track non-page view actions such as video views, downloads etc. in Google Analytics
  • Google Analytics 101 to Google Analytics reports mastery

GrowthPenguin is reader-supported. When you buy through links on our site, we may earn an affiliate commission.

Google Analytics segments

When you have successfully implemented Google Optimize on your website you will have had to make a connection with Google Analytics to track your data. If you haven’t done this yet check out my other guide: How to use the Google Optimize a b testing tool? But how are you able to view the data that has been sent to Google Analytics? You can evaluate your test in Google Analytics using segments.

Choosing your segments

Before we start creating segments in Google Analytics we need to decide what segments we want to evaluate for our test. Here is a list of possible segmentations that I use when evaluating my ab test:

  • Users
    • Returning users
    • New users
  • Device category
    • Desktop
    • Tablet
    • Mobile
  • Gender
    • Male
    • Female
  • Location – (Depending on the locations of your audience)
    • Netherlands
    • Belgium

This is just a fraction of the segmentations that are possible in Google Analytics. I don’t recommend making personalizations for segments that have to many variables. Let’s take interests for example. 

There must be over 100 different interests that will just take to much of your time for little to no gain because the segments will be extremely small. Next to that, it takes a lot of work if you would need to implement different personalizations on this scale.

Interests in Google Analytics for evaluating your ab test with google analytics
Interests in Google Analytics – 2020

Pick the segments you want to analyze create your own list in a text editor. You will need this for the next step.

Naming your ab test in Google Analytics

Next up is naming your segments. I recommend making individual segments for the items on your list so you can easily use them to recheck older tests and on every dashboard. When I first starting making segments I made a horrible mistake. Giving your segments random names that don’t follow a strict pattern will make it so you can’t find your segments in the future quickly. You might become frustrated and just create a new segment and you’ll have even more of a mess. 

You can come up with your own pattern but I use this one: Test number – Test type – Segment category – Segment – Variant. Here is what it looks like when I use this pattern for the list I made.

My segment list

Segment list - 2020
Segment list – 2020

You can also choose to only make the first two and use existing reports to filter out your specific needs. That’s all up to you.

Creating your ab test segment with Google Analytics

You will need to create segments for every test you make because every test has its own unique variable ID. Below are the steps to create a segment in your Google Analytics environment. 

Step #1:  Admin panel

Make sure you are logged into Google Analytics and look at the bottom left navigation. Click on the Admin button with the cogwheel.

Home dashboard for evaluating your ab test with google analytics
Home dashboard in Google Analytics – 2020

Step #2: Segment view

Next look at the third column under view. Click on the “Segments” button under “Personal tools & assets”.

View column in admin panel in Google Analytics
View column in admin panel in Google Analytics – 2020

Step #3: Creating a new segment

In the next menu, u can see a list of all segments that you have made or already exist within this view. You will also see all shared segments that are connected to your account. We use this menu instead of making segments in a random report page because otherwise, you would need to reload the page data for every segment you make. This is obviously faster. Next click on the red “New segment” button to start creating a segment.

Segment overview in Google Analytics
Segment overview in Google Analytics – 2020

Step #4: Copying segment name

Furthermore, you will start adding your information to segment data in Analytics. On the top left, you can add your segment name. Copy the first item from your list that follows your naming convention.

Segment settings in Google Analytics
Segment settings in Google Analytics – 2020

Step #5: Segment conditions

On the left of the menu click on conditions. This is where you will add all your rules.

Conditions option in segment settings – 2020

Step #6: Condition dimension

Click on the first drop-down field that now says “Ad Content”.

Condition dimension in Google Analytics
Condition dimension in Google Analytics – 2020

Step #7: Experiment ID with variant

You will be able to type in a search query to find the dimension you need. We are looking for the dimension “Experiment ID with Variant” because this is directly imported from Google Optimize. Click the dark green button with “Experiment ID with Variant” to add it.

Dimension finder in Google Analytics
Dimension finder in Google Analytics – 2020

Step #8: Getting your ID

In our previous post: How to use the Google Optimize a b testing tool? We discussed how to use the experiment ID in the browser application function the change the variant we are looking at. We will need to use that same experiment ID to create our segment. In Google Optimize, go to your test details and copy your experiment ID.

Google Optimize experiment ID
Google Optimize experiment ID – 2020

Step #9: Adding your ID

Going back to Google Analytics you will need to paste your variant ID and add .0 to it if you want to segment the control. If you want to add the variant you can add .1 or any other number depending on how many variants you have. In the example, I added the control. Copy my other settings if they are different from yours.

Setting your Google Optimize experiment
Setting your Google Optimize experiment – 2020

Step #10: Saving your segment

Now click on the blue “Save” button and you should have saved the first segment that you can use in Google Analytics.

Save and cancel button in Google Analytics
Save and cancel button in Google Analytics – 2020

Step #11: Repeat steps

Repeat steps 1 till 10 until you have created all the segments you want to analyze in Google Analytics. Again you can also choose to only make the control and variant and just use the different dashboards that Google Analytics has to offer.

Evaluating your ab test with Google Analytics

Now let’s get to the interesting part! We can use our new segments to start evaluating your ab test with Google Analytics.

Creating your report

First, we need to create the report that we can apply our segments to. For this tutorial, I will create a custom report because I can then choose precisely what metrics I want to analyze. 

Step #1: Customisation

In Google Analytics click on the Customisation drop-down in the top-left.

Customisation menu in Google Analytics
Customisation menu in Google Analytics – 2020

Step #2: Viewing custom reports

In this drop-down menu click on “Custom Reports” to view all your custom reports.

Selecting custom reports
Selecting custom reports – 2020

Step #3: New custom report

Now click on the “New custom report” button on the top-left of the page.

New custom report button in Google Analytics
New custom report button in Google Analytics – 2020

Step #4: Custom report title

Fill your custom report title so you can easily find it in the future. I just called it AB testing. I didn’t make a naming convention here because I don’t make a lot of custom reports. But it might not be a bad idea if you are going to use this functionality frequently.

Custom report screen
Custom report screen – 2020

Step #4: Choosing metrics

Now you need the choose the metrics you are interested in. This really depends on the ab test you have done. In this example, I will choose the metrics that can be important for improving the e-commerce conversion rate. Click on “add metric” to get a drop-down with options.

Metric finder in Google Analytics
Metric finder in Google Analytics – 2020

Step #5: Search the metrics

Find the metrics e-commerce conversion rate, sessions & transactions and add them to your metric group selection. You can add by first searching for the metric in the search bar and then clicking on the button with your metric name.

Selected metric in custom reports
Selected metric in custom reports – 2020

Step #6: Default channel grouping

Now we need to add our dimensions the same way you added your metrics. I will use the dimension “Default Channel Grouping”.

Dimension drilldown in Google Analytics
Dimension drilldown in Google Analytics – 2020

Step #7: Saving your custom report

To create your custom report just press the grey “Save” button at the bottom of the page.

Save button
Save button – 2020

Adding your segments

When you have created your custom report you can always find it under Customisation > Custom reports in the Google Analytics menu. This is a report where we can add segments to.

Custom report without segments
Custom report without segments – 2020

Step #1: Adding your segment

On the top page click on the grey “Add Segment” outline to start adding our segments.

Add segment button in Google Analytics
Add segment button in Google Analytics – 2020

Step #2: Selected segment

First, we need to clear the standard segment. Click on the “selected” button under the “view segments” menu.

Selected option in segment view
Selected option in segment view – 2020

Step #3: Deselecting the standard segment

Then click on the blue “All Users:  segment to deselect it from your current dashboard. The box will uncheck and the blue selection will disappear.

Selected all users segment in Google Analytics
Selected all users segment in Google Analytics – 2020

Step #4: Custom segment

Now on the left menu click on custom to view the custom segments you made in the previous steps.

Custom selection in view segments
Custom selection in view segments – 2020

Step #5: Control and variant segment

Now select your control and variant segment of the test, make sure you add them in this order because it will change the way you look at your data otherwise. To me, it makes more sense to always show your control first. The blue selection will be the first segment shown.

Selected Google Optimize segments for evaluating ab test with Google Analytics
Selected Google Optimize segments for evaluating ab test with Google Analytics – 2020

Step #6: Apply

And press the blue “Apply” button to show your segmented data in Google Analytics.

Apply and cancel button in Google Analytics
Apply and cancel button in Google Analytics – 2020

Step #7: Setting your date range

Make sure you select the correct date range that corresponds to your test planning otherwise you might miss out on data. You can find your test date range in Google Optimize.

Date range selector in Google Analytics
Date range selector in Google Analytics – 2020

Evaluating your data

Look at all that data! You can now already see some of the results of your test! You can choose to check out more of your segments to look at specific data or just be happy with the data you have now. 

Custom report with Google Optimize segments applied
Custom report with Google Optimize segments applied – 2020

But when has your test been successful for each of your segments? Because having a higher conversion rate isn’t enough information to decide this fact. Luckily a lot of companies and institutions have made ab testing calculators that make it extremely easy to evaluate your test.

My personal favorite is the one made by http://www.conversionxl.com. Their calculator helps you to answer 4 specific questions post-test to decide if your test was successful or not. I will run through the answers with my test data as an example. 

Does the test variant beat the original?

The variant beats the original because it had a lift of 14,29%. 

AB test calculator by ConversionXL
AB test calculator by ConversionXL – 2020

This coincides with the minimal detectable effect that is needed for this sample test. In one week you need at least a lift of 12.32% for statistical significance. Imagine that you only had 9% in one week of testing. This tells you should keep collecting more data to successfully finish your test. Just look at this table if you need to decide to stop or continue testing. I would probably stop my test if I had only 2% in two weeks. Better of testing something else.

Minimal detectable by ConversionXL
Minimal detectable by ConversionXL – 2020

Does the test have the needed sample size?

The test does have the need sample size as shown on the calculator. For these results, we needed about 13.000 sessions per variant and we are well over that. 

Sample size calculator by ConversionXL
Sample size calculator by ConversionXL – 2020

Does the test have the needed duration?

The test does have the needed duration as shown on the calculator because we needed 5 days of data. Our test data has 7 days of data. I always recommend testing in cycles of 7 days otherwise you might have polluted data. Every test will always act differently at the weekend for instance.

Duration calculator by ConversionXL
Duration calculator by ConversionXL – 2020

What is the monetary ROI of the test variant?

This one always makes people pay attention to. If you answered yes on the previous answers it will mean that implementing your test will probably have a return on investment. If you are an e-commerce website make sure to fill in your average order value for one non-control variant. 

Monthly monetary contribution in ab test calculator by ConversionXL
Monthly monetary contribution in ab test calculator by ConversionXL – 2020

For this test, it would be €31175 extra a month because of 1247 transactions with an order value of €25. These numbers really depend on the size of your business of course. The return on investment will be much lower on a website with less traffic and it will also be a lot harder to reach statistical significance. I recommend having a minimum of 10000 visitors a month before it’s even worth testing.

Extra transaction and monthly monetary contribution in ab test calculator by ConversionXL
Extra transaction and monthly monetary contribution in ab test calculator by ConversionXL – 2020

You can check out my test data live at this link.

Checking for significance

To make sure your test is successful you need to check it’s statistical significance. This basically means that the results of the test are most likely because of the changes you made. I will make a whole article about statistical significance in the future. Luckily you can just scroll down in the ab test calculator and look at the Bayesian and Z Test results. They will show you the probability of a variant is a winner and if the test was statically significant.

Results in in ab test calculator by ConversionXL
Results in in ab test calculator by ConversionXL – 2020

Definitely check out conversionxl.com if you want to know more about growth and testing in general.

Deciding your next steps

Depending on your results you need to decide if you want to implement the test on your website or if you want to keep the test running because you couldn’t answer yes to ab test calculator questions. You can also adjust the confidence level or statistical power of the calculator the get significance for lower levels and make higher risk implementations. I would recommend just keeping the standard settings if you want to play it safe.

Confidence level and statistical power in ab test calculator by ConversionXL – 2020

Want to learn more about CRO? Then check out this great course on Udemy.

Udemy logo

The NEW 2020 Complete Growth Hacking & Conversion Course

  • 5x your existing growth and conversion rate
  • Add 1,000s of leads and subscribers to your list
  • See where your visitors are dropping off and how to fix it
  • Bring cart abandoners back to your site and guarantee 10x ROI
  • Introduce virality into every aspect of your marketing campaigns
  • Price your product or service using 30+ rare techniques
  • Optimize the 1 goldmine on your site you probably pay no attention to
  • Create high-converting landing pages and CTA buttons
  • Test every element on your site for performance
  • Use sales psychology to increase transactions and your prices
  • Write copy that gets attention, clicks, and generates sales
  • Use screen recordings, heatmaps, and your visitors’ actions to increase profit
  • Make your pages load faster and improve your rankings in the search results

GrowthPenguin is reader-supported. When you buy through links on our site, we may earn an affiliate commission.

Conclusion

I hope this guide helped you to evaluate your Google Optimize ab test with Google Analytics. You should now be able to create segments in Google Analytics to dive deeper into the test data of Google Optimize. Next to that, you have some first time experience with evaluating your test for statistical significance and return on investment. If you have any questions feel free to ask them under this topic and I will answer all of them respectively. Good luck with your future tests!

Kind regards,

Jeroen Wiersma

GrowthPenguin

Leave a Reply

Your email address will not be published. Required fields are marked *