Did you know it’s possible to prepare and run tests on a zero budget?
This chapter will tell you how.
It contains all the information you need without spending a dime!
Step 1: How do you prepare your testing tools?
In the first step, we are going to set up the tools you need.
What tools do I use?
The tools that I use are Google Optimize and Google Analytics. They cost nothing and get the job done.
Optimize is a testing solution for companies with a smaller budget. It’s not perfect and will need some workarounds to get it working the way we want.
After implementing these workarounds, this tool will be perfect for the job at hand.
Google Analytics is software that helps you measure and analyze website behavior. It’s one of the most popular solutions out there. We will need it to extract data from Google Optimize to Google Sheets.
We will use Analytics Toolkit to make our test designs, which we can create with our exported data from Google Sheets. The trial is 14 days and doesn’t require a credit card.
Now let’s set up all pieces of software. We will start with Google Analytics.
How to set up Google Analytics?
There are many ways of implementing Google Analytics. All need you to make an account.
Once you have an account, you will need to connect your website to Google Analytics. If the website is using WordPress, you can install a plugin to make the connection.
You can also use Google Tag Manager to make the connection. Another method would be implementing Google Analytics with a global site tag.
After implementing Google Analytics, you will need to set up e-commerce tracking to track conversions for your experiments.
Next up is Google Optimize.
How to set up Google Optimize?
Start by getting the Google Optimize plugin for chrome to edit your tests visually and debugging.
Then you can install Google Optimize and link it to Google Analytics.
We need to implement Google Optimize in a certain way to make sure you don’t get a flicker effect.
The flicker effect is when a user sees the control version before the variant loads. This effect will make your test inconclusive because the group will be aware of the change.
Imagine looking at an add-to-cart button that reads “buy now” and having it change to “buy today.” This occurrence might influence your behavior.
To prevent this, we are going to connect Google Optimize to our website with an asynchronous snippet.
You must implement this script at the top of your <head> tag before anything else loads to prevent flickering.
How to set up Analytics Toolkit?
You need to sign up for a trial version by setting up an account. This software includes all the tools you need for A/B testing preparation, monitoring, and analysis. In this guide, we will be using the tool to create a test design.
The trial is for 14 days. If you want to support the company, you can buy a subscription with your winning tests.
Now you have the tools to start running tests.
Step 2: How to run tests?
I will describe this step by using a fictional example case to help you understand it better.
How to choose your hypothesis?
If you have been following this series, you will have a bunch of hypotheses prioritized in the framework of your choosing.
For this guide, the following example had the highest prioritization score:
“Because we saw in our research that currency symbols are a pain point for visitors, we expect that removing the currency symbols for all visitors will cause an increase in conversion rate. We expect to see a 2% conversion rate increase over four weeks.
Let’s start by making a test design.
How to make a test design?
Before creating a test in Optimize, we want to make a test design in Analytics Toolkit.
The test design will help us decide when to stop and continue our tests. I found a video that will help you set it up. When creating a new test design, you need to keep in mind that you have two slots in the trial version.
We will use one test slot for your test’s mobile version and the other one for the desktop & tablet version.
It’s crucial to segment these devices to prevent device pollution. Mobile and desktop devices are too different from each other. Combining them would invalidate your results.
We are setting up the mobile version.
Go to the A/B testing calculator.
Then scroll down and create a new test.
I like to use a standard naming convention to prevent confusion in the future.
Test (number) – (description) – (devices)
In our example, this would create the following experience name:
Test 1 – Removing currency symbols – Mobile
Then choose the manual data source.
We will test for superiority because we want to know if our test is better than the control.
The data type will be binomial because we are using transactions as our conversions.
After that, we need the baseline conversion rate of the page or event you will test.
You must make an accurate assumption of the conversion rate because you will not be able to change it later.
There is one manual way of exporting users the right way from Google Analytics without sample pollution.
It involves turning off this Enable User Metrics in reporting toggle in Google Analytics:
You can find in Admin > Property > Property Settings > User Analysis.Then you will need to make a custom report where we can see the number of users that have visited on your specific page of that given period. We are using users because sessions can skew your results.
In this example, I would be looking for the number of users on the amazon.com homepage. I recommend looking at the last four weeks of data for your baseline conversion analysis.
Then get the transactions from the Conversion > E-commerce > Overview report.
Did you get a green shield for both reports? If not, your data might be sampled and has the risk of skewing your results.
To get the minimum effect of interest, we need to use the statistical calculators on Analytics Toolkit.
We want to use the Effect Size & Power calculator.
Fill in the sample size of 4 weeks of users that you gathered earlier. Then input the baseline conversion rate and hit calculate.
The graph will give you the minimum effect of interest marked blue in the picture below.
Input that number into the test design creator.
You can keep the confidence level at 95% because this is an industry-standard. It will give you a 1 in 20 chance that your results are due to chance.
We will keep the chance to detect effect at 80% because increasing this would mean you need more traffic. You will have a 1 in 5 chance to miss the result of a significant positive or negative change, which is acceptable if the costs are low.
I recommend testing one variant at a time because increasing the variant will significantly increase your testing time. It’s better to test the next variant in a follow-up test.
Want want to make three interim analyses and one final analysis for four weeks of testing.
We will keep the advanced setting as they are.
Now click on evaluate design.
You will generate your test design beneath the form you just filled in. It will show you the sample size you need to get the results you want. If this does not correlate with your traffic, you need to recalculate the minimal effect of interest and baseline conversion rate.
It will also generate this graph, which we will input data into to figure out when to stop or continue testing.
Save the test design if you think the results needed are realistic.
Every seven days, you will need to input your results here.
How to create a new test?
We will need to create an A/B test in Google Optimize to test our changes.
This specific test will run on all pages with currency symbols on them, so choosing an editor page will be easy.
We will use Amazon as an example. The homepage of Amazon will be my editor page.
How to create your variant?
Give the variant a name that indicates what is changing. In this example, we are removing the currency symbols. I usually combine the test and segment the devices later if the changes are the same, thus creating a universal test name.
Then you can edit your variant with all the changes you would like.
I like to create my testing variants using JavaScript as it allows me to change everything I need without calling a developer.
You will also have a lot more testing options than just using the visual editor. It’s impossible to change every dynamic currency symbol using the visual editor solely.
A simple JavaScript snippet will be perfect for this test:
//get all the titles that might have dollar signs in them
let titles = document.querySelectorAll("div.a-section.as-title-block > h2 > span");
//remove all dollar signs
for(title of titles){
let newTitle = title.innerHTML.replace(/\$/g, '');
title.innerHTML = newTitle;
}
Before applying the script – Control
After applying the script – Variant
You can try out a script by running it in the console. Go to amazon.com and right-click to inspect.
Clicking inspect will open the console in Google Chrome and will allow you to input JavaScript next to the blue arrow.
Copy the script in and press enter to run the changes. If Amazon has not updated their website, the script will remove any dollar signs on the homepage titles.
The above code is an example script and will not work on your own website. If you don’t know how to write JavaScript code, you can ask a developer to do it for you.
Because we are loading our script before the DOM loads, we need to run our script after all the elements we want to change are visible.
I like to choose an element that is on every page and add the script there. Something like the footer will do. Hover over the elements in the visual editor and choose the element you want.
Click on the edit element button and run JavaScript.
Now you can add your JavaScript code in the field and hit apply.
I learned to code JavaScript by using W3schools and taking a 6-month course. Learning JavaScript isn’t something you can do overnight, so make it a long-term personal goal if you want to become a flexible optimizer.
If you don’t have a developer nearby, you can focus on tests that don’t require complex changes.
An example of this would be changing the copy on the homepage of a website.
Once you have your changes setup, you will need to test if they are working before starting the test.
Check all the devices with the preview option to see if your variant is applied correctly.
You can also share the preview with a developer or other stakeholders to show them the test’s implementation before running them.
Share your tests because you are bound to make small mistakes when you are starting.
How to set up your targeting?
The targeting of your tests decides where the test runs. You have to create rules for the URL that you want to target.
We want to run our test on the homepage, so a simple “ends with” rule makes sure that we target the URL we need.
Sometimes you will also have to run tests on many different URLs with the same PageType if you want to change something on a product page.
Using these methods requires you to know and write regular expressions or send PageType data with Google Tag Manager. You can use these techniques to target the pages you need for the test.
Use the audience targeting section to target PageType and demographic segments.
I find PageType especially useful as it makes creating a test design very easy. You can use this to make a segment in Google Analytics and look at the traffic for that PageType.
How to connect Google Analytics?
If you set up Google Analytics the right way, you can connect it to your test. Just select the view you are using from the drop-down menu and make the connection. If you don’t see the view, you have not connected Google Optimize to Google Analytics yet.
How to set up your goals?
When running tests with Optimize, we need to define our goal metric. Our goal will usually be something like e-commerce transactions, as you would want to increase your conversion rate.
If you don’t see transactions, you have not setup your e-commerce tracking.
How to validate your installation?
To make sure that you have installed Google Optimize correctly, you will need to validate your test.
Optimize recommends doing this for every test you run because many factors can upset your installation.
An example of this would be a failed installation on the product page, but it works fine on the homepage. You will need to have set up the script and install the chrome extension for the validation to pass their check.
How to set up email notifications?
Just flick the grey switch to activate e-mail notifications about your test. I recommend enabling this as this reminds you when tests are about the end.
How to set up traffic allocation?
Just keep it at 100% because a lower setting will pollute your samples. It would also take longer to complete a test because you would get less traffic.
How to trigger your test?
In this example, you can trigger the test on page load as we have accounted for this problem with our script placement. So if you are not trying to change something at an event, you can go to the next step.
Sometimes you want to change something on a page when something happens.
What if you want to change how the add-to-cart overlay looks after someone adds a product to their shopping cart?
To achieve event activation of your test, you will need to send an activation event with Google Tag Manager.
By running this script at the same that an add to cart happens, you will be able to trigger the test:
dataLayer.push ({'event': 'optimize.activate'});
I recommend sending this event to Google Analytics beforehand because you will need the traffic data for your test design.
How to set up a schedule?
Now scroll up and click the schedule icon.
I recommend basing the time you are testing on your test design. If you analyzed for four weeks of testing, you should schedule for four weeks.
Set up a starting time where you will be working to check if everything is running correctly when the test starts.
When you schedule for four weeks of testing time, it doesn’t necessarily mean that you will be running it for that time, so make sure your next test is ready to run before that.
How to debug your test?
Before starting our test, we need to launch the debugger. This tool will allow us to check your targeting settings.
It looks like everything is ready for our test.
How to start your test?
Click start at the top right corner of your test.
Then click start again to start your test. The experiment will be applied when the beginning of the schedule is reached.
How to check if it’s running?
Let’s check if everything is working on the live version of your site just to make sure. In our example, this would be the homepage of Amazon.
There is a 50% chance that you don’t see the changes because Optimize has put you in the control group. You could open up many browsers to get the version you want, but this can be annoying if you are unlucky.
We can use the chrome browser console to run some code to find out what experiment is running.
var propertyId = "UA-227286-32";
var experimentId = Object.keys(gaData[propertyId].experiments)[0];
var variationId = gaData[propertyId].experiments[experimentId];
console.log("The experiment ID is " + experimentId + " and the variant is " + variationId);
You will need to change the property id to match it to your own.
You can find it by going into Google Optimize and clicking on Settings in the dashboard.
The settings will contain what your property ID is:
Run your code in the console and look at the response.
If you get a response, you will know that the experiment is running. If the variant is 0, you are looking at the control. If the variant is 1, you are looking at the variant.
How can we change this?
We need to go to the application tab in the developer console.
Then look for cookies:
Look for the domain that you are running your test on and click that. Here you can see all the first-party cookies.
Type “gaex” in the search bar.
As you can see, we get a cookie that contains a value with our experiment ID. The last number is 0. Zero means we are getting the control.
By double-clicking the value and changing the last value to 1, we can get the different variants.
Now refresh the page, and you have the requested variant!
You can rerun the code to verify that the experiment is running and you are looking at the correct variant.
Now it’s the waiting game!
When to stop your test?
Because you made a test design in Analytics Toolkit, you need to input your data every seven days for analysis.
I added one data point for visualization. As you can see, it recommends continuing testing because the blue data point is between the green and red sectors. The green sector would signal that you have a winner to declare. The red sector means that it’s probably not worth it to keep testing anymore.
A test design is primarily used for interim decision-making and can be skipped by just evaluating the test after four weeks. Skipping this process will decrease your testing velocity and make it a longer journey to get your first win.
Now you are ready to prepare and run tests on a zero budget! The next step is interpreting your ab testing results when stopping your test.
Next chapter
Chapter #4: How to interpret your ab testing results when stopping your test?
You input your observation data into the test design and notice you need to end your test.
Let’s finish our test the right way.
This chapter will show you how to interpret your ab testing results and get the most out of your tests.
This chapter contains:
- How to interpret your ab testing results?
- How to calculate the monetary contribution?
- How to log your results?
- How to share your learnings?
- How to decide on your next test?
- How to increase your budget after your first win?