Categories
CRO

Chapter #3: How to prepare and run tests on a zero budget?

Did you know it’s possible to prepare and run tests on a zero budget?

This chapter will tell you how.

It contains all the information you need without spending a dime!

Step 1: How do you prepare your testing tools?

In the first step, we are going to set up the tools you need.

What tools do I use?

The tools that I use are Google Optimize and Google Analytics. They cost nothing and get the job done.

Optimize is a testing solution for companies with a smaller budget. It’s not perfect and will need some workarounds to get it working the way we want.

After implementing these workarounds, this tool will be perfect for the job at hand.

Google Analytics is software that helps you measure and analyze website behavior. It’s one of the most popular solutions out there. We will need it to extract data from Google Optimize to Google Sheets.

We will use Analytics Toolkit to make our test designs, which we can create with our exported data from Google Sheets. The trial is 14 days and doesn’t require a credit card.

Now let’s set up all pieces of software. We will start with Google Analytics.

How to set up Google Analytics?

Google Analytics on a laptop
(Image source)

There are many ways of implementing Google Analytics. All need you to make an account.

Once you have an account, you will need to connect your website to Google Analytics. If the website is using WordPress, you can install a plugin to make the connection.

You can also use Google Tag Manager to make the connection. Another method would be implementing Google Analytics with a global site tag.

After implementing Google Analytics, you will need to set up e-commerce tracking to track conversions for your experiments.

Next up is Google Optimize.

How to set up Google Optimize?

Google Optimize on a laptop
(Image source)

Start by getting the Google Optimize plugin for chrome to edit your tests visually and debugging.

Then you can install Google Optimize and link it to Google Analytics.

We need to implement Google Optimize in a certain way to make sure you don’t get a flicker effect. 

The flicker effect is when a user sees the control version before the variant loads. This effect will make your test inconclusive because the group will be aware of the change.

Imagine looking at an add-to-cart button that reads “buy now” and having it change to “buy today.” This occurrence might influence your behavior.

To prevent this, we are going to connect Google Optimize to our website with an asynchronous snippet.

You must implement this script at the top of your <head> tag before anything else loads to prevent flickering.

How to set up Analytics Toolkit?

Analytics tookit logo
(Image source)

You need to sign up for a trial version by setting up an account. This software includes all the tools you need for A/B testing preparation, monitoring, and analysis. In this guide, we will be using the tool to create a test design.

The trial is for 14 days. If you want to support the company, you can buy a subscription with your winning tests.

Now you have the tools to start running tests.

Step 2: How to run tests?

Running a usability test
(Image source)

I will describe this step by using a fictional example case to help you understand it better.

How to choose your hypothesis?

If you have been following this series, you will have a bunch of hypotheses prioritized in the framework of your choosing.

For this guide, the following example had the highest prioritization score:

“Because we saw in our research that currency symbols are a pain point for visitors, we expect that removing the currency symbols for all visitors will cause an increase in conversion rate. We expect to see a 2% conversion rate increase over four weeks.

Let’s start by making a test design.

How to make a test design?

Test design
(Image source

Before creating a test in Optimize, we want to make a test design in Analytics Toolkit.

The test design will help us decide when to stop and continue our tests. I found a video that will help you set it up. When creating a new test design, you need to keep in mind that you have two slots in the trial version.

Analytics toolkit subscription plan
(Image source)

We will use one test slot for your test’s mobile version and the other one for the desktop & tablet version. 

It’s crucial to segment these devices to prevent device pollution. Mobile and desktop devices are too different from each other. Combining them would invalidate your results.

We are setting up the mobile version.

Go to the A/B testing calculator.

A/B testing calculator in menu
(Image source)

Then scroll down and create a new test.

Creating a new test in analytics toolkit
(Image source)

I like to use a standard naming convention to prevent confusion in the future.

Test (number) – (description) – (devices)

In our example, this would create the following experience name:

Test 1 – Removing currency symbols – Mobile

Naming your test design
(Image source)

Then choose the manual data source.

Data source from your test design
(Image source)

We will test for superiority because we want to know if our test is better than the control.

Test type from your test design
(Image source)

The data type will be binomial because we are using transactions as our conversions.

Data type from your test design
(Image source)

After that, we need the baseline conversion rate of the page or event you will test.

Baseline conversion rate from your test design
(Image source)

You must make an accurate assumption of the conversion rate because you will not be able to change it later.

There is one manual way of exporting users the right way from Google Analytics without sample pollution.

It involves turning off this Enable User Metrics in reporting toggle in Google Analytics:

Enable users metric in reporting button
(Image source)

You can find in Admin > Property > Property Settings > User Analysis.Then you will need to make a custom report where we can see the number of users that have visited on your specific page of that given period. We are using users because sessions can skew your results.

Report table exporting using from Google Analytics
(Image source)

In this example, I would be looking for the number of users on the amazon.com homepage. I recommend looking at the last four weeks of data for your baseline conversion analysis. 

Then get the transactions from the Conversion > E-commerce > Overview report.

E-commece overview report in Google Analytics
(Image source)
Transactions in Google Analytics
(Image source)

Did you get a green shield for both reports? If not, your data might be sampled and has the risk of skewing your results.

Green shield in Google Analytics
(Image source)

To get the minimum effect of interest, we need to use the statistical calculators on Analytics Toolkit.

Statistical calculator in Analytics toolkit
(Image source)

We want to use the Effect Size & Power calculator.

Effect size & power calculator in Analytics toolkit
(Image source)

Fill in the sample size of 4 weeks of users that you gathered earlier. Then input the baseline conversion rate and hit calculate.

The graph will give you the minimum effect of interest marked blue in the picture below.

Minimun effect of interest in Analytics Toolkit
(Image source)

Input that number into the test design creator.

Minimum effect of interest
(Image source)

You can keep the confidence level at 95% because this is an industry-standard. It will give you a 1 in 20 chance that your results are due to chance.

Confidence level
(Image source)

We will keep the chance to detect effect at 80% because increasing this would mean you need more traffic. You will have a 1 in 5 chance to miss the result of a significant positive or negative change, which is acceptable if the costs are low.

Chance to detect effect
(Image source)

I recommend testing one variant at a time because increasing the variant will significantly increase your testing time. It’s better to test the next variant in a follow-up test.

Number of variants
(Image source)

Want want to make three interim analyses and one final analysis for four weeks of testing.

Number of analyses
(Image source)

We will keep the advanced setting as they are.

Advanced parameters
(Image source)

Now click on evaluate design.

Evaluate design button
(Image source)

You will generate your test design beneath the form you just filled in. It will show you the sample size you need to get the results you want. If this does not correlate with your traffic, you need to recalculate the minimal effect of interest and baseline conversion rate.

Sample size prediction
(Image source)

It will also generate this graph, which we will input data into to figure out when to stop or continue testing.

Sample size per variant
(Image source)

Save the test design if you think the results needed are realistic.

Save test design button
(Image source)

Every seven days, you will need to input your results here.

Entering new observations in Analytics Toolkit.
(Image source)

How to create a new test?

Creating a new test in Optimize
(Image source)

We will need to create an A/B test in Google Optimize to test our changes.

This specific test will run on all pages with currency symbols on them, so choosing an editor page will be easy.

We will use Amazon as an example. The homepage of Amazon will be my editor page.

How to create your variant?

Creating a new variant in Google Optimize
(Image source)

Give the variant a name that indicates what is changing. In this example, we are removing the currency symbols. I usually combine the test and segment the devices later if the changes are the same, thus creating a universal test name.

Naming the variant in Google Optimize
(Image source)

Then you can edit your variant with all the changes you would like.

Editing your variant.
(Image source)

I like to create my testing variants using JavaScript as it allows me to change everything I need without calling a developer.

You will also have a lot more testing options than just using the visual editor. It’s impossible to change every dynamic currency symbol using the visual editor solely.

A simple JavaScript snippet will be perfect for this test:

//get all the titles that might have dollar signs in them
let titles = document.querySelectorAll("div.a-section.as-title-block > h2 > span");

//remove all dollar signs
for(title of titles){
  let newTitle = title.innerHTML.replace(/\$/g, '');
  title.innerHTML = newTitle;
}

Before applying the script – Control

Control image
(Image source)

After applying the script – Variant

Variant image
(Image source)

You can try out a script by running it in the console. Go to amazon.com and right-click to inspect.

Inspect in chrome browser
(Image source)

Clicking inspect will open the console in Google Chrome and will allow you to input JavaScript next to the blue arrow.

Console in chrome browser
(Image source)

Copy the script in and press enter to run the changes. If Amazon has not updated their website, the script will remove any dollar signs on the homepage titles.

Change in Google Chrome console
(Image source)

The above code is an example script and will not work on your own website. If you don’t know how to write JavaScript code, you can ask a developer to do it for you. 

Because we are loading our script before the DOM loads, we need to run our script after all the elements we want to change are visible.

Selecting the footer element in Google Optimize
(Image source)

I like to choose an element that is on every page and add the script there. Something like the footer will do. Hover over the elements in the visual editor and choose the element you want.

Running JavaScript code in Google Optimize
(Image source)

Click on the edit element button and run JavaScript.

Adding JavaScript code in Google Optimize
(Image source)

Now you can add your JavaScript code in the field and hit apply.

I learned to code JavaScript by using W3schools and taking a 6-month course. Learning JavaScript isn’t something you can do overnight, so make it a long-term personal goal if you want to become a flexible optimizer.

If you don’t have a developer nearby, you can focus on tests that don’t require complex changes.

An example of this would be changing the copy on the homepage of a website. 

Once you have your changes setup, you will need to test if they are working before starting the test.

Check all the devices with the preview option to see if your variant is applied correctly.

Previewing your variants
(Image source)

You can also share the preview with a developer or other stakeholders to show them the test’s implementation before running them.

Share your tests because you are bound to make small mistakes when you are starting.

How to set up your targeting?

Setting up targeting in Google Optimize
(Image source)

The targeting of your tests decides where the test runs. You have to create rules for the URL that you want to target.

Url targeting in Google Optimize
(Image source)

We want to run our test on the homepage, so a simple “ends with” rule makes sure that we target the URL we need.

Sometimes you will also have to run tests on many different URLs with the same PageType if you want to change something on a product page.

Using these methods requires you to know and write regular expressions or send PageType data with Google Tag Manager. You can use these techniques to target the pages you need for the test.

Audience targeting in Google Optimize
(Image source)

Use the audience targeting section to target PageType and demographic segments.

I find PageType especially useful as it makes creating a test design very easy. You can use this to make a segment in Google Analytics and look at the traffic for that PageType.

How to connect Google Analytics?

Connecting Google Analytics with Google Optimize
(Image source)

If you set up Google Analytics the right way, you can connect it to your test. Just select the view you are using from the drop-down menu and make the connection. If you don’t see the view, you have not connected Google Optimize to Google Analytics yet.

How to set up your goals?

Setting up goals in Google Optimize
(Image source)

When running tests with Optimize, we need to define our goal metric. Our goal will usually be something like e-commerce transactions, as you would want to increase your conversion rate.

Choosing an objective in Google Optimize
(Image source)

If you don’t see transactions, you have not setup your e-commerce tracking.

How to validate your installation?

To make sure that you have installed Google Optimize correctly, you will need to validate your test.

Validating your installation in Google Optimize
(Image source)

Optimize recommends doing this for every test you run because many factors can upset your installation. 

An example of this would be a failed installation on the product page, but it works fine on the homepage. You will need to have set up the script and install the chrome extension for the validation to pass their check.

How to set up email notifications?

Just flick the grey switch to activate e-mail notifications about your test. I recommend enabling this as this reminds you when tests are about the end.

Setting up e-mail notifications in Google Optimize
(Image source)

How to set up traffic allocation?

Just keep it at 100% because a lower setting will pollute your samples. It would also take longer to complete a test because you would get less traffic.

Setting up traffic allocation in Google Optimize
(Image source)

How to trigger your test?

In this example, you can trigger the test on page load as we have accounted for this problem with our script placement. So if you are not trying to change something at an event, you can go to the next step.

Triggering your test in Google Optimize
(Image source)

Sometimes you want to change something on a page when something happens.

What if you want to change how the add-to-cart overlay looks after someone adds a product to their shopping cart?

Activating your test with an event in Google Optimize
(Image source)

To achieve event activation of your test, you will need to send an activation event with Google Tag Manager.

By running this script at the same that an add to cart happens, you will be able to trigger the test:

dataLayer.push ({'event': 'optimize.activate'});

I recommend sending this event to Google Analytics beforehand because you will need the traffic data for your test design.

How to set up a schedule?

Now scroll up and click the schedule icon.

Schedule button in Google Optimize
(Image source)

I recommend basing the time you are testing on your test design. If you analyzed for four weeks of testing, you should schedule for four weeks.

Start and end time in Google Optimize schedule dashboard
(Image source)

Set up a starting time where you will be working to check if everything is running correctly when the test starts.

When you schedule for four weeks of testing time, it doesn’t necessarily mean that you will be running it for that time, so make sure your next test is ready to run before that.

How to debug your test?

Before starting our test, we need to launch the debugger. This tool will allow us to check your targeting settings.

Debugging in Google Optimize
(Image source)

It looks like everything is ready for our test.

Debugging in Google Optimize
(Image source)

How to start your test?

Click start at the top right corner of your test.

Starting your test in Google Optimize
(Image source)

Then click start again to start your test. The experiment will be applied when the beginning of the schedule is reached.

How to check if it’s running?

Let’s check if everything is working on the live version of your site just to make sure. In our example, this would be the homepage of Amazon.

There is a 50% chance that you don’t see the changes because Optimize has put you in the control group. You could open up many browsers to get the version you want, but this can be annoying if you are unlucky.

We can use the chrome browser console to run some code to find out what experiment is running.

var propertyId = "UA-227286-32";
var experimentId = Object.keys(gaData[propertyId].experiments)[0];
var variationId = gaData[propertyId].experiments[experimentId];
console.log("The experiment ID is " + experimentId + " and the variant is " + variationId);

You will need to change the property id to match it to your own. 

You can find it by going into Google Optimize and clicking on Settings in the dashboard.

Setting in Google Optimize
(Image source)

The settings will contain what your property ID is:

Property ID in Google Optimize
(Image source)

Run your code in the console and look at the response.

Running a experiment check for Google Optimize in the Google Chrome browser
(Image source)

If you get a response, you will know that the experiment is running. If the variant is 0, you are looking at the control. If the variant is 1, you are looking at the variant.

How can we change this?

We need to go to the application tab in the developer console.

Application tab in Google chrome
(Image source)

Then look for cookies:

Looking for your cookie domain
(Image source)

Look for the domain that you are running your test on and click that. Here you can see all the first-party cookies.

Type “gaex” in the search bar.

Looking up the optimize cookie.
(Image source)

As you can see, we get a cookie that contains a value with our experiment ID. The last number is 0. Zero means we are getting the control.

By double-clicking the value and changing the last value to 1, we can get the different variants.

Changing the optimize cookie
(Image source)

Now refresh the page, and you have the requested variant!

You can rerun the code to verify that the experiment is running and you are looking at the correct variant.

Now it’s the waiting game!

When to stop your test?

Because you made a test design in Analytics Toolkit, you need to input your data every seven days for analysis.

I added one data point for visualization. As you can see, it recommends continuing testing because the blue data point is between the green and red sectors. The green sector would signal that you have a winner to declare. The red sector means that it’s probably not worth it to keep testing anymore.

Evaluating your test design in Analytics Toolkit
(Image source)

A test design is primarily used for interim decision-making and can be skipped by just evaluating the test after four weeks. Skipping this process will decrease your testing velocity and make it a longer journey to get your first win.

Now you are ready to prepare and run tests on a zero budget! The next step is interpreting your ab testing results when stopping your test.

Next chapter

Chapter #4: How to interpret your ab testing results when stopping your test?

You input your observation data into the test design and notice you need to end your test. 

Let’s finish our test the right way. 

This chapter will show you how to interpret your ab testing results and get the most out of your tests.

This chapter contains:

  • How to interpret your ab testing results?
  • How to calculate the monetary contribution?
  • How to log your results?
  • How to share your learnings?
  • How to decide on your next test?
  • How to increase your budget after your first win?

Get the full guide below!

How to win your first a/b test on a zero budget?

If you read this guide of 79 pages, you will learn:

Are your tests statistically valid?

This statistics course for A/B testing will help you avoid costly testing mistakes.