A Short Tutorial for Running A/B Tests in Shopify with Optimizely

Many store owners are reluctant to run A/B tests on their store, often because they know what they want to test, but aren't entirely sure about how to go about it. We're diving into testing with Optimizely on Shopify today, and going over the five key steps to take for a successful test.

Many store owners are reluctant to run A/B tests on their store, for a variety of reasons. One of the biggest reasons is not knowing where to start. Luckily, Jonathan Poma and I talked about this issue several weeks ago, so make sure to start there before reading this article.

Another big reason owners are reluctant to test, even if they know what they want to test, is because they aren’t entirely sure how to go about it. That’s what we’ll talk about today.

A quick note about testing tools: since Shopify handles your site hosting, you only need to worry about your testing platform. At Rocket Code, Optimizely is the platform of choice for our clients and our own website (full disclosure: Rocket Code is a one-star Optimizely Solutions Partner.) Optimizely offers robust testing tools that are easy enough for anyone to use, but also powerful enough to give you the data you need to make the right decisions.

When it comes to testing in Shopify with Optimizely, there are five key steps to take for a successful test: writing your code, setting up your test in Optimizely, previewing your test, letting it run, and cleaning up. In cases where your control — the original experience you’re testing against — wins, you may need to repeat these steps until one of your variations wins. These repetitions are called iterations, and they’re critical to the testing process as they help you truly optimize your store.

Let's take a look at this process, based on an experiment we did a few months ago with one of our clients, Chubbies Shorts (the shorts of choice in our office).

You might also like: How A/B Testing Will Make You a Better Web Designer

Step 0: Check yo’ self

Before we begin, a caveat about tracking goals: if you intend to track revenue, or other goals related to completing checkout, you or your client must be on a Shopify Plus plan, and your checkout domain name must be the same as the base domain of your store.

Using Chubbies Shorts as an example, since their store is on Shopify Plus, Optimizely can clearly tie the checkout page to the store, since their checkout resides at https://www.chubbieshorts.com/checkout

Were their site not on Plus, the checkout URL would look something like this:

https://checkout.shopify.com/123569/checkouts

Optimizely wouldn’t be able to tie the above to their store.

But in many cases, you may not need to tie revenue or checkout with a goal. For example, you may want to see if more people interact with a button or link that opens your size chart. There may be a revenue change based on that, but if you’re more worried about interactions with elements in your store, rather than if users are getting all the way through checkout, then you’ll do just fine with basic click goals.

Step 1: Make the code happen

Another caveat before we get started: you must do your initial work in a theme that is not live in your store; we'll call this the development theme. Using a development theme will allow you to make changes without accidentally affecting your users. To create one, go to the Themes admin area, click on the three dots next to your theme name, and select “Duplicate.” This will create a copy of your live theme that you can edit, without worrying about interrupting your user’s experience.

Typically, we like to keep as much of the experiment markup as possible locked within a single file; for Shopify this would be a Liquid snippet. For this nav experiment, we created a snippet called nav-tabs, and included the HTML and Liquid markup in there.

What you see above is an abbreviated version of the tabbed experiment snippet, to give you an idea of what the code looks like. This is all pretty standard Liquid templating, except for one line: if settings[tab-toggle]. This is a checkbox within your theme settings that defaults to false, and allows us to turn the experiment on and off from within the Shopify admin.

What’s great about this setup is that you can launch the code for an experiment into production, but not show it to users until the experiment is live in Optimizely, and turned on within the Shopify admin. This gives you greater control over when users might see this experiment; when you have control of the majority of code in Shopify, you have access to more targeted data about your visitors (such as if they’re a customer and what kind of template they’re seeing), which makes it easier to segment them, and select what experiments are exposed to.

That covers the markup. All of the functionality (JavaScript) and any styling (CSS) should also reside in your theme, if possible. We like to have as little JS and CSS as possible in the Optimizely editor. This helps keep our included Optimizely file as small as possible, since we may be running several experiments at a time.

We also recommend that if you don’t already have a process to build out your JavaScript and CSS files, put your experiment's JS and CSS into appropriately named JS and CSS files (such as tabs-experiment.js and tabs-experiment.css that are included within your theme.liquid. The JS and CSS are typically locked under a particular class attached to the <body> tag, also known as a “namespace.” This helps keep any particular functionality and styling invisible until your page has the correct class.

This approach can quickly become a bit overwhelming with the large number of snippets, .js files, and .css files in your theme.liquid and other templates. It’s therefore extremely important to remove these items once you’re ready to make an experiment variation the native experience on your site, especially if you’re running several experiments at a time. But don't worry, we'll touch on how to do this closer to the end.

You might also like: How to Successfully Execute an Iterative Redesign in Shopify with Optimizely

Step 2: Set it up in Optimizely

Setting up in Optimizely can be done in a number of ways, and depends on what kind of test you’re running; multivariate, redirect, and so forth. For this demonstration, we’ll assume it’s a simple A/B test, as it encompasses most of the same involved in multivariate and redirect tests.

Once you’re in Optimizely and have set up your goals and traffic allocation, it’s dead simple to implement and see your experiment. (The control is, well, the control so you don't need to do anything with it.)

The first variation (and any other variation if you’re running a multivariate test) is where you would start adding code for the experiment, specifically where you click on the "Edit Code" button.

 

As you can see in the images above, we’ve simply instructed Optimizely to add a class to the body for that variation. Once the class is added, the CSS we wrote earlier will trigger, and voila!

Step 3: Preview your test

Previewing is a bit tricky with Optimizely and a development theme. First, get the preview link for your theme, which you can find by clicking on the eye icon in your Theme’s admin area:

Then, you’ll need to add something extra to the URL, to tell Optimizely you want to preview this experiment. You can find the nitty-gritty of how this works here, but to boil it down, each experiment in Optimizely has an ID you’ll need, along with the variation you want to look at: zero is the control, one is the first variation, two is the second variation, and so on.

Once you put it all together, it will look something like this:

http://chubbiesshorts.com/?preview_theme_id=123076935&optimizely_x12345678901=1

?preview_theme_id=123076935 is what tells Shopify to show the theme (with an ID of 123076935) you had been working on, and optimizely_x12345678901=1 is the experiment ID (12345678901) and variant number (1) from Optimizely.

From this link, you can interact with the experiment on your site just as a user would see things. Make sure to test your entire purchase flow before moving on to the next step, so you can be confident you’ve changed only the parameters of your experiment, and that the rest of your store is functioning as expected.

Step 4: Let it run (in the) wild

At this point, all you need to do is activate your development theme to make it your live theme, and then start the experiment in Optimizely. This step, oddly enough, is both the easiest and the most daunting. Depending on your store's daily traffic, and how much of that traffic you’re allocating to the experiment, it can take a few days or a few weeks to gather enough data to get to statistical significance.

Step 5: Go native (in your theme)

Once your experiment has reached statistical significance, you should have the confidence to know that the data from the experiment is conclusive, usually within an extremely small margin of error. Assuming one of your variations won over the control, it’s time to make that variation native to your theme. Luckily, the setup you did beforehand makes this much simpler, and gives you two options.

The first option is the simplest: just add the class you added to the body in the Optimizely experiment editor, to the <body> in your theme.liquid. This option is best for a quick turnaround, while you wait to perform a more concerted cleanup effort, or if you know you’ll want to remove that functionality in the near future without having to switch themes. The drawback of this option is that if you’re not careful, your theme will quickly become cluttered with obsolete or even completely overridden classes, functionality, and styles.

The second option is to first remove the namespacing you added in step one, and then clear out any functionality and/or styling you no longer need. You can then move on to remove any particular logic that might be in the markup, replace the markup that lost the test with your new markup, and remove the snippet from your theme.

Cleaning up your theme in this way is a best practice you should consider doing several times a year — if not every time you update your theme. A clean theme with only the functionality, styling, and markup you need helps keep your store fast and your conversions up. The only downside here is that it takes more time to implement; you need to be extra vigilant in testing your theme to make sure you’ve removed only what you needed to, and didn’t accidentally touch another portion of the page or site.

In the case that your control wins, it’s still worth cleaning up your theme. After that, you can start iterating; tweaking your original hypothesis and adjusting the changes you make, to determine if you can ask a better question or come up with a better solution to the problem. Start back at step one, and keep going through this cycle until you’ve optimized to the best of your ability.

Summary

1. Write the code for your experiment in a duplicated version of your live theme.
2. Set up your goals, traffic, and other test parameters in Optimizely.
3. Preview your test and run through the shopping experience.
4. Run the test with your new theme, resisting the urge to tweak it.
5. Read the results, and either iterate if the control won or push your variation live.
6. Always be iterating. Go back to one.

Okay, I lied. There are six steps. You should always be iterating and experimenting to some degree, as trends and user habits are constantly evolving.

Now that you know the big picture of what to test, and how to set up your test with Shopify and Optimizely, you have gained a very powerful set of skills to help boost conversions on your store. In our next and final installment, we’ll cover how to execute iterative redesigns via Optimizely.

You might also like: Why, How, and When to Utilize Usability Testing