Split-Testing: How to Run a Revenue-Driven Test in Shopify

Most designers and developers know conversion optimization is key, and can see how split-testing near-instantly turns an online store around. Yet it isn’t entirely clear to many how to actually perform a revenue-driven split-test in Shopify. Today, we'll show you, step-by-step, how to test anything in Shopify stores to inject profit into a client’s bottom-line.

by Joshua Uebergang

Last updated 21 Jul 2016

Split-testing makes you a better designer because it’s the ultimate litmus test of revenue, handling client disputes, and satisfying clients (you can’t argue with a statistically proven 12 per cent increase in revenue).

And while tests can be done on micro-conversions, like form engagements or click-throughs, these types of actions are not always correlated with revenue.

I could claim to offer visitors a free Ferrari if they visited the cart page. A lot more people would get one-step closer to the purchase, but fewer would convert due to the distraction, deception, and ultimate disappointment.

Micro-conversions should be tracked but optimizations need to be revenue-driven. Profit is the essence of ecommerce, and few retailers are optimized enough to warrant attention on micro-conversions.

But how do you actually perform a revenue-driven split-test in Shopify?

We know conversion optimization is key, and can see how split-testing near-instantly turns around an unprofitable store. Yet it isn’t entirely clear to many designers and agencies how to A/B test a new product page, or how to test an entire theme.

Yet it can all be done.

Today, I’ll show you, step-by-step, how I test anything in Shopify stores to inject profit into a client’s bottom-line.

You might also like: How to Successfully Execute an Iterative Redesign in Shopify with Optimizely

The challenge with testing in Shopify

It is technically simple to run split-tests of visual elements on Shopify Plus stores. If you want to test “Add To Bag” text on the button of your product page against “Add To Cart”, you create a jQuery that selects a unique CSS identifier inside your favorite split-testing tool. There is little difference to testing within Shopify Plus compared to most websites because you have full control over the checkout funnel.

The challenge with testing in Shopify is with the majority of merchants who use the checkout.shopify.com domain, and don’t have the required control over the checkout funnel. This means it can be hard (or impossible in some tools) to test for revenue.

It all has to do with cross-domain tracking with little munchies called “cookies.” Testing tools set a first-party cookie, associated with the store’s domain. Due to browser restrictions, these cookies do not exist outside the domain name. Once a visitor converts on the Shopify checkout page, tools cannot link them back to the experiment.

As we’ve covered, revenue is critical to track. Safari is the primary browser known to block third-party cookies by default. Browsers that do not automatically block third-party cookies, allow users the option to block them. Visitors are tracked on your domain, but their conversions outside of your domain are lost on browsers that do not support these cookies.

This means you have to exclude a portion of users from the experiment right off the bat (namely Safari users), which can introduce a bias to your results. If you view the “Tech” reports in Google Analytics, most stores will have browsers with different conversion performance:

If you do not exclude these browsers from the experiment, your results can be skewed by those users who entered the test and converted, but were not tracked because the third-party cookies were not available at the time of their conversion. Therefore, the conversion rate reflected in your results would not be accurate.

Even if browsers we know block third-party cookies and are excluded from the experiment, visitors using browsers like Chrome and Firefox ,or ad blockers, may block third-party cookies in their settings. Since we have no way to determine this, the conversions from users with this setting enabled, will not be tracked even though they were initially included in the experiment.

The best we can do is run a large number of conversions per variation (300 minimum) and integrate the data with analytics, (which we will cover) so a breakdown can be done.

Finally, a smart store will have an abandoned campaign that markets to well-qualified visitors. Abandoned cart emails present an unavoidable factor you should be aware of, which may not convert in the split-test. You will get sessions but no tracked conversions.

The primary reasons for this are multi-device swapping (someone enters the test, abandons cart, then clicks the abandoned email on another device and buys) or email campaigns that take visitors to resume orders at checkout.shopify.com (stores with applications like Klaviyo do this).

Step-by-step on creating a split-test

The first step is to identify what to optimize. This is a balance between what is easiest and what you estimate will create the greatest profit.

Sometimes the best path to 50 per cent jumps in revenue is a whole new theme, which can easily take a month of programming. You need to consider the cost of development, and the lost sales each day the site is unoptimized — so the biggest profit opportunity is not always the best.

I have two default recommendations. The first is to make the simplest change that is likely to grow profit the most (low hanging fruit). My second recommendation is to work closest to the payment as possible, then progress backwards along elements in which every customer must engage. Tommy Walker discusses optimizing closest to the money in his article on reducing shopping cart abandonment. The bigger the profit boost and the sooner it happens, the happier everyone will be.

An example of a simple change from Bryan Eisenberg’s hierarchy of optimization is functionality.

Does the store satisfy a visitor’s needs? Every interested buyer wants to know how much their order will cost. If they cannot find clear costs for delivery, expect this change to increase profit. A lot of stores like BlackMilk have good success in mentioning their delivery information in the header.

Here are a some of my favorite questions, taken from my book Shopify Conversion Rate Optimization, that you can ask in surveys to solve problems at various levels of the hierarchy of optimization:

What almost stopped you from buying from us? The best question a merchant can ask on their thank you page.
What other options were you considering before buying from us? Another good question post-purchase to understand the psychology of options and competition.
It seems you didn't complete your purchase, can you share why not? Great for when people leave the cart page or stay on it for more than 20 seconds.
What other information would you find helpful on this page? Trigger after 20 seconds to show only to an engaged visitor. Good for product pages to fill in information gaps.

HotJar also has a great action plan and set of tools to help you identify what to test.

How to configure a test for Shopify

You’ve learned about cookies and revenue tracking, now it’s important to select a testing tool that works in Shopify. The tool also has to be affordable and simple to implement.

I also recommend picking an option that is fast, because we know from study after study that speed is correlated with revenue in ecommerce. ConversionXL recently published a great study on the speed of split-testing tools:

When all these criteria are considered, your best options are Convert Experiments and Visual Website Optimizer (VWO). Other tools, like Optimizely, are viable for Shopify Plus stores.

My preferred tool of choice is Convert Experiments. Each of the three tools I recommend have clear tutorials to set-up in Shopify:

Once you have added the required scripts, configured the test settings, and set-up your goals, there’s another critical integration you must set-up in order to optimize for revenue.

Google Analytics integration

One last configuration you need before creating the actual split-test, is to integrate your testing tool with Google Analytics. Always integrate with Google Analytics for segmentation.

Go to Reports > Tech > Tech details > Platform/ device category to review the store’s performance by platform or device category and identify outliers.

The first requirement of any analytics integration is to get a clean analytics set-up. There’s a lot more you need to do than setup enhanced ecommerce tracking, like keep out spam hits, track your big customers across multiple devices, see the full funnel process, set-up content groups, and use cross-domain tracking. Follow my Google Analytics Setup for Shopify to get clean data in all these scenarios.

Most testing tools provide a good guide to integrate with Google Analytics 4.

Here are the main testing tools, and their steps to integrating with GA4:

You might also like: A Short Tutorial for Running A/B Tests in Shopify with Optimizely

Video example of creating a split-test

Here’s a short video tutorial I created that demonstrates an actual test being created for a Shopify store. This shows a simple split-test, using the visual editor of Convert Experiments:

Bonus testing tips for Shopify

How can I test price?

You can’t test price in Shopify with a split-testing tool because the tool alters data at the DOM level. Previously in Shopify, to test a product's price you had to make changes to the product itself over a set period, then contrast it to the old price. It's imperfect given the variable nature of traffic sources and seasonal trends.

Now you can use the Quant Price app. Products with high turnover are the most suited to be tested. If you haven’t discussed optimizing the pricing of products with clients on Shopify Plus, do it now. It’s devastatingly effective — the simplest thing an established store can do to jack up profit.

I’m having trouble designing my test. Any advice?

It’s possibly a wiser step is to make the change on the site so it becomes the “control.” You can then create a test in the visual editor of the testing tool, to bring it back to the original version. The amount of jQuery required can otherwise get convoluted for complex tests.

How can I test alternate templates?

A new template can be created with the syntax:

default_template_name.*.liquid

If you append ?view=list to any collection URL, your store will use the collection.list.liquid template. If you use ?view=grid at the end, the store will call the collection.grid.liquid template. Keir Whitaker has written about this method. The same goes for other pages like the cart.

This is a less than ideal way to test a Liquid template on a page-by-page basis. Unfortunately, site-wide theme tests do not work with this, because the “view” query string is lost with user navigation. The above is a better solution.

And speaking of URLs, learn what a canonical URL is and why they're so important.

How can I test a new design site-wide?

This is a neat hack I figured out to test diverse design changes. You can test new product page designs, and have it stick across all product pages for the duration of the test!

Duplicate the existing theme, then make changes as needed in the duplicate theme. Also add the following CSS:

.shopify-preview-bar { display: none }

The line of code removes the preview text at the bottom insert by Shopify, so users in the variation do not see it.

Preview the new theme from within the Shopify admin, then view the source code in the head to get the theme’s ID. In the below example, we see the ID is 65568381:

<script> //<![CDATA[ var Shopify = Shopify || {}; Shopify.shop = "mygoods.myshopify.com"; Shopify.theme = {"name":"[LIVE] My Goods","id":65568381,"theme_store_id":null,"role":"main"}; //]]> </script>

Then, what we need to do is append ?preview_theme_id=65568381 as a variation for a URL test so that the new design loads for 50 per cent of people who visit the store. Here is exactly how to do it inside Convert Experiments:

Create an A/B URL test. Make the control URL:

http://www.store.com([^\?]*)??(.*)$

Use the following as the variation 1 URL, where the preview_theme_id variable is what you discovered from the source code. This code appends the preview_theme_id variable to load a new theme:

http://www.store.com$1?preview_theme_id=65568381&$2

Then enable “Support Regular Expressions”. Your test variations will look like:

The last thing you need to do is include all pages in the test with the site area settings:

http://www.store.com/{0,1}(([\?&]{1}[^=]{1,}=[^=]*[&]{0,1})*)$

And exclude pages when the query string contains:

preview_theme_id=65568381

Be sure there is the necessary Convert Experiments testing code in the Liquid files of the duplicated theme. You can now test any design change you want in Shopify.