5 Guidelines for A/B Testing Your Email Campaigns

Out of all the different platforms available, email marketing has been shown to consistently generate among the highest returns on investment. Some businesses get as much as $44 for every $1 spent on their campaigns, and gain 40x as many customers as Facebook and Twitter combined.

This incredible return on investment is due in large part to how relatively quick and inexpensive it is to create and send out mass email campaigns. However, with so many other businesses engaging in email marketing, it can be hard to get and keep your audience’s attention. Without some kind of analysis, it can be almost impossible to tell why one email was successful, and another one wasn’t.

Enter: A/B testing. A/B tests utilize a simple but effective experimental model to determine what changes will get better engagement from your prospects.

If you’re interested in experimenting on your own emails, or if you’ve started testing but aren’t sure where to go now, take a look at these guidelines to get your A/B tests up and running:

Develop a strong hypothesis

One of the major factors that sets apart email A/B testing from other A/B test subjects is the wide variety of success metrics you can choose from. By contrast, web A/B tests share the same variables for success—conversions (usually to sales, sometimes converting to email leads) or clicks. Whichever version of your webpage gets the most is considered successful. But how do you measure which email performed best? When conducting an experiment, you’ll have data on open rates, clicks, subscription changes, drop-off rates, attachment downloads, and conversions.

It’s always important to craft an experimental hypothesis before testing, but it’s especially important with emails. The amount of data available means you’ll have to take extra care to define your specific goals before you start running tests. Otherwise, you can quickly end up lost and frustrated before your A/B testing journey has even begun.

A strong hypothesis consists of three essential parts:

1. Which variable you’re modifying
2. The specific outcome you’re expecting (which metric will increase/decrease)
3. The rationale for why you think the change will have the expected impact

Here’s an example hypothesis:

“Our customers are unsubscribing because they’re receiving too many emails from us. If we send out our promotional email series at a rate of 1 email a week, rather than biweekly, the rate of users unsubscribing will decrease.”

Describing the reason for the change is important. Doing so allows you to evaluate the assumptions you have about your audience, slowly replacing them with facts.

Boost your open rate

Email marketing can seem like a bit of a paradox. You’ve been given a direct line into your prospects’ inbox. At the same time, you’re competing with dozens of other businesses who have the same level of access that you do. In order to win your prospects’ attention, you need to get them to open your email based on whatever you can squeeze into your subject line and preheader text. Increasing your open rate is the foundation to email marketing success. It means you’re getting your virtual foot in the proverbial door.

There are the four main variables you can test to make your unopened emails stand out in your prospect’s inbox. Here they are, along with some suggestions for improving each of them:

Subject line

• The most common factors affecting the success of your subject line are the length, the content, and whether you’ve included your company name.
• Ideal length: around 60 characters max for full visibility on mobile, but this can differ somewhat between screens and platforms.

“From” field

• A common test you can run is if your audience responds better to generic professional branding (sales@zylker.com) versus a person’s name (tyler@zylker.com).
• There’s a surprising amount of room for creativity here. You can have your email name reference a specific product (shoes@zylker.com), or have it serve as a friendly greeting (hello@zylker.com).

Preheader text

• Try removing generic salutations that take up space, like “How are you?” Make sure that your preheader text matches the tone of your subject line.
• Ideal length: mobile preheaders are typically displayed at between 30 to 55 characters, and up to 130 on a desktop browser.

Personalization

• Get your readers’ attention and add extra flair to your visible text by customizing it with prospect data.
• You can use personalization elements to include your recipient’s name in the subject line or preheader, or even the number of days since their last purchase.

Establish your testing strategy

Compared to other aspects of online marketing, creating emails is quick, inexpensive, and easy to implement. It’s much easier to be ambitious with your email experiments, compared to your homepage or online store. However, if you’re planning on making significant changes to your emails, like a drastically different design or copy approach, you need a plan. Before you start your first experiment, you need to determine whether your plan is based on strategic testing or optimization testing.

Strategic testing

Strategic testing consists of drastic style or language changes. The idea is that these changes can potentially help you figure out a better long-term communication strategy for connecting with your prospects and significantly increase your engagement.

This plan works best if:

• You’re either just starting out your email marketing operations
• You have strong evidence that you’re missing out on much more engagement than your current strategy is yielding

Otherwise, you take the risk of alienating your readers.

Optimization testing

Optimization testing is the far more common testing strategy. It should be used when you’re confident in your overall technique, but want to optimize reader response by tweaking small variables.

Creating a plan

Whichever approach is the better fit for your situation, you need to commit to it, and plan it out carefully. If you choose strategic testing, you should make a detailed document that explains the following:

• The qualitative differences between your new and current strategy
• Why you think the new strategy will perform better than the current one
• The areas you’re making the changes in
• What metrics you think will improve through this test

For example, you could discover that your audience responds much more strongly to a casual writing style than a professional one. However, if you’re not precise in documenting your approach, you may not be able to replicate your success.

When laying out your testing plan, you should always aim to transition to optimization testing as quickly as possible. That way, you can both fine-tune your new technique and avoid giving your audience whiplash from too many drastic changes.

Avoiding drop-off

In email marketing, you have the advantage of a captive audience, in the form of your subscriber list. This can be a source of stability, compared to the fickleness of social media algorithms and organic traffic. Part of that stability comes from emails being a linear, one-dimensional medium. Unlike your website, where prospects can browse any combination of your pages in almost any order—and still never go where you want them to—you can control what they see and the order they see it in. Prospects receive your emails in exactly the order you send them, and they either open the email or don’t.

This aspect of email marketing is a double-edged sword. The increased control allows for more stable experiments, but the linear nature of email means that changes made to an earlier email can unexpectedly impact the effectiveness of later emails in the series. To avoid accidentally setting off an unwanted chain reaction, you should prioritize testing emails that already have a consistently high level of engagement. That way, you can get faster, more conclusive results to identify any drop-off points before it’s in circulation for too long.

This is why it helps to monitor your campaigns’ performance long before you start experimenting. It helps you to establish a baseline level of engagement that can serve as a benchmark. Even if you’re only making a small change to one email in a series, make sure you’re paying attention to everything downstream. This advice is applicable across the board, but it’s especially important if you’re testing the first email in a series or sending out an educational or content marketing series.

Audience segmentation

One of the benefits of email is the ability to control the demographics of your test subjects. Even with basic data collected from your subscription form or a previous purchase, you can segment your email list into broad demographic groups. From there, you can start with basic forms of personalization, like localization.

When localization is executed correctly, your users shouldn’t be able to notice that you’ve done anything at all. For instance, you can use their city/state information to divide your audience up into geographic territories. That way, instead of sending your whole list an email featuring the #1 product in the entire US, you can send them personalized emails that show the #1 product in their region.

If you want to determine the best time to send out your emails, don’t send them all out simultaneously. For more consistent results, you can send them out in batches. That way, they arrive at the same hour of the day in each of your prospects’ respective time zones.

Make sure your signup forms have 1-2 questions, as well. You can ask about their business or reason for interest in your brand during email signup or checkout. That will also help you do more finely targeted testing.

Email marketing has massive potential to help your business grow, but it can be an uphill struggle if you don’t create a strategy. However, with a solid grasp of the medium and powerful techniques like A/B testing, you can hone your skills and stand out among your competitors.