Creating A/B test email campaigns

Overview

When it comes to contact engagement, even small variations to your newsletters can have a big impact. A/B testing helps you learn what works best for your contacts (and what doesn't).

In this article we will help you understand:

What is an A/B test?

An A/B test (or split test) is a way to test how two different variations of an email campaign (A or B) perform with a portion of your recipient list. This type of testing enables you to present the "winning" or most successful variation of your campaign to the rest of your audience to help improve your results. 

When creating a test, you will:

  • choose one element to test, such as the subject line or email content
  • select the sample size of recipients for the test (each variation will be sent to exactly half of the sample)
  • choose the method for determining the winner: best open rate or best click rate
  • set the duration for when the test should run

Each recipient in your sample will receive only one of the two variations: version A or version B. After the test period, the winning variation will be selected based on your criteria (open or click rate) and will be immediately sent to the rest of your campaign's recipient list. 

This process ensures the best version of your campaign is sent to the largest group of recipients.

What email elements can you A/B test on SendinBlue?

Email subject lines are the very first thing a recipient looks at since it's seen before opening your email. Research shows that 68% of people open an email based on the subject line alone.

For this reason, the subject line is the first element available for A/B testing in SendinBlue email campaigns.

For example, you can try running an A/B test on campaign email subject lines to determine things like whether it's more effective to: 

  • use short or longer subject lines
  • use punctuation
  • include your contact's first name, a creative teaser, or mention an incentive, etc.

Good to know: A/B testing will be available for other elements, like email content and "from" name in the near future.

How to create an A/B test email campaign

A/B test email campaigns are created the same way as a regular email campaign but include additional steps to set the test criteria.

Before getting started

Please consider these recommendations when creating an A/B test email campaign: 

  1. It should include at least 20,000 recipients (for statistically relevant test results). 
  2. It may not be scheduled with our Send Time Optimization tool Send at the best time
  3. Your very first email with SendinBlue should be a regular campaign, rather than an A/B test.

Campaign setup

To create an A/B test email campaign, navigate to the Email Campaigns page and click on the Create an email campaign button.

Camp._EN_1.png

On the Setup page, click on the A/B Test tab to start creating an A/B test email campaign. 

ABT-2.png

Give your campaign a name for easy internal reference. In the Subject Lines section, enter the two versions of the subject line you want to test in the A and B fields 

As a best practice, please ensure that both versions of your subject lines clearly describe the content inside your email campaign. 

ABT-3.png

Good to know: It is recommended to test only one variable per A/B test. For example, try testing whether your results improve by changing the subject line length, or by phrasing it as a question.

Campaign Design

The next step is to create a new design using your favorite SendinBlue tool, or by customizing an existing template. 

ABT-4.png

Campaign Recipients

Once you've designed your email, select the recipients to whom the campaign will be sent. 

ABT-5.png

Campaign Split Rules

The next step is to define the A/B test conditions or the Split rules. These rules will determine the size of the test groups, how the winning version will be selected, and the amount of time for which the test will run. 

The default test group size is 50% of the total recipient list with versions A and B each being sent to 25% of the recipient list. The remaining 50% of the recipients would receive the winning version after the test ends. To change the test group size, simply move the slider right to increase the size or left to decrease it. 

Next, select the method for determining the winning version. You can choose whether the version with the highest Open Rate or the highest Click Rate wins.

Finally, select how long the test will run. After the test duration completes, the winning version will be identified and sent to the remaining recipients. The duration may be defined in hours or in days with a maximum test duration of 7 days. 

ABT-6.png

Good to know: 
1. We recommend including enough contacts in your campaign so that each test group contains at least 5,000 recipients (20,000 contacts per campaign). This volume ensures that your test results will be statistically relevant. 

2. Include only contacts who have already received at least one email from you via SendinBlue.

3. Test group recipients will be randomly selected from total recipients based on the ratio you select. Each recipient in the test groups will receive only one version of your campaign.

4. The maximum size of each test group is 50%. In this scenario, all recipients participate in your A/B test and thus there will be no one to receive the "winning" version.

Campaign Split Rules

Now you are ready to schedule your A/B test email campaign to send to the test groups. You can either choose to Send it now or Schedule it for a specific time

Send it at the best time may not be used for an A/B test email campaign.

ABT-7.png

The test versions will be sent to the respective test groups immediately if you select Send it now, or at the specific date and time that you select. The winning version will be sent automatically to the rest of the recipients as soon as the test duration ends.

For example, if the test duration is 4 hours and the campaign is scheduled to send at 12:00 pm on a certain date, the winning version will be sent out to the remaining recipients at 4:00 pm on the same day.

A/B test email campaign features

To help you identify an A/B test campaign from a regular campaign on the campaign listing, an A/B test tag will be added to the campaign name.

ABT-8.png

The campaign type may be changed to/from a regular draft into an A/B test draft (and vice versa). However, once a campaign is Scheduled or Sent its type cannot be changed.

After an A/B test email campaign is scheduled, you may still Edit or Suspend it until the time it is sent to the test recipients. Once the campaign status changes from Scheduled to either Running or Sent it cannot be suspended or edited. 

Campaign Reports

In the campaign Report, you can view the A/B test overview and engagement metrics for each version of the campaign by clicking on the A/B Split Overview tab.

 ABT-9.png