When it comes to contact engagement, even small variations to your newsletters can have a big impact. A/B testing helps you learn what works best for your contacts (and what doesn't).
What is an A/B test?
An A/B test (or split test) is a way to test how two different variations of an email campaign (A or B) perform with a portion of your recipient list. This type of testing enables you to present the "winning" or most successful variation of your campaign to the rest of your audience to help improve your results.
When creating a test, you will:
- Choose one element to test, such as the subject line or email content
- Select the sample size of recipients for the test (each variation will be sent to exactly half of the sample)
- Choose the method for determining the winner: best open rate or best click rate
- Set the duration for when the test should run
Each recipient in your sample will receive only one of the two variations: version A or version B. After the test period, the winning variation will be selected based on your criteria (open or click rate) and will be immediately sent to the rest of your campaign's recipient list.
This process ensures the best version of your campaign is sent to the largest group of recipients.
What email elements can you A/B test on Sendinblue?
Email subject lines are the very first thing a recipient looks at since subject lines are seen before opening your email. Research shows that 68% of people open an email based on the subject line alone.
For this reason, the subject line is the first element available for A/B testing in Sendinblue email campaigns.
For example, you can try running an A/B test on campaign email subject lines to determine things like whether it's more effective to:
- Use shorter or longer subject lines
- Use punctuation
- Include your contact's first name, a creative teaser, or mention an incentive, etc.
You can test different elements of the campaign itself, for example: section titles, article length, call-to-action buttons, images and more. You might even test two completely different designs to see which one gets the most clicks.
How to create an A/B test email campaign
A/B test email campaigns are created the same way as a regular email campaign but include additional steps to set the test criteria.
Before you start
- It should include at least 20,000 recipients (for statistically relevant test results).
- It may not be scheduled with our Send Time Optimisation tool Send at the best time.
- Your very first email with Sendinblue should be a regular campaign, rather than an A/B test.
- Make sure you're on the classic version of the campaign creation flow:
✅ Classic Campaign creation flow
❌ New Campaign creation flow
(check this article)
To create an A/B test email campaign, navigate to the Email Campaigns page and click on the Create an email campaign button.
On the Setup page, click on the A/B Test tab to start creating an A/B test email campaign.
You can chose which variable to test :
In the Subject Lines section, enter the two versions of the subject line you want to test in the A and B fields.
As a best practice, please ensure that both versions of your subject lines clearly describe the content inside your email campaign.
Create 2 different versions of your content on design step.
The next step is to create a new design using your favourite Sendinblue tool, or by customising an existing template.
You can create 2 different content versions, if you selected Email content as a test variable.
Once you've designed your email, select the recipients to whom the campaign will be sent.
Campaign Split Rules
The next step is to define the A/B test conditions or the Split rules. These rules will determine the size of the test groups, how the winning version will be selected, and the amount of time for which the test will run.
The default test group size is 50% of the total recipient list with versions A and B each being sent to 25% of the recipient list. The remaining 50% of the recipients would receive the winning version after the test ends. To change the test group size, simply move the slider right to increase the size or left to decrease it.
Next, select the method for determining the winning version. You can choose whether the version with the highest Open Rate or the highest Click Rate wins.
Finally, select how long the test will run. After the test duration completes, the winning version will be identified and sent to the remaining recipients. The duration may be defined in hours or in days with a maximum test duration of 7 days.
Now you are ready to schedule your A/B test email campaign to send to the test groups. You can either choose to Send it now or Schedule it for a specific time.
Send it at the best time may not be used for an A/B test email campaign.
The test versions will be sent to the respective test groups immediately if you select Send it now, or at the specific date and time that you select. The winning version will be sent automatically to the rest of the recipients as soon as the test duration ends.
For example, if the test duration is 4 hours and the campaign is scheduled to send at 12:00 pm on a certain date, the winning version will be sent out to the remaining recipients at 4:00 pm on the same day.
A/B test email campaign features
To help you identify an A/B test campaign from a regular campaign on the campaign listing, an A/B test tag will be added to the campaign name.
The campaign type may be changed to/from a regular draft into an A/B test draft (and vice versa). However, once a campaign is Scheduled or Sent its type cannot be changed.
After an A/B test email campaign is scheduled, you may still Edit or Suspend it until the time it is sent to the test recipients. Once the campaign status changes from Scheduled to either Running or Sent it cannot be suspended or edited.
In the campaign Report, you can view the A/B test overview and engagement metrics for each version of the campaign by clicking on the A/B Split Overview tab.