Skip to main content

Creating an A/B Test campaign

When it comes to contact engagement, even small variations to your newsletters can have a big impact. A/B testing campaigns help you learn what works best for your contacts (and what doesn't).

Sendinblue allows you to run A/B Test campaigns on your email subject lines or content.

❓What is an A/B Test campaign?

An A/B Test campaign (or split test campaign) allows you to test two different variations of an email campaign (A or B) with a portion of your recipient list. The "winning" or most successful variation of your campaign will be sent to the rest of your audience to help improve your statistics.

When creating an A/B Test campaign, you will:

  • Choose one element to test, such as your email subject line or content.
  • Select the sample size of recipients for the test (each variation will be sent to exactly half of the sample).
  • Choose the method for determining the winner: best open rate or best click rate.
  • Set the duration for when the test should run.

Each recipient in your sample will receive only one of the two variations: version A or version B. After the test period, the winning variation will be selected based on your criteria (open or click rate) and will be immediately sent to the rest of your campaign's recipient list. 

This process ensures the best version of your campaign is sent to the largest group of recipients.

🧪 What elements can you test in A/B Test campaigns?

Subject Lines

Email subject lines are the very first thing your recipients will see when receiving your campaign since subject lines are visible to them before they even open your campaign. 
Research shows that 68% of people open an email based on the subject line alone.
campaigns_subject-line-view_EN-US.png

It is important to use an effective subject line for your campaigns depending on your audience. You can run A/B Test campaigns on email subject lines to determine for example if it is more effective to: 

  • Use short or long subject lines.
  • Use punctuation and/or emojis. 
  • Include personalization (like your contact's first name), a creative teaser, or mention an incentive, etc.

Email Content

You can test different elements in the campaign design itself that could influence or not the conversion of your clients like different section titles, article length, call-to-action buttons, images, and more. You might even test two completely different designs to see which one gets the most clicks.

Version A Version B
Campaigns_test-version-A_EN-US.png Campaigns_test-version-B_EN-US.png

📝 Create your A/B Test campaign

A/B Test campaigns are created in the same way as regular email campaigns but include additional steps to set the test criteria.

Before you start

  • For statistically relevant test results, you should send your A/B Test campaign to at least 5,000 recipients.
  • It is impossible to send A/B Test campaigns with our Send time optimization tool (Send at the best time
  • If you are creating a campaign for the first time with Sendinblue, we recommend first sending a regular email campaign to get used to our tool. 
  • Make sure you imported your contacts into Sendinblue and have your lists of contacts or segments ready to be used. 
  • If you wish to use your own email template, make sure it is ready to be used or learn how to create an email template before creating your campaign. 

⚙️ Configure your A/B Test campaign

To create an A/B Test campaign:

  1. Go to Campaigns > Email.
  2. Click Create an email campaign.
    create_email_campaign.png
  3. Click on the A/B Test tab and fill in the relevant information:
    1. Choose which element you would like to test between Subject lines and Email content.
    2. Write a Campaign name that will help you find it easily in the list of your campaigns.
      This name will only be visible to you.
  4. Click Create A/B campaign
    2022-09-30_15-34-19__1_.gif
  5. Choose a sender and recipients from the From and To sections respectively and set Advanced settings if needed.
    To learn more, check our dedicated article on how to Configure your campaign.

A/B Test campaign on Subject Lines

If you decided to test two different subject lines in your campaign, you will see in the Subject section that there are two Subject Line fields: A and B.

Click Add subjects and enter the two different subject lines you want to test in the A and B fields respectively:

campaigns_ab-test-subject_EN-US.gif

💡 Good to know
You can insert personalization into your subject lines and preview text using the 👤  icon. To learn more, check our dedicated section on Subject in Creating an email campaign.

A/B Test campaign on Email Content

If you decided to test two different email content, you will see in the Design section that you can create two email designs:

campaigns_ab-test-email-content_EN-US.png

Click on Design email A to create your first email design. Design your email campaign as if you were designing a regular email campaign. You can choose a template or start it from scratch. While designing campaign A, you can preview it on different mobile devices, as customers and test it. To learn more, check our dedicated section on how to Design your email campaign.

Once you are done with designing email campaign A, you can click Design email B. From Select a template, you can choose a template or start from scratch but we recommend copying version A of your email campaign to change the elements you want to test:

campaigns_ab-test-copy-A_EN-US.gif

💡 Good to know
You can switch from Version A to Version B directly from the header of the Drag & Drop Editor.

✂️ Define Split Rules

When creating an A/B Test campaign the Split Rules section is added to your campaign creation flow. These rules will determine the size of the test groups, how the winning version will be selected, and the amount of time for which the test will run. Click Add split rules to start configuring them. 

Select the size of your test groups

❗️ Important
You must first select your recipients to define test groups. 

The default test group size is 50% of the total recipient list with versions A and B each being sent to 25% of the recipient list. The remaining 50% of the recipients would receive the winning version after the test ends. The maximum size of each test group is 50% and each recipient in the test groups will receive only one version of your campaign. To change the test group size, simply move the slider right to increase the size or left to decrease it. 

campaigns_ab-test-slider_EN-US.gif

How will the winning version be chosen?

Select the method for determining the winning version: choose whether the version with the highest Open Rate or the highest Click Rate wins. campaigns_ab-test-open-click-rate_EN-US.png

How long should we run the test? 

Select how long the test will run. After the test duration completes, the winning version will be identified and sent to the remaining recipients. The duration may be defined in hours or in days with a maximum test duration of 23 hours or 7 days. 

campaigns_ab-test-duration_EN-US.gif

🚀 Finalize your campaign

After completing each section of the email campaign creation flow, you will find in the top-right corner of your screen three options: Save & QuitPreview & Test, and Schedule.

❗️ Important
The Best time option cannot be used to schedule the sending of A/B Test campaigns. You can only choose between the Send now or Schedule options to send your A/B Test campaign. 

The test versions will be sent to the respective test groups depending on the schedule option you choose and the winning version will be sent automatically to the rest of the recipients as soon as the test duration ends.

➡️  For example
If the test duration is 4 hours and the campaign is scheduled to be sent at 1:00 pm on a certain date, the winning version will be sent out to the remaining recipients at 5:00 pm on the same day.

You will be able to edit and see your A/B Test campaign in your campaign list. To help you identify an A/B Test campaign from a regular campaign in your campaign list, an A/B Test tag is added to your campaign name.

campaigns_tag-ab-test_EN-US.png

📊 Access and understand your A/B Test campaign reports

Once your campaign is sent to all your recipients, you can access the report and see which version of your A/B Test campaign won. To access your report, click on the name of your A/B Test campaign from your campaign list once it has been sent to all your recipients. You will have access to all the statistics on your A/B Test campaign. Click the A/B Test results tab to see which version of your A/B Test  campaign won. 

campaigns_ab-test-report_EN-US.png

🤔 Have a question?

If you have a question, feel free to contact our support team by creating a ticket from your account. If you don't have an account yet, you can contact us here.