Cold Emailing

Cold email A/B testing: Get more opens, clicks & replies with split tests

lemlist team
LAST UPDATED
June 9, 2024
READING TIME
7 min.

It's quite difficult to create the perfect cold email campaign on your first try.

There are no silver bullets, but there's A/B testing.

Cold email A/B testing is the ultimate way to quickly identify what works and what doesn't work in your outreach campaigns.

What is A/B testing?

A/B testing is an experiment where you're split-testing two or more variations randomly, to figure out which variation brings better results.

In the world of cold emails, you can A/B test subject lines, two emails, different personalized images, or the entire sequences if you decide so.

cold email ab testing

How do you perform an A/B test with cold emails?

There are three important rules to live by:

  1. Test one thing at a time: e.g. test subject lines only, not both that and image vs. text
  2. Measure: always analyze all three cold email metrics (open, click, and reply rates) to get a clear picture of what variation is working better
  3. Have a big enough audience: to gain relevant insights, make sure your email list contains a minimum of 100-200 prospects.

What should you A/B test in your cold emails?

A/B testing helps you accomplish goals faster in the short-term, as long as you know how to analyze your current cold email performance and identify existing problems.

Let's imagine a situation where you are trying to book meetings using cold emails.

To assess the current state of your outreach campaigns, we need to analyze the following:

  • Are people receiving and seeing your emails? Before analyzing any metric, make sure your email deliverability is clean
  • Are people opening your emails? Analyze your open rate
  • Are you using links or personalized video thumbnail cold email strategies? If so, the next step for prospects is to click on that link, to assess your click rates
  • How many replies did you get? Analyze your reply rate and the type of replies you got

To make this even more actionable, let’s analyze some example cold email stats.

bad cold email results to avoid

What problems do you see?

Two KPIs are standing out here, and not in a good way.

People are not opening these emails. Plus, there are almost no replies.

Considering that the click rate is zero, let's assume there are no links in the cold email, and the sender's sole goal is to get more replies and book meetings.

Goal KPI benchmarks:

  • Open rate → 50%+
  • Click rate → 40%+
  • Reply rate → 8%+

In our example, we know where our problems lie.

Assuming that audience targeting is on point, people are not opening our emails and they're definitely not replying.

We need to fix that. Precisely where A/B testing can help.

A/B testing to increase open rates

Before you start playing with open rates, make sure to check your email deliverability status.

When your open rate is lower than 50%, it might mean that people are not seeing your emails to begin with, so A/B testing subject lines would be a waste of time.

So you need to start with:

  1. Audit the current situation with your email domain
  2. Check your technical email settings (like SPF and DKIM)
  3. Start email domain warm-up

Once checked, you can move to A/B tests.

Finding the subject line

That's the first thing people see after receiving your cold email.

In other words, it makes a direct impact on your open rate.

Recommended read:
Top 56 subject lines to test

Just click on the A/B test button.

Add 2 different subject lines that you feel work better and don't change anything else.

lemlist will automatically split your audience into 2 groups and send emails accordingly.

how to ab test cold email campaign

Ideas to test when it comes to subject lines:

  1. Short vs. long subject lines
  2. Question vs. statement
  3. Add personalization to the subject line vs. the general subject line

For example, after 5 different A/B tests, I found a subject line that gave me an 86% open rate. 😱

I started with a 55% open rate.

Results:

open rate ab testing

Experimenting with intro lines

That's the second thing people see before they open a cold email.

It should be catchy, so people will understand "It's not just another spammy email".

In this example, I tested "Hi {firstName}" against "Heyo {firstName}! Just joking Ted".

intro lines ab testing

Here are my results after A/B testing it with usual "Hi {firstName}:

open rate ab testing results
PRO TIP: You shouldn’t only consider the desktop version of your email, as most people receive their emails in their phones or smart watches.

Send emails at different times

Although it might be tricky to test and sometimes it's a waste of time, there are situations where it's a meaningful factor.

In order to figure out if time is a factor, A/B testing is a great tool to do it.

Try to make 2 different campaigns and send them on different schedules.

  1. Morning (from 8 to 11 am)
  2. After lunch (from 2pm to 5pm)

Also, consider the country the person you are targeting is in, as timeframes may vary.

Boost your cold email click rate

There are different cases when you add links to cold emails.

You can send links to articles, use cases, forms, calendar availability, and so on...

Since most people try to generate leads with cold emails, I'll show you examples with links to "book a call" tools, such as lemcal.

Test different anchors

People always check the link before they click on it.

If it looks suspicious, they won't click on it in case it’s spam.

So one of the ideas is to A/B test the way your link looks in the cold email. For example:

  • full link:
  • link with anchor:
book more sales meetings

After 5 tests of different anchors, the results weren't different significantly.

But in all tests, links with anchor won on +3-5% to the click rate.

Dynamic landing pages vs. regular cold email

You really need to A/B test this tactic, it gave me 2x more leads generated from cold emails.

The trick? Dynamic landing pages.

For example, there are 2 ways to send a lemcal link to your leads:

  • Include it in the email:
book more sales meetings
  • Add a dynamic landing page with a custom video + embedded lemcal on it:
generate leads with cold emails

So, my email looked like this:

Best cold email template example

When people click on the video, they go to a dedicated landing page, where they will see a few more personalized touchpoints, along with the lemcal link to book a demo with me.

And here are the results I got:

results

The difference was huge! 🔥

Recommended read:
How to land more sales meetings

If you want to try dynamic landing pages, use lemlist for free for 14 days and get unrestricted access to all the cool features. 🔥

A/B testing to get more replies

The reply rate is the key metric for the majority of people who send cold emails.

The response is the beginning of a conversation that can potentially lead to a win-win situation.

"Tiramisu" strategy vs. generic intro

This is a strategy that we talk a lot about in lemlist family, the largest closed community for salespeople.

The Tiramisu strategy consists of writing genuine intro lines and adding custom sentences that change for each prospect.

tiramisu cold email strategy

The results were amazing!

This strategy scored a 92% reply rate.

Personalized image vs. text-only cold emails

With lemlist, you can personalize not only the text but also the visual part.

For example, you can add personalized images to your cold emails.

lemlist will automatically change text, logos, website screenshots, and other elements on the image.

Here is an example of a cold email with personalized images:

sales follow-up cold email template

For instance, you can decide to test a cold email template with text-only content vs. the one with a personalized image.

You can find the most successful cold email templates here.

cold email hub

Liquid syntax A/B test

You can add personalization based on the info you have with liquid syntax.

For example, have "Hi {{fistName}}" if the location is the USA, and "Bonjour

{{firstName}}" if the location is France.

how to personalize cold emails

You can also test different call-to-actions based on location.

how to personalize cold emails

That's a great thing to A/B test in your cold emails.

You can easily personalize small things based on some general info that you have.

Bottom line

A/B testing is the key to making your cold email perform better.

Don't stop when you find a template that works. Always try to improve and leverage split tests in the process.

Try your 14-day free trial at lemlist now and start A/B testing your campaigns!

lemlist team
Your source of actionable outreach tips and strategies that will help you get replies and grow your business.
Get weekly outreach tips
SHARE THIS ARTICLE
Thanks! You've successfully subscribed to lemlist newsletter
Oops! Something went wrong while submitting the form.
G2 Rating
Price
Best for
Standout feature
Con
4.9
star
star
star
star
star
$30/mo
$75/mo
$2,999/mo
Large, distributed sales teams
AI evaluation precision, gamified KPIs
Lack of tracking system
4.6
star
star
star
star
star-half
Not publicly available
Sales operations and finance teams
Powerful configurability
Limited training resources and complex to navigate
4.4
star
star
star
star
star-half
Not publicly available
Mid-market and enterprise businesses
Comprehensive incentive management
Potentially high cost and steep learning curve
4.7
star
star
star
star
star-half
$15/user/mo
$40/user/mo
Enterprise: custom price
Complex sales structures and businesses of all sizes
Complex sales structures and businesses of all sizes
Steep learning curve
4.6
star
star
star
star
star-half
Not publicly available
Collaborative teams
Connected planning
Complexity and steep learning curve
4.6
star
star
star
star
star-half
Not publicly available
Companies with complex sales structures
Complex incentive compensation management (ICM) with high efficiency and accuracy
Complexity for smaller teams and potentially high costs
4.7
star
star
star
star
star-half
Not publicly available
Companies who want to automate commission calculations and payouts
Simplicity and ease of use
Lack of features like redirection
4.7
star
star
star
star
star-half
$30/user/mo
$35/user/mo
Custom: upon request
Businesses that need a comprehensive and user-friendly sales compensation management software
Ease of use and adoption
Lack of ability to configure the product based on user needs
4.8
star
star
star
star
star-half
Not publicly available
Companies with modern sales culture and businesses who want real-time insights
A built-in dispute management and real-time visibility
Users say it works slowly, customer support is slow
4.9
star
star
star
star
star
$30/user/mo
$50/user/mo
Smaller sales teams
Powerful automation
Lesser user base and average user interface
4.7
star
star
star
star
star-half
Not publicly available
Companies with scalable needs
Automated Commission Calculations
Lack of filtering by date, no mobile app
ERP vs. CRM
ERP
CRM
Summary
Backbone of a business's internal operations.
Backbone of customer-centric interactions and operations.
Goal
To centralize and streamline core business processes in a company.
To increase customer experience, satisfaction and loyalty, and boost sales.
Focus
Internal operations and processes across departments (finance, accounting, inventory, supply chain, HR, and sales).
All interactions with leads and customers.
Manages
Internal business data like financial data, inventory levels, production details, supply chain, HR info.
All customer data like contact info, purchase history, communication history, customer preferences and more.
Users
Finance, accounting, operations, supply chain, and HR departments.
Customer-facing teams like sales, marketing, and customer service.
Benefits
Streamlines operations, improves data accuracy, enhances decision-making, boosts collaboration, increases productivity.
Improves customer relationships, increases sales, strengthens customer service, personalizes marketing campaigns, provides insights.
Price
$150 per user per year on average.
$10 to $30 per user per month on average.
PRM Tool
Rating
Feature
Pro
Con
Mobile App
Integrations
Free Plan
Pricing
4.65
star
star
star
star
star-half
Org-wide alignment
User-friendly layout and database
Suboptimal as a personal CRM
square-check
Lack of tracking system
square-check
Team: $20/month
Business: $45/month
4.7
star
star
star
star
star-half
Social Media Integration
Easy contact data collection
No marketing/sales features
square-check
Lack of tracking system
square-xmark
7-day trial
$12/month
4.75
star
star
star
star
star-half
Block Functions
High customization capability
Not a dedicated CRM
square-check
Limited
square-check
Plus: €7.50/month
Business: €14/month
N/A
Open-source
Open-source flexibility
Requires extensive manual input
square-xmark
Limited
square-check
Self-hosted
$9/month or
$90/year
3.1
star
star
star
Simple iOS app
Ideal for non-tech-savvy users
iPhone only
square-check
iOS only
Limited
square-xmark
1-month trial
$1.49/month or
$14.99/month
3.6
star
star
star
star-half
Smart Contact Management
Feature-rich and flexible
Reported bugs
square-check
Rich
square-xmark
7-day trial
Premium: $13.99/month
Teams: $17.99/month
4.4
star
star
star
star
star-half
Customizable Interface
Customizable for teamwork
Pricey for personal use
square-check
Rich
square-xmark
Standard: $24/member
Premium: $39/member
4.7
star
star
star
star
star-half
Integrated Calling
Integrated Calling
Too sales-oriented & pricey
square-check
Rich
square-xmark
14-day trial
Startup: $59/user/month
Professional: $329/user/month
4.8
star
star
star
star
star
Business Card Scanning
Business Card Scanning
Mobile only
square-check
Limited
square-check
$9.99/month
4.45
star
star
star
star
star-half
160+ app integrations
Comprehensive integrations
No free app version
square-check
Rich
square-xmark
14-day trial
$29.90/month or
$24.90/month (billed annually)
Capterra Rating
Free Trial
Free Plan
Starting Price (excluding the free plan)
Maximum Price (for the most expensive plan)
Best for
4.5
star
star
star
star
star-half
square-check
14-day
square-check
€15/month/seat billed annually
€792/month/3 seats billed annually + €45/month for each extra seat
Versatility and free plan
4.2
star
star
star
star
square-check
30-day
square-xmark
But it offers reduced price to authorised nonprofit organisations
€25/user/month
€500/user/month billed annually (includes Einstein AI)
Best overall operational CRM
4.3
star
star
star
star
star-half
square-xmark
square-check
Limited to 3 users
Comprehensive incentive management
€52/user/month billed annually
Small-medium businesses and automation
4.5
star
star
star
star
star-half
square-check
14-day
square-xmark
€14/seat/month billed annually
€99/seat/month billed annually
Sales teams and ease of use
4.1
star
star
star
star
square-xmark
square-check
Limited 10 users
$9.99/user/month billed annually
$64.99/user/month billed annually
Free plan for very small teams up to 10
CRM goal
Increase the sales conversion rate for qualified leads from marketing automation campaigns by 10% in the next 6 months.
SMART Breakdown
1. Specific: It targets a specific area (conversion rate) for a defined segment (qualified leads from marketing automation).
2. Measurable: The desired increase (10%) is a clear metric, and the timeframe (6 months) allows for progress tracking.
3. Achievable: A 10% increase is possible based on historical data and potential improvements.
4. Relevant: Boosting sales from marketing efforts aligns with overall business objectives.
5. Time-bound: The 6-month timeframe creates urgency and a clear target date.
Actions
Step 1: Refine lead qualification criteria to ensure high-quality leads are nurtured through marketing automation.
Step 2: Personalize marketing automation campaigns based on lead demographics, interests, and behavior.
Step 3: Develop targeted landing pages with clear calls to action for qualified leads.
Step 4: Implement lead scoring to prioritize high-potential leads for sales follow-up.
Step 5: Track and analyze campaign performance to identify areas for optimization.
Outcomes
Increased sales and revenue
Improved marketing automation ROI
Marketing and sales alignment
Data-driven marketing optimization
Table
CDP Software
CRM Software
Approach
Data-centric
Customer-centric
Focus
Interactions across various channels and touchpoints, both online and offline.
Sales, marketing, and customer service interactions.
Functionality
Automatically collects, organizes, tags, and makes data available in real-time.
Helps businesses track customer interactions, sales pipelines, prospects, and service requests.
Goals
Personalized customer experiences across all channels.
Better customer relationships, streamlined processes, and improved profitability.
Benefits
Data integration, management, and accessibility, allowing for detailed analysis and segmentation.
Better communication within teams and with customers by organizing information about customer interactions and history.
Data Handling
Handles both identified and anonymous data, stitches together various data points.
Deals primarily with identified customer data.
Use Cases
Personalized marketing campaigns, targeted advertising, content customization across multiple channels.
Managing campaigns and leads, enhancing customer service, providing better customer support, increasing customer satisfaction and loyalty.
Examples
Insider, Bloomreach, Salesforce Marketing Cloud CDP
HubSpot, Salesforce Sales Cloud Lightning Professional, and Zoho CRM
CRM
Free plan
Best feature
Best for
Con
1. HubSpot CRM
square-check
Sales automation
Sales teams
Up to 1,000 contacts
2. Insightly
square-check
Custom fields
Basic needs
Not enough info about the free plan
3. Agile CRM
square-check
Deal and sales pipeline tracking
Small teams
Up to 10 users
4. Zoho CRM
square-check
Lead and contact management
Businesses of all sizes
Limited to 3 users
5. ClickUp
square-check
Unlimited tasks and unlimited members
Personal use
Up to 100MB storage
6. EngageBay
square-check
Live chat
Small and midsize enterprises
Up to 1,000 branded emails per month
7. Bitrix24
square-check
Unlimited users and 5 scrum teams
Big teams
Up to 5GB of cloud storage
8. FreshSales
square-check
Easy to use and simple setup
Beginners
Up to 3 users
9. Mailchimp
square-check
Very beginner friendly
Marketing teams
Send up to 500 branded emails per month
Type of Affiliate Marketing
Unattached
Related
Involved
Format
Paid advertising
Social media or YouTube channels
Dedicated website or blog
Focus
Quick income
Your niche
Your audience
Engagement with your audience
square-xmark
square-check
square-check
square-check
square-check
Very close connection with your audience
Pro
Little effort
Higher credibility thanks to your niche
Long-lasting and scalable
Con
Paid ads cost a lot
Potential for bias since you don’t use the thing you promote
Require time, effort, and dedication

What you should look at next

Cold Emailing

How Much Do Leads Cost In lemlist?

You need your perfect leads, but you want to plan your monthly budget. We get it. Here's a full breakdown to understand how much leads cost when you're using lemlist.
September 9, 2024
Cold Emailing

Instantly vs Apollo: Are they the right fit?

Apollo or Instantly? Which one should you choose? Read this article to help you figure it out!
August 8, 2024
Cold Emailing

9 Best Email Mailing and Marketing Services

An email mailing service is a tool for building customizable templates, sending campaigns, and gathering data. Here is a list of the 9 best ones.
August 5, 2024
Cold Emailing

Using Gmail's SMTP to send emails - What you need to know

Learn how to use Gmail's SMTP server with third-party apps and services, including the new Sign in with Google requirement, and discover how to add non-Gmail SMTP servers to your Gmail interface.
August 1, 2024
August 1, 2024
Cold Emailing

What is an SMTP server? A simple explanation

Discover what an SMTP server is and how it works behind the scenes to enable email delivery across the internet, ensuring your messages reach their destinations.
August 1, 2024
July 31, 2024

What you should look at next

Receive weekly outreach tips in your inbox, sent to 210 000+ salespeople, marketers, founders, and entrepreneurs worldwide!

Subscribe to the lemlist newsletter
You've successfully subscribed to the lemlist newsletter!
Oops! Something went wrong while submitting the form.