A/B testing cold emails (without a statistically-significant sample size)

In the early days of your startup, you shouldn't send thousands and thousands of emails. The worst way to begin your marketing push is by spamming people—that’s not how you want to build awareness for your business.

Instead, focus on sending small, high-quality batches of 50 to 100 emails daily to heavily targeted audiences.

But if you’re only sending out 50 emails or so a day, it’s hard to figure out what you’re doing right and what you’re doing wrong.

With such a small sample size, it’s nearly impossible to conduct statistical analysis on your cold emails and improve their overall quality. And even if you send out 10,000 emails a day, it takes a huge amount of time before you can pull meaningful, statistically significant insights from A/B testing.

Get around this and learn faster with this simple hack. You won't be able to just stare at numbers on a screen—you’ll need to pick up the phone and talk to people!

Ramp up A/B testing by picking up the phone

Quantitative data is integral to good business, but you’ll never have enough just starting out. You can bypass this lack through an intuitive, but rigorous, approach that plays to your strengths as a salesperson.

Take a sample of the emails you send, and follow up with a phone call. Your objective in the call isn’t to make a sale, though it could lead to that.

Instead, you want to find out what worked in your emails and what didn’t. You’re gathering intel.

When you start with a sample of 50 emails a day, it’s an opportunity to drill into the qualitative side and find out what connects to people on an emotional level. By making the call, you can gain valuable insights to your sales strategy and business that are unavailable to even the largest of data sets.

3 steps to accelerate your cold email outreach

1. Write high-quality emails to targeted audiences

Don’t put yourself in the spam folder. Prioritize quality in both who you email, and how you email.

Start by building a targeted list of leads that includes the key decision-makers and players that occupy your market.

There are many cold email templates you can use—but it’s important to customize each email and tailor it to the specific person you’re reaching out to. Write like a human. The person you’re reaching out to gets hundreds of these emails, and it’s vital to rise above the noise.

Here’s one example:

Perfect your message and only include necessary information. You only have around 40 words to get their attention—so don’t waste them. And keep in mind that the best cold outreach templates don't actually sound like templates; they're real conversation starters with real people.

Experiment with different subject lines and email copy. Iterate. Email consistently, every single day. Once you’ve streamlined your cold email process, you can start discovering what works for your business.

2. Segment your email outreach into three groups

Send emails using a CRM software (like Close) that allows you to track open and response rates. You’re on the right track if you have around a 35% open rate and a 10% response rate. Any response rate below 5% means you’re doing something wrong.

You can segment your cold outreach into three groups:

• People who opened the email and responded
• People who opened the email and didn’t respond
• People who didn’t open your email

Each of these actions tells us something different about how each segment reacted to your emails—whether they were thrown off your subject line and didn’t bother to open the email, or weren’t sold by your pitch. These signals allow you to experiment and test for better results.

It’s never going to be an exact science—but luckily, it’s not about making perfectly correct assumptions. It’s about how you make it work.

3. Follow up with a cold call

By segmenting your email outreach, you create testing groups that you can call and probe further for insights.

For the first group, call them and ask, “I’m sure you get hundreds of cold emails each day. I’m curious, why did my cold email reach you? What peaked your interest? Why did you decide to respond to it—what did you like about it?”

For the second group, learn why they found the subject line compelling enough to open the email, and for the third group, discover why they didn’t even bother to open the email.

Ask them, “I’d like to be respectful of your time. I sent you a cold email this morning and you never replied. You probably don't open or reply to most cold emails—I don't either. This call’s purpose is not to sell you. From one professional to another, can I just ask why didn't you like it? I know we have a really valuable product—why wasn’t I successful in conveying that? I’d highly appreciate even the smallest bit of feedback.”

If you come to the call with vulnerability and authenticity, you’re guaranteed to provoke some magical moments—human insight that’s unmatched in statistical analysis. Of course, some people will still hang up the phone or shoo you away.

But others will give you valuable responses, for example:

• “I didn’t like your email because it was obviously automated.”
• “I don’t even remember your email—nothing stuck out to me.”
• “I honestly thought your email was a scam.”

Resist the urge to defend your product or email. Don’t try to sell. Seize the opportunity to learn more about what you’re doing right, and what you’re doing wrong.

Once you start identifying patterns within these responses, you can use them to revise your pitch, subject line, or email body and improve the quality of your overall cold email strategy. Plug them back in to the first step, and you’ll start seeing results immediately.

Forget significant statistical analysis in the short-term

In the early stages of your startup, you will lack the data to make perfect decisions. Overcome this hump by gathering qualitative data and making intuitive decisions based on talking to customers.

Realize that the startup business is messy by nature. Even if you send 10,000 emails daily, it takes a long time before you can use A/B testing to find a winning variation.

By implementing cold calls into your cold email strategy, you can quickly gather data and improve the quality of your emails, short-circuiting the need for large data sets. The information and context you’ll receive from just 10 of these conversations is unbeatable, giving you real, human insight on how to connect to people.

Want to learn more about how to write killer cold emails? Click below!