How not to split test your emails:
Email platforms made it way too easy to A/B test subject lines… and marketers ran with it.
Now, half the industry thinks “split testing” means changing a couple of words in a subject line and sending.
Open rates are the only metric to measure your subject line 'success'
But they are not reliable, nor are they just a positive metric.
They’re not reliable enough to validate anything.
Proper email testing is scientific 🧪
It’s hypothesis-led, outcome-driven, and focused on meaningful changes, not vanity tweaks to your emails.
If you can’t validate the result with confidence (like by sending the same thing to the same people at the same time), it’s not a test.
It's a calculated guess
Here's what not to split test:
- Subject lines
- Button colours
- Preheaders
- That small tweak to the font or image that's at the bottom of your email
Instead, test the big picture stuff:
- Messaging frameworks
- Type of content vs different audiences
- Email structures
- Timing
- Flow logic
- Tone
Then look at the right metrics to assess impact.
In this edition, I’m testing formats in this very email.
Infact, you could be in one of my control segments (AKA my testing pot).
Scroll, click, explore. And remember: the goal isn’t opens. It’s impact.