WTH is A/B Testing?
Episode 9: A Creator's guide to A/B testing. In this episode we tackle that gut-punch feeling when your carefully crafted newsletter disappears into subscribers' inboxes without a trace. Learn how to shift from guesswork to data-driven strategies that work.
So, this one is really all about answering the questions: "What is A/B Testing?" and "How to get started with A/B Testing?".
We'll cover which elements to test so you can understand what truly resonates with readers. Plus, discover why focusing your tests (and avoiding common mistakes) leads to those compound gains that separate growing newsletters from stagnant ones. Featuring real-world examples where tiny changes delivered major engagement boosts.
Timestamps:
- 00:00 The newsletter black hole phenomenon
- 00:55 Why data beats gut feelings every time
- 01:24 Split testing explained (without the jargon)
- 03:15 Real creators, real results
- 05:13 What elements actually move the needle
- 09:42 The "test one thing" rule you can't ignore
- 12:39 How most creators shoot themselves in the foot
- 13:54 Small gains → big results (compound interest for newsletters)
- 15:30 Your first test: simpler than you think
- 16:46 From spreadsheet tears to actionable insights
Some free Reply Two resources available 24/7
- Listen at https://replytwo.com/podcast
- Wiki and Prompt Library: https://replytwo.com/wiki
- Boring Long Form: https://replytwo.com/wiki/a-b-testing
Shout out to Dan Oshinsky and Inbox Collective for their insights: https://inboxcollective.com/a-list-of-things-you-can-a-b-test/