- Klaviyo A/B test sample size has two answers: the minimum Klaviyo needs to show a label, and the number you need to detect a lift you care about. The second is much bigger.
- Klaviyo labels a campaign test significant with more than 50 deliveries per variation and over 90% win probability. Flows need 500+ per variation; sign-up forms 1,000+ views.
- Klaviyo’s own blog example: a 2% conversion rate and a 30% lift needs about 9,800 profiles per arm at 95% confidence and 80% power.
- Small list? Test bigger changes, use click rate over opens, and run tests in flows, where recipients pile up over time.
- When the test ends, check it in the A/B test significance calculator.
Here's the uncomfortable truth about Klaviyo A/B test sample size. Klaviyo can label a campaign test “statistically significant” once each variation has just over 50 deliveries. But to reliably detect a lift you'd actually act on, you usually need thousands of recipients per variation.
Both facts come from Klaviyo. Its campaign significance guide sets the 50-delivery minimum. Its own blog on A/B testing works an example where a 2% conversion rate and a 30% lift needs roughly 9,800 profiles per test arm.
So the label tells you which variation is likely ahead. It doesn't tell you whether your test was big enough to find the difference you were looking for. Here are the tricks first, then Klaviyo's thresholds, the formula and a cheat sheet you can screenshot.
A/B Test Significance CalculatorTest finished? Enter recipients and conversions for A and B. See the uplift, p-value and confidence in seconds.Run my numbers →6 tricks for Klaviyo A/B tests you can actually trust
1. Size the test before you send it
Klaviyo's blog lists four inputs: your baseline rate, the smallest lift worth detecting, statistical power and the significance level. Standard practice is 5% significance and 80% power. Plug them into the formula below. If you can't reach the number, change the test, not the maths.
2. Test clicks or orders, not opens
Klaviyo's A/B testing guide warns that Apple Mail Privacy Protection inflates opens, and that open-rate tests count those automatic opens. If more than about 45% of your opens come from Apple Mail, Klaviyo suggests a custom report with an MPP property. My rule is simpler: pick click rate for content, placed order rate when you have the volume.
3. Test big swings, not button colours
The smaller the lift, the bigger the sample. Klaviyo's blog says the smallest lift worth detecting in most real CRM programs is around 20–30%. A new offer, a different hero product or a founder note vs a designed template can move numbers like that. A blue vs green button usually can't.
4. Move small-list tests into flows
A campaign gets one shot at your list. A flow keeps adding recipients every day. Klaviyo's flow significance rules need more than 500 deliveries per variation, and a busy cart or welcome flow gets there on its own. Set a date limit as a backstop.
5. Use a 100% test size when the list is small
Klaviyo's example: a 40% test size sends 20% of recipients version A and 20% version B, then the winner goes to the rest. On a 5,000-person list that's only 1,000 per arm. If you'd rather learn than win today, send to 100% and split it 50/50.
6. “Promising” means test again
Between 75% and 90% win probability, Klaviyo calls the result promising and recommends repeating it. If a test keeps coming back not significant, Klaviyo says move on to a different topic. And don't test near a holiday weekend, when your audience behaves differently.
Klaviyo's own sample size thresholds
These are the minimums Klaviyo uses before it will label a result. They're floors, not targets.
| Test type | Minimum per variation | Other conditions for “significant” |
|---|---|---|
| Email campaign | More than 50 successful deliveries | Win probability over 90% |
| SMS campaign | More than 50 recipients | Win probability over 90% |
| Flow message | More than 500 deliveries | Win probability over 90%, at least 10 completed actions in total, less than 10% overlap in likely performance ranges |
| Sign-up form | More than 1,000 views | Win probability over 90%, at least 10 submissions in total, little overlap in likely submit rates |
For campaigns, Klaviyo uses four labels. Statistically significant as above. Promising at 75–90% win probability. Not statistically significant when win probability is 75% or lower and every variation has more than 1,800 deliveries, or when it's below 60% and the gap between the top two is under 4 percentage points. Everything else, including any test with 50 or fewer deliveries in a variation, is inconclusive.
The Klaviyo A/B test sample size formula
This is the two-proportion formula from Klaviyo's blog. It tells you how many people each variation needs:
Don't want to do it by hand? Use the cheat sheet. I ran the formula for common email baselines:
| Baseline rate | Detect +10% | Detect +20% | Detect +30% | Detect +50% |
|---|---|---|---|---|
| 1% | 162,910 | 42,645 | 19,804 | 7,741 |
| 2% | 80,590 | 21,085 | 9,786 | 3,821 |
| 3% | 53,150 | 13,898 | 6,447 | 2,514 |
| 5% | 31,198 | 8,149 | 3,776 | 1,469 |
| 10% | 14,734 | 3,837 | 1,772 | 685 |
Recipients needed per variation, at 95% confidence and 80% power. Lifts are relative, so +20% on a 2% rate means 2.4%. The 2% / +30% cell matches Klaviyo's “roughly 9,800” example.
Read the table once and you'll never trust a 300-person test again. Halving the lift you want to detect roughly quadruples the sample you need.
Worked examples
A Shopify store has 20,000 engaged profiles. Its campaign click rate is about 2%. It wants to test a new email layout.
- To detect a +20% lift (2% to 2.4%), it needs 21,085 per variation, or 42,170 in total. That's more than the whole list. Not possible in one send.
- To detect a +50% lift (2% to 3%), it needs 3,821 per variation. With a 40% test size, each arm gets 20,000 × 20% = 4,000. That works.
- So the store tests a bold change, like a single-product email vs a six-product grid, not a new button colour.
When it's done, the store checks the result in the A/B test calculator before rolling it out.
The same store's abandoned cart flow sends email 1 to about 1,500 people a month, with a click rate around 6%.
- To detect a +30% lift in clicks, it needs about 3,108 per variation, or 6,216 in total. That's about 4 months of traffic.
- To detect a +50% lift, it needs about 1,207 per variation, or 2,414 in total. That's about 7 weeks.
- Both clear Klaviyo's 500-per-variation flow minimum long before they clear the formula. If the flow test ends at 600 per side, the label might say significant, but the test was too small to reliably catch a 30% lift.
What “statistically significant” in Klaviyo does and doesn't mean
Klaviyo is clear that a significant result means one variation is highly likely to outperform the other for your chosen metric, based on the test data. It also says the result doesn't guarantee the same outcome with a different audience, message or send time.
Two more traps:
- Date limits aren't significance. In flows and forms, if the date limit arrives first, Klaviyo picks the variation with the highest observed rate. That winner isn't necessarily significant.
- Significant isn't the same as worth it. Klaviyo's blog puts it well: a lift can be real and still not worth operationalising. Check the size of the win, not just the label.
What to test first
Klaviyo's testing guide suggests starting points by goal. My order for a Shopify store:
- Orders: social proof (reviews or customer posts), CTA placement, send day or time.
- Clicks: template layout, CTA text, images vs GIFs. Track it with the email CTR calculator, and see the click to open rate benchmark for ecommerce to know what a normal result looks like.
- Opens: subject line length, questions vs statements, personalisation, from name. Score ideas in the subject line tester first, then test the best two.
Test one variable at a time, and Klaviyo recommends only 2 variations. Every extra variation splits your sample further.
Flows are the best place to start because they run all year. My Klaviyo post-purchase flow examples give you seven flows with plenty to test.
Watch the guardrails
Klaviyo's blog names unsubscribes, spam complaints and bounces as guardrail metrics. A version that wins on clicks but doubles unsubscribes is not a winner. Check it with the unsubscribe rate calculator, and if bounces move, my guide to hard bounce vs soft bounce in Klaviyo shows what to fix.
Pre-send checklist
- Did I write down the baseline rate and the smallest lift I care about?
- Does my audience reach the sample size in the table?
- Am I testing clicks or orders, not opens?
- Am I changing only one thing, with only 2 variations?
- Is this a normal week, not a holiday weekend?
- Have I decided what I'll do if it comes back promising or inconclusive?
Six yeses, then send. Anything less and you're measuring noise.
Want Inboxoo to set this up for you?
We build and run Klaviyo email & SMS retention for Shopify brands and get paid on results.
Chat on WhatsAppWant to know the exact quotation? Contact now
FAQ
What sample size does Klaviyo need for an A/B test?
For a campaign to get the statistically significant label, Klaviyo needs more than 50 successful deliveries per variation for email (or 50 recipients for SMS) and a win probability above 90%. Flows need more than 500 delivered messages per variation and at least 10 completed actions. Sign-up forms need more than 1,000 views per variation and at least 10 submissions.
Is 50 recipients enough for a Klaviyo A/B test?
It is enough for Klaviyo to classify the result, but rarely enough to detect a realistic lift. Klaviyo’s own blog gives an example where a 2% conversion rate and a 30% lift needs about 9,800 profiles per test arm. Size your test with the formula before you send.
What is win probability in Klaviyo?
Win probability is Klaviyo’s estimate of how likely a variation is to have the highest rate for your chosen winning metric, based on the test data. Above 90% (with enough deliveries) is labelled statistically significant. Between 75% and 90% is promising, and Klaviyo recommends repeating the test.
Should I A/B test on open rate in Klaviyo?
Be careful. Klaviyo notes that Apple Mail Privacy Protection inflates open rates and that open-rate tests count those automatic opens. Click rate or placed order rate is a more reliable winning metric for most tests.
How long should a Klaviyo A/B test run?
For campaigns, Klaviyo recommends a test duration based on the winning metric. In flows, you can end a test when it reaches statistical significance, set a date limit, or both. If the date limit hits first, Klaviyo picks the variation with the highest observed rate, which is not the same as a significant result.
Can I A/B test with a small email list?
Yes, but change your approach. Test big differences rather than small tweaks, use click rate instead of placed order rate, use a 100% test size, and run the test inside a flow so recipients add up over weeks. Treat a single small test as a hint and repeat it.
Want Inboxoo to set this up for you?
We build and run Klaviyo email & SMS retention for Shopify brands and get paid on results.
Chat on WhatsAppWant to know the exact quotation? Contact now
Keep reading
The Klaviyo abandoned cart flow setup
Trigger, filters, timing and discount rules for a cart flow that recovers revenue without training buyers to wait for a code.
7 min read →Klaviyo flows7 Klaviyo post-purchase flow examples
Seven copy-ready post-purchase builds with triggers, splits and timing, from thank-you to replenishment and VIP.
8 min read →DeliverabilityHard bounce vs soft bounce in Klaviyo
What Klaviyo does with each bounce type, the 7-in-a-row rule, healthy rates and a monthly cleanup routine.
8 min read →- Klaviyo Help Center: Understanding statistical significance in Klaviyo campaigns
- Klaviyo Help Center: Understanding statistical significance in flows
- Klaviyo Help Center: Understanding statistical significance for sign-up forms
- Klaviyo Help Center: How to A/B test an email campaign
- Klaviyo blog: Stop testing wrong, how to run A/B tests that actually mean something
Worked examples in this post use illustrative numbers, not client results. Benchmarks are linked to their original publisher.