Did That Email Create Sales? A Practical Guide to Campaign Holdouts

A rainy weekend is coming. You send a timely email about rain jackets. Sales rise, and the campaign report looks good.

Some of those customers may have bought a jacket because of the forecast even if you had stayed quiet. To learn what the email added, you need a comparison that experiences the same weather without receiving that campaign.

A randomized campaign holdout can provide that comparison. A group of eligible customers is randomly assigned to receive the campaign; another eligible group does not receive it. You then compare purchasing over the same period.

The useful result is an estimate with uncertainty, not a guaranteed answer from a single send. This guide explains how to plan the test and works through a fictional result that looks positive but remains inconclusive.

Attribution and incremental impact answer different questions

An attributed-revenue report assigns credit according to a platform's rules. Klaviyo's glossary, for example, describes revenue credited when a customer engages with a message and purchases within the attribution window. That is useful for understanding the journey your system recorded. Klaviyo glossary

The incremental question is different: how much purchasing changed because the campaign was added?

That distinction matters for signal-driven email marketing. A seasonal shift, search trend, or event can affect both people who receive your message and people who do not. Comparing this week's campaign revenue with last week's campaign revenue does not separate the message from those changes.

Keep attribution reporting. Add a controlled comparison when the decision warrants it.

1. Write down the decision the test should inform

Start with one sentence:

“For eligible jacket shoppers in the forecast area, does adding this rainwear campaign to our existing program increase purchasing over the next seven days enough to justify repeating this kind of campaign?”

The region, seven-day window, product, and business decision here are illustrative. Your window should reflect how customers buy and what you need to learn. Set it before the campaign, not after the sales curve looks attractive.

Be precise about the comparison:

• Campaign versus no additional campaign estimates the effect of adding that message to the existing program
• Timely campaign versus your usual planned campaign compares two uses of the same send opportunity
• Subject line A versus subject line B compares creative choices among people receiving email

Choose the experiment that matches the decision. If the real question is which campaign should occupy Thursday's slot, adding an extra email to only one group answers a different question.

2. Randomize within one eligible audience

Define eligibility before assigning groups. For the rainwear example, that might include verified campaign permission, reliable region data, relevant category interest, and the normal suppression rules.

Then randomly split that eligible audience using a supported experiment tool or a validated process maintained by your analytics team. Preserve the assignment list and matching customer identifiers under your existing access and retention controls.

Do not put highly engaged subscribers in the send group and everyone else in the holdout. Do not use one rainy city as treatment and a sunny city as control. Those choices mix the campaign effect with existing differences.

Keep a customer's assignment stable for the experiment. Where multiple profiles belong to the same customer, resolve that before randomization where possible. Avoid sending the held-out campaign to the same person through another address.

A campaign-specific holdout also differs from a global holdout. Klaviyo's documented global feature tests the wider marketing program and has eligibility and duration considerations. Do not activate a program-wide exclusion when you meant to test one optional email. Verify that your chosen tool supports the scope you need. Klaviyo global holdout groups

3. Choose the metric and test size together

Pick one primary outcome before sending. Two useful options are:

• Purchaser rate: customers who place at least one qualifying order during the window, divided by all customers assigned to that group
• Net revenue per assigned customer: qualifying revenue, under a consistent refunds and cancellations policy, divided by all customers assigned to that group

Purchaser rate is easier to explain. Net revenue can better reflect order value, but a few unusually large orders may make it noisy. If margin is the real decision, define the cost data you will need rather than treating revenue as profit.

For either metric, specify which orders qualify. Whole-store purchasing can reveal whether a featured product merely replaced another purchase. Product-specific sales can be a useful secondary view.

Next, define the smallest improvement worth acting on. Use the baseline outcome rate or revenue variability, the chosen split, and that meaningful effect to plan sample size and statistical power. A 90/10 split may feel commercially comfortable, but a small holdout can leave you unable to distinguish a useful effect from noise. There is no universally correct holdout percentage. Microsoft Research on power and metric sensitivity

Get an analyst to check the design if the decision is consequential. A large subscriber count alone does not establish adequate power, especially when purchases are rare.

4. Keep the comparison intact

Both groups should follow the same background marketing rules, with only the planned campaign difference. Record other promotions, price changes, fulfillment problems, and relevant site changes.

If the extra campaign causes a customer to browse and receive an existing follow-up flow, the measured effect includes that downstream response under your current program. Describe it that way. Isolating the creative alone would need a different design.

Be cautious with scarce inventory. If the treatment group buys the last jackets, it can prevent holdout customers from buying them. The groups then affect one another, making a simple comparison harder to interpret. Choose a well-stocked campaign for an initial test. Do not withhold essential service messages or promised individual restock alerts.

If a broken checkout, dangerous weather event, or customer-harm issue occurs, stop or change the campaign as needed. Document the disruption and reassess what the test can support. Preserving an experiment never outranks protecting customers.

5. Count everyone assigned, including people who never click

Use the original randomized groups for the primary analysis. If the treatment group contains 10,000 customers, keep all 10,000 in that denominator, including non-openers and customers whose email was not delivered.

This measures the effect of assigning the campaign under real sending conditions. Delivery and engagement rates remain useful operational diagnostics, but choosing only delivered recipients or clickers afterward can break the randomized comparison.

Count qualifying purchases linked to customers in both groups over the same observation window, whether or not they clicked the campaign. Click-based purchase attribution cannot fully measure the holdout group's purchasing because that group received no campaign to click.

Check the data before the result:

• Does the assignment count match the planned split?
• Did either group lose customers during an export or order-data join?
• Are purchases linked to customer identities consistently in both groups?
• Did any holdout customers accidentally receive the tested campaign?
• Are zero-purchase customers still present in the analysis?

Also check what your systems cannot observe. If clicking the email makes a purchase easier to link to a customer, the apparent difference may partly reflect tracking. Do not silently count known tracking failures as no purchase; investigate the missing data and qualify the conclusion.

Missing or incorrectly allocated customers can make an apparent win unreliable. Microsoft Research documents this problem through sample ratio mismatch and related data-quality checks. Diagnosing sample ratio mismatch

6. Read the uncertainty: a worked example

The following numbers are fictional and chosen to illustrate interpretation. They are not a recommended sample size or an expected email result.

Assume customers were independently randomized, purchasing was captured consistently, and the test completed its planned seven-day window. Purchaser rate was the primary metric.

The observed difference is 0.3 percentage points. Applied to the 10,000 treatment customers, that point estimate corresponds to 30 additional purchasers compared with the holdout rate.

It would be premature to report “the campaign created 30 extra customers.” The estimate has sampling uncertainty.

Using the adjusted-Wald method for a difference between two proportions, an approximate 95% confidence interval is −0.18 to +0.78 percentage points. This calculation uses the method described by NIST; it assumes the independent, correctly measured customer outcomes in this example. NIST difference-of-proportions confidence intervals

The interval includes zero and a modest negative effect, as well as a useful positive effect. These data do not establish a positive lift. They also do not establish that the campaign had no effect.

A responsible summary is:

“The campaign group purchased at 3.2%, versus 2.9% in holdout. The estimated difference was +0.3 percentage points, with an approximate 95% interval from −0.18 to +0.78 points. The result is inconclusive; we should not scale this campaign type on this test alone.”

Do not apply a purchaser-rate interval to revenue. Revenue needs an analysis appropriate to its own customer-level distribution.

7. Finish at the planned endpoint

For a fixed-horizon test, do not check the result every morning and stop the first time it crosses your preferred significance threshold. Use the planned endpoint, or an appropriately designed sequential method that accounts for repeated looks.

A short-lived signal creates a practical constraint: the moment may end before you can collect enough evidence. Extending the observation window indefinitely does not turn an underpowered campaign into a clean test of the original buying moment.

If the result is uncertain, preserve that conclusion. A later, preplanned series of comparable campaigns may provide stronger evidence, but repeated exposure to the same customers and differences between moments need to be handled in the design and analysis. Do not pool unrelated sends or report only the ones that won.

For small lists, the best immediate outcome may be learning that a modest effect cannot yet be measured precisely. You can still improve source quality, review speed, stock accuracy, and customer experience. Just keep those operational gains separate from a claim about incremental sales.

8. Keep customer cost on the scorecard

Purchases should not be the only thing you inspect. Track complaints, unsubscribes, fulfillment problems, discount cost, and returns using clearly defined denominators and observation windows. Set any stop thresholds before launch where possible.

Open rate is particularly unsuitable as proof of purchase impact. Apple Mail Privacy Protection can prefetch the tracking pixel, and Klaviyo explains that affected open events cannot reliably distinguish a human open from an automated one. Klaviyo on Apple privacy opens

A campaign can generate clicks while adding little purchasing. It can also increase orders while producing a poor margin or customer experience. Decide what success means before you have a tempting headline.

A compact test brief

Before scheduling, write down:

1. The business decision and exact campaign comparison
2. Eligibility, exclusions, randomization unit, and assignment method
3. Primary metric and smallest commercially useful effect
4. Sample-size plan, group split, and analysis method
5. Purchase window, refund policy, and planned stopping point
6. Background-message rules, stock assumptions, and customer safeguards
7. The person responsible for validating the data and reporting uncertainty

Attach that brief to the reviewed campaign so the learning survives beyond one dashboard screenshot.

Superdraft helps with the earlier part of this workflow: connecting brand context and timely signals to campaign opportunities and reviewable drafts. Run experiments and measure outcomes in the appropriate sending and analytics tools. The broader signal-to-send workflow closes the loop when those learnings inform the next campaign decision.

A strong measurement habit makes room for three answers: useful evidence of improvement, useful evidence of harm, and not enough evidence yet.