Incrementality Testing
Controlled experiments that prove whether advertising campaigns cause conversions or merely correlate with them
A specialized form of A/B testing where the variable is "ads vs. no ads."
How It Works
- Define your target audience
- Randomly split into test group (80%, sees ads) and control group (20%, doesn't see ads)
- Run campaign for 2-4 weeks
- Measure conversions in both groups
- Calculate incremental lift: (test conversion rate) - (control conversion rate)
Both groups must be truly equivalent—matched on behavioral onchain targeting patterns, activity levels, and wallet characteristics.
The Holdout Tradeoff
You intentionally don't show ads to 10-20% of your qualified audience. This feels wasteful—you're leaving conversions on the table. But without a control group, you can't prove causality.
Think of it as insurance: accept 10-20% reach reduction during testing to gain certainty about whether your advertising actually works.
Sizing the Control Group
10-15% holdout: Large campaigns (1M+ impressions) 15-20%: Medium campaigns (100K impressions)20-30%: Small tests (<50K impressions)
You need enough conversions in both groups for statistical significance. Minimum 50 per group, comfortable at 100+, high confidence at 500+. Smaller campaigns need larger holdouts to reach significance.
When to Test
Essential before: Scaling campaigns 10x+, making platform decisions, reporting to stakeholders, setting long-term budgets
Good practice: Quarterly validation, major campaign changes, market condition shifts
Not necessary: Initial small brand awareness campaigns (<$10K spend)
Why Most Don't Do It
Most advertisers optimize endlessly for attributed conversions without ever proving those conversions are incremental. They're potentially scaling campaigns that don't work, claiming credit for organic growth, and miscalculating ROI.
Incrementality testing is the most important test you can run because it validates whether advertising works at all.