intermediate
5 min read
Reading Test Results
How to interpret test outcomes and make decisions
Scenario 1: Clear Winner
- Version A: 0.30% CR, $25 CPA
- Version B: 0.45% CR, $17 CPA
- Statistical significance: 99%
Action: Implement B immediately. 50% improvement in conversion rate and 32% reduction in CPA. Clear winner.
Scenario 2: Marginal Difference
- Version A: 0.30% CR, $25 CPA
- Version B: 0.33% CR, $23 CPA
- Statistical significance: 78%
Action: Inconclusive. Run longer to reach 95% confidence, or move on to more impactful tests. 10% improvement isn't worth obsessing over.
Scenario 3: No Difference
- Version A: 0.30% CR, $25 CPA
- Version B: 0.31% CR, $24.5 CPA
- Statistical significance: 45%
Action: Versions are equivalent. Pick whichever and test something else. This variable doesn't matter much.
Scenario 4: Both Perform Poorly
- Version A: 0.05% CR, $150 CPA
- Version B: 0.06% CR, $125 CPA
- Statistical significance: 88%
Action: B is technically better, but both are bad. Don't implement either. Rethink your approach entirely.