statistical-significance-in-tests

name	statistical-significance-in-tests
description	Use when evaluating A/B test results. Significance = the test result is unlikely due to chance.

Statistical Significance in Tests

Statistical significance: 95%+ confidence (p<0.05) standard. Need: 100+ conversions per variant. Stop tests at win OR sufficient sample.

Significance calculation

Need:

1. Conversion rate of control + variant
2. Sample size of each
3. Calculate Z-score (or use online calculator)
4. P-value < 0.05 = 95% confidence (industry standard)
5. P-value < 0.01 = 99% confidence (rigorous)

Sample size guidance

Required sample size depends on:

1. Baseline conversion rate (lower rate = more samples needed)
2. Desired lift detection (10% lift requires more samples than 50% lift)
3. Significance threshold (95% standard)

Rule of thumb: 100+ conversions per variant for moderate lift detection. Tools: VWO sample size calculator, Optimizely calculator.

Common test mistakes

1. Peeking — stopping at first win (false positive risk)
2. Insufficient sample — declaring winner too early
3. Multiple variables — can't isolate cause
4. Outlier data — single huge purchase skews result
5. Seasonal effects — test during normal traffic
6. Survivorship bias — only counting completed funnel

Where this fits in the X3 empire

Test rigor for CardPrepAI CRO program.