run-ab-test

name	run-ab-test
description	Use when designing and executing an A/B test on a webpage, email, app, or ad to validate a conversion or engagement hypothesis
source	Optimizely "A/B Testing Guide"; VWO testing methodology; Kohavi et al. "Trustworthy Online Controlled Experiments" (2020, Cambridge)
tags	["ab-testing","experimentation","growth","conversion","statistics"]
related	["apply-ladder-of-causation"]
verified	true

Run A/B Test

Design, execute, and interpret a statistically valid A/B test that produces actionable, trustworthy results for product and marketing decisions.

Why This Is Best Practice

Adopted by: Google, Amazon, Netflix, Booking.com — all run thousands of A/B tests per year as the primary mechanism for product improvement Impact: Netflix reports testing drove $1B+ in annual subscriber retention improvement; Booking.com runs 1,000+ concurrent tests; Kohavi (2020) shows organizations with mature experimentation programs grow 2–4× faster than peers Why best: A/B testing is the gold standard for causal inference on live systems — without it, correlation is mistaken for causation, producing decisions that feel informed but reflect bias.

Sources: Kohavi, Tang & Xu "Trustworthy Online Controlled Experiments" (2020) Cambridge; Optimizely "Statistical Significance Calculator"; VWO "A/B Testing Guide" (2023)

Steps

1. Write the hypothesis — use this format: "Because [observation/insight], we believe [change] will cause [result] for [audience], which we'll measure by [metric]." A bad hypothesis cannot produce learning regardless of the outcome.
2. Define primary and secondary metrics — primary metric: the one metric that decides winner/loser; secondary metrics: guardrail metrics that must not deteriorate (e.g., conversion rate is primary; revenue per visitor is guardrail).
3. Calculate required sample size — use a power calculator: input baseline conversion rate, minimum detectable effect (MDE, typically 10–20%), significance level (α = 0.05), and power (β = 0.80); this gives required visitors per variant.
4. Estimate test duration — duration = required sample size per variant ÷ daily traffic to the test page; minimum 7 days (weekly seasonality), maximum 4–6 weeks (novelty effect fades); if duration >6 weeks, increase MDE or accept lower power.
5. Implement the test — use a testing platform (Optimizely, VWO, AB Tasty, Statsig, Eppo, or built-in platform tools); randomize at user level (not session level) to avoid re-assignment; verify traffic split (50/50 for standard A/B).
6. Run a pre-test check — run the experiment for 24h with both variants identical (A/A test); if results show significant difference, fix the randomization or data collection bug before proceeding.
7. Monitor for sample ratio mismatch (SRM) — check daily that variant traffic volumes match the intended split ±1%; SRM (e.g., variant A gets 60% of traffic instead of 50%) indicates a tracking bug that invalidates results.
8. Do not peek — avoid checking results before the planned end date; peeking and stopping early inflates false positive rates from 5% to 30%+ (Kohavi 2020); use sequential testing if early stopping is required.
9. Read results correctly — declare a winner only if: p < 0.05 (or 95% CI excludes zero) AND primary metric moved in the expected direction AND guardrail metrics are not significantly harmed.
10. Ship, learn, and document — if significant: ship the winner; if not significant: document the null result as a learning (null results are equally informative); add to the experiment log; plan the next iteration.

Rules

• One primary metric per test — multiple primary metrics require multiple hypothesis tests and Bonferroni correction; keep it simple.
• Never end a test early because the winner "looks obvious" — this is the most common source of invalid A/B test conclusions.
• Sample ratio mismatch must be investigated before reading results — SRM makes all statistics meaningless; it must be resolved or the test must be restarted.
• Minimum 7 days runtime for any web/app test — day-of-week effects can make early results reverse completely by day 7.
• Null results must be documented — a test that shows no effect tells you what not to build; it is equally valuable to the organization as a positive result.

Common Mistakes

• No pre-calculated sample size — ending the test when "it feels like enough data" produces wildly wrong conclusions.
• Changing the test mid-run — modifying the variant, traffic percentage, or metric after the test starts invalidates all prior data.
• Peeking and stopping early — the #1 source of false positives in A/B testing; p-values are not valid at arbitrary stopping points.
• Testing too many variants — 4+ variants require much larger sample sizes and longer run times; 2-variant tests are faster and more reliable.
• Ignoring guardrail metrics — a variant that improves CTR while halving revenue per session is not a winner.

When NOT to Use

• Traffic too low to reach statistical significance within a reasonable timeframe
• Changes that cannot be randomized at the user level (e.g., pricing changes visible to groups)
• Qualitative questions (use user interviews, surveys, or usability tests instead)