| name | ab-test |
| description | Design an A/B test - hypothesis, single variable, variants, sample size/duration, and the metric - and interpret results honestly. Use when optimizing a page, ad, email, or funnel step. |
ab-test
Test the high-leverage thing, change one variable, and read the result honestly.
Process
- Hypothesis: "Because [insight], changing [element] will [effect] on [metric]." Test the high-leverage
element first (headline, offer, CTA, hero) - not the button colour.
- One variable between A and B; everything else equal.
- Metric & guardrail: the primary conversion metric, plus a guardrail metric you don't want to harm.
- Sample size & duration: estimate the needed sample for a meaningful result; run full weeks; don't peek
and stop early.
- Interpret honestly: report sample size and whether the difference is actually significant - small or
short tests often aren't. No declaring a winner from noise.
Output
- A test plan: hypothesis, variants, primary + guardrail metric, sample/duration, and a results-reading rule.
Guardrails
- Don't claim significance without the data (
truth-telling); report sample size + result honestly.
- One variable; avoid stopping early on a lucky swing.
