| name | co-growth-experiment-design |
| description | Experiment design skill. Hypothesis-driven A/B test planning with sample size
calculation, MDE definition, guardrail metrics, and runtime estimation.
Use when DESIGNING A/B tests, calculating sample sizes, defining hypotheses,
setting guardrail metrics, or planning experiment rollout.
|
Experiment Design
Hypothesis-driven A/B test design with statistical rigor.
Use This Skill When
- Designing a new A/B test or multivariate experiment.
- Formulating a testable hypothesis with success criteria.
- Calculating required sample size given MDE and baseline.
- Defining guardrail metrics that must not degrade.
- Estimating experiment runtime.
Workflow
- Formulate hypothesis: "If [change], then [metric] will [direction] by [MDE]."
- Define metrics: primary, secondary, and guardrail.
- Calculate sample size: inputs = baseline rate, MDE, α=0.05, β=0.20.
- Estimate runtime: sample size ÷ daily traffic × seasonality buffer.
- Document randomization unit, allocation ratio, and exclusion criteria.
- Save experiment plan using template.
Deliverables
report.md: experiment rationale and design summary.results/experiment-plan.md: full plan using template.results/sample-size-calc.md: calculation with all inputs.
Quality Gates
If any gate fails: identify the issue, fix, and re-validate.
Gotchas
- An MDE of "any improvement" is not valid. Require a minimum business-meaningful threshold (e.g., +2pp conversion).
- Experiments shorter than 2 weeks miss weekly seasonality cycles and produce biased estimates.
- Guardrail metrics added after launch are post-hoc and invalidate the pre-registered analysis plan.
Validation Loop
- Generate experiment design
- Check: falsifiable hypothesis, sample size calc, guardrails, runtime ≥ 2 weeks
- If any check fails → revise design
- User approval before launch ⏸️
