| name | designing-content-experiments |
| description | Designs and documents A/B tests and experiments for content optimization. Use when testing messaging, content formats, or user experience hypotheses. |
Designing Content Experiments
Quick start
Collect or infer:
- Hypothesis to test
- Success metric and current baseline
- Audience and traffic volume available
- Technical constraints (what can be tested)
- Timeline and decision criteria
Then produce output using TEMPLATES.md. Validate with RUBRIC.md.
Workflow
- Define clear, falsifiable hypothesis
- Identify primary metric and guardrail metrics
- Calculate required sample size for statistical significance
- Design control and variant(s)
- Document targeting and traffic allocation
- Set duration and stopping criteria
- Define decision framework (what actions follow which outcomes)
- Run the rubric check. Revise until it passes.
Degrees of freedom
Freedom level: Low
- Default: follow templates exactly
- Allowed variation: number of variants (recommend 2-3 max), specific metrics tested—as long as rubric passes
- Strict constraints: Must include sample size calculation; must define stopping criteria; must have decision framework
State awareness
- First experiment: Start simple (A/B, single variable); focus on learning process
- Mature program: Can run multivariate tests; consider interaction effects
- Low traffic: Longer duration or sequential testing; consider Bayesian approach
- High stakes: Require higher confidence level (99% vs 95%)
References