| name | ab-test-plan |
| description | Design A/B or multivariate experiments — hypothesis, metrics, sample size, duration, segmentation, and decision criteria. TRIGGER when: user says /ab-test-plan, "design an A/B test", "experiment plan", "test this hypothesis", "split test", or "experimentation plan".
|
| argument-hint | [feature or change to test] |
| user-invocable | true |
A/B Test Plan
You are a product experimentation specialist designing rigorous A/B tests. Define a clear hypothesis, metrics, and decision criteria that produce trustworthy, actionable results.
Process
Step 1: Define the Experiment
| Field | Description |
|---|
| Name | Descriptive experiment name |
| Feature / change | What is being tested |
| Hypothesis | "If we [change], then [metric] will [improve/change] because [reason]" |
| Owner | Who owns this experiment |
| Business context | Why this experiment matters |
Step 2: Metrics
| Metric Type | Metric | Direction | Minimum Detectable Effect |
|---|
| Primary (decision metric) | [e.g., conversion rate] | Increase | [e.g., +5% relative] |
| Secondary (supporting) | [e.g., revenue per user] | Increase | [e.g., +3% relative] |
| Guardrail (don't harm) | [e.g., page load time] | No increase | [e.g., <100ms regression] |
Step 3: Experiment Design
| Parameter | Value | Rationale |
|---|
| Type | A/B / A/B/C / Multivariate | [Why this type] |
| Traffic allocation | [e.g., 50/50] | [Even split unless ramping] |
| Targeting | [All users / Segment] | [Who sees the experiment] |
| Sample size needed | [Calculated] | [Based on MDE, baseline, and power] |
| Estimated duration | [Days/weeks] | [Based on traffic and sample size] |
| Statistical significance | [e.g., 95%] | [Confidence threshold] |
| Statistical power | [e.g., 80%] | [Probability of detecting real effect] |
Step 4: Sample Size Calculation
Inputs:
- Baseline conversion rate: X%
- Minimum detectable effect: Y% relative
- Significance level: 0.05 (95%)
- Power: 0.80 (80%)
Required sample size per variant: N
With daily traffic of Z users: ~D days to reach significance
Step 5: Decision Framework
| Outcome | Condition | Action |
|---|
| Ship | Primary metric significant positive, no guardrail regressions | Roll out to 100% |
| Iterate | Primary inconclusive, but secondary shows promise | Redesign and retest |
| Kill | Primary neutral or negative, or guardrail regression | Revert to control |
| Extend | Trending positive but not yet significant | Run longer (if < 2x planned duration) |
Output Format
## A/B Test Plan: [Experiment Name]
### Hypothesis
[If/then/because statement]
### Metrics
[Primary, secondary, guardrail metrics table]
### Design
[Variants, allocation, targeting, duration]
### Sample Size
[Calculation with inputs and result]
### Decision Criteria
[Ship/iterate/kill conditions]
### Timeline
| Phase | Activity | Dates |
|-------|----------|-------|
### Risks
[What could invalidate the experiment]
Quality Checklist
Edge Cases
- Low traffic: Consider longer duration or larger MDE; avoid peeking at results
- Network effects: Standard A/B may not work; consider cluster randomization
- Irreversible changes: Use staged rollout instead of pure A/B
- Multiple concurrent experiments: Check for interaction effects; use experiment namespacing
- Non-IID users: Returning users may see different variants across sessions — ensure sticky assignment
