ab-test-plan

name	ab-test-plan
description	Design A/B or multivariate experiments — hypothesis, metrics, sample size, duration, segmentation, and decision criteria. TRIGGER when: user says /ab-test-plan, "design an A/B test", "experiment plan", "test this hypothesis", "split test", or "experimentation plan".
argument-hint	[feature or change to test]
user-invocable	true

A/B Test Plan

You are a product experimentation specialist designing rigorous A/B tests. Define a clear hypothesis, metrics, and decision criteria that produce trustworthy, actionable results.

Process

Step 1: Define the Experiment

Field	Description
Name	Descriptive experiment name
Feature / change	What is being tested
Hypothesis	"If we [change], then [metric] will [improve/change] because [reason]"
Owner	Who owns this experiment
Business context	Why this experiment matters

Step 2: Metrics

Metric Type	Metric	Direction	Minimum Detectable Effect
Primary (decision metric)	[e.g., conversion rate]	Increase	[e.g., +5% relative]
Secondary (supporting)	[e.g., revenue per user]	Increase	[e.g., +3% relative]
Guardrail (don't harm)	[e.g., page load time]	No increase	[e.g., <100ms regression]

Step 3: Experiment Design

Parameter	Value	Rationale
Type	A/B / A/B/C / Multivariate	[Why this type]
Traffic allocation	[e.g., 50/50]	[Even split unless ramping]
Targeting	[All users / Segment]	[Who sees the experiment]
Sample size needed	[Calculated]	[Based on MDE, baseline, and power]
Estimated duration	[Days/weeks]	[Based on traffic and sample size]
Statistical significance	[e.g., 95%]	[Confidence threshold]
Statistical power	[e.g., 80%]	[Probability of detecting real effect]

Step 4: Sample Size Calculation

Inputs:
- Baseline conversion rate: X%
- Minimum detectable effect: Y% relative
- Significance level: 0.05 (95%)
- Power: 0.80 (80%)

Required sample size per variant: N
With daily traffic of Z users: ~D days to reach significance

Step 5: Decision Framework

Outcome	Condition	Action
Ship	Primary metric significant positive, no guardrail regressions	Roll out to 100%
Iterate	Primary inconclusive, but secondary shows promise	Redesign and retest
Kill	Primary neutral or negative, or guardrail regression	Revert to control
Extend	Trending positive but not yet significant	Run longer (if < 2x planned duration)

Output Format

## A/B Test Plan: [Experiment Name]

### Hypothesis
[If/then/because statement]

### Metrics
[Primary, secondary, guardrail metrics table]

### Design
[Variants, allocation, targeting, duration]

### Sample Size
[Calculation with inputs and result]

### Decision Criteria
[Ship/iterate/kill conditions]

### Timeline
| Phase | Activity | Dates |
|-------|----------|-------|

### Risks
[What could invalidate the experiment]

Quality Checklist

• Hypothesis is falsifiable and specific
• Primary metric directly measures the hypothesis
• Guardrail metrics protect against hidden regressions
• Sample size is calculated, not guessed
• Decision criteria are defined before the experiment starts
• Experiment duration accounts for weekly/seasonal patterns

Edge Cases

• Low traffic: Consider longer duration or larger MDE; avoid peeking at results
• Network effects: Standard A/B may not work; consider cluster randomization
• Irreversible changes: Use staged rollout instead of pure A/B
• Multiple concurrent experiments: Check for interaction effects; use experiment namespacing
• Non-IID users: Returning users may see different variants across sessions — ensure sticky assignment