experiment-designer

name	experiment-designer
description	Designs A/B tests and product experiments with hypothesis structuring, sample size calculation, duration estimation, and result interpretation. Unified framework for both product and marketing experiments.
metadata	{"version":"1.0.0","category":"product","domain":"experimentation"}
triggers	["A/B test","experiment","hypothesis test","split test"]

Experiment Designer - A/B Test & Product Experiment Guide

Overview

Designs statistically sound experiments so you make decisions based on data, not opinions. Especially critical when traffic is low and every experiment counts.

---

1. When to Apply

Trigger	Behavior
Testing a product change	Full experiment design
"A/B test", "experiment"	Interactive experiment builder
Interpreting test results	Statistical analysis guide

---

2. Experiment Design Template

# Experiment: [Name]

## Hypothesis
> If we [change X], then [metric Y] will [improve/decrease] by [Z%],
> because [rationale based on user behavior/data].

## Design
| Field | Value |
|-------|-------|
| **Type** | A/B / A/B/n / Multivariate |
| **Primary Metric** | [e.g., conversion rate] |
| **Guardrail Metrics** | [metrics that should NOT worsen] |
| **Baseline** | [current value of primary metric] |
| **MDE** | [Minimum Detectable Effect, e.g., 5%] |
| **Significance** | 95% (α = 0.05) |
| **Power** | 80% (β = 0.20) |

## Sample Size & Duration
| Variant | Traffic Split | Required Sample | Est. Duration |
|---------|--------------|----------------|---------------|
| Control (A) | 50% | [N] | [days] |
| Variant (B) | 50% | [N] | [days] |

## Variants
### Control (A)
[Current experience — no changes]

### Variant (B)
[Specific change being tested]

## Success Criteria
- **Win**: Primary metric improves by ≥ MDE AND guardrails stable
- **Lose**: Primary metric does not improve OR guardrails degrade
- **Inconclusive**: Not enough data (extend or abandon)

## Risks & Mitigations
- [Risk]: [Mitigation]

---

3. Sample Size Quick Reference

For conversion rate experiments (95% significance, 80% power):

Baseline Rate	MDE 5%	MDE 10%	MDE 20%
1%	380K	95K	24K
5%	72K	18K	4.6K
10%	34K	8.6K	2.2K
20%	16K	3.9K	1K
50%	3.1K	780	200

Per variant. Total = N × number of variants.

---

4. Low-Traffic Strategies

When you don't have enough users for traditional A/B testing:

Strategy	When	How
Accept larger MDE	>20% effect expected	Reduces sample size
Sequential testing	Need early stopping	Bayesian approach
Before/after	No traffic for split	Compare time periods
Qualitative	<100 users	User interviews + usability tests
Fake door test	Testing demand	Measure clicks on non-existent feature

---

5. Result Interpretation

## Experiment Results: [Name]

| Metric | Control | Variant | Δ | p-value | Significant? |
|--------|---------|---------|---|---------|-------------|
| [Primary] | [N] | [N] | [+/-]% | [p] | Yes/No |
| [Guardrail] | [N] | [N] | [+/-]% | [p] | - |

## Decision: [Ship / Don't Ship / Iterate]
**Rationale**: [Why this decision based on data]

## Learnings
- [What we learned about user behavior]
- [Implications for future experiments]

---

6. Tool Coordination

Tool	Purpose
Write	Generate experiment design documents
Read	Reference existing experiment history

---

7. Boundaries

Will:

• Design statistically sound A/B tests
• Calculate sample sizes and durations
• Provide low-traffic experimentation alternatives
• Guide result interpretation

Will Not:

• Implement feature flags or A/B test code
• Run statistical analyses on real data
• Connect to analytics platforms
• Make ship/no-ship decisions (present data, you decide)