| name | experiment-design |
| description | Design product experiments to test hypotheses. Use when the user says "run an experiment", "A/B test this", "build-measure-learn", "validate this assumption", "test this hypothesis", "we don't know if this will work", "minimum viable test", "lean experiment", or wants to reduce risk before fully building a feature - even if they don't explicitly say "experiment".
|
Overview
Based on The Lean Startup by Eric Ries. The core insight: startups (and product teams) don't fail from lack of execution - they fail from executing on the wrong plan. The build-measure-learn loop is how you find the right plan faster. Every feature is a hypothesis. Every release is an experiment. Validated learning - not shipped features - is the unit of progress.
Workflow
Step 1: State the hypothesis
Format: "We believe [action/feature] will cause [user behavior change], resulting in [metric change], because [assumption]."
Example: "We believe adding a progress bar to onboarding will cause more users to complete setup, resulting in a 15% increase in activation rate, because users abandon when they can't see how much is left."
The "because" is critical - it forces you to name the assumption you're testing.
Step 2: Identify the riskiest assumption
Most hypotheses have multiple assumptions embedded. Find the one that, if wrong, makes the whole hypothesis collapse.
List assumptions - rank by risk - test the riskiest one first.
Step 3: Design the minimum viable test
What's the cheapest, fastest way to test the riskiest assumption?
Ries's ladder of experiment types (least to most expensive):
- Customer interview - ask 5 users if the problem exists
- Fake door / 404 test - show the feature, measure click rate before building
- Concierge MVP - do the job manually for a few users
- Wizard of Oz - appear automated, run manually behind the scenes
- A/B test - build both versions, split traffic
- Full release - ship to everyone
Start as low on the ladder as possible. Only move up when the cheaper test is insufficient.
Step 4: Define success criteria before running
Set the threshold before you see results. This prevents moving the goalposts.
"We will consider this experiment successful if [metric] changes by [amount] within [time period] with [sample size]."
Step 5: Calculate required sample size
For A/B tests: use a significance calculator. Undersized tests produce false results.
- Minimum detectable effect: what's the smallest change that matters?
- Baseline conversion rate: what's the current rate?
- Statistical significance: 95% standard (p < 0.05)
Step 6: Run the experiment and measure
- Don't peek at results early (peeking inflates false positives)
- Measure the primary metric and guard metrics
- Document what you observe, not what you expected
Step 7: Extract validated learning
Whatever the result:
- Confirmed - assumption was right, proceed with confidence
- Refuted - assumption was wrong, pivot or kill
- Inconclusive - sample too small or test too short, iterate
Ries: a failed experiment that teaches you something is more valuable than a successful one that teaches you nothing.
Anti-Patterns
1. HiPPO-driven decisions instead of experiments
Bad: "The CEO thinks we should add X, so we're adding X."
Good: "The CEO thinks X will increase retention. Let's run a 2-week experiment with 10% of users."
2. Moving the goalposts
Bad: Setting a 20% lift as success, seeing 8%, calling it a win anyway.
Good: Success criteria set before the experiment runs. Results are binary: met or not met.
3. Testing too many variables at once
Bad: Redesigning the entire onboarding and A/B testing it.
Good: Change one variable per experiment. Otherwise you can't attribute the result.
4. Skipping the concierge step
Bad: Building a full automated feature to test an unvalidated assumption.
Good: Do it manually for 10 users first. If the manual version doesn't work, the automated one won't either.
Quality Checklist
