experiment-design

name	experiment-design
description	Design and run lean experiments — hypothesis, cheapest test, success criteria, decision. Use when you need to validate before you build.

Experiment Design & Monitoring

The core lean loop: build-measure-learn. This skill structures the full cycle — from shaky assumption to validated learning — so you test cheaply and decide fast. Claude designs the experiment framework and monitors the results. You define what you're betting on and make the call when the data comes in.

When to Use

• You have a hypothesis but aren't sure it's worth a full build
• Leadership wants evidence before committing engineering resources
• You're choosing between multiple solutions and need data, not opinions
• A feature launched but you need to validate it's actually changing behavior
• You want to run an A/B test and need a rigorous setup

When NOT to Use

• You're exploring a problem space — use Discovery Process first
• You need to align stakeholders on whether to experiment — use Stakeholder Alignment
• You're reviewing existing metrics, not testing a hypothesis — use Metrics Review

The AI-Native Approach

Step	Time	Claude Does	You Do
Map assumptions	20 min	Structure assumptions from your context, rank by risk	Validate which assumptions are truly unproven
Design the experiment	30 min	Propose cheapest valid test, define metrics and criteria	Confirm feasibility with engineering, set guardrails
Build the brief	20 min	Draft experiment brief with hypothesis, method, success criteria	Review and share with the team
Monitor results	Ongoing	Track metrics, flag significance and guardrails	Interpret results with product context
Make the call	15 min	Summarize findings, frame the decision	Decide: ship, iterate, or kill

Process

Step 1: Map Assumptions (20 minutes)

Every product bet is a stack of assumptions. Most teams skip straight to building. Lean teams identify which assumptions are riskiest and test those first.

We're considering [initiative/feature]. Here are the assumptions behind it:

List every assumption we're making:
- About the user problem (is it real? how painful?)
- About the solution (will this approach work?)
- About the market (will people pay? switch? adopt?)
- About feasibility (can we build it in time? at this cost?)

For each assumption:
- How confident are we? (high / medium / low)
- What's the consequence if we're wrong? (high / medium / low)
- Is there existing evidence? (data, research, analogues)

Rank by risk: low confidence + high consequence = test this first.

Step 2: Design the Experiment (30 minutes)

The goal is the cheapest, fastest test that gives you a decision-quality signal. Not a perfect study — a lean experiment.

For our riskiest assumption: [assumption]

Design the cheapest valid experiment:

Hypothesis: "We believe [this] will cause [that] for [whom]."
Null hypothesis: "There is no meaningful difference when [this] changes."

Method options (from cheapest to most expensive):
1. Fake door / painted door test
2. Concierge / manual-behind-the-scenes
3. Wizard of Oz prototype
4. A/B test on existing feature
5. Limited beta / feature flag rollout
6. Full A/B test on new feature

For the recommended method:
- What do we build/change? (minimum implementation)
- Who do we expose it to? (segment, sample size)
- How long do we run it? (minimum duration for significance)
- What do we measure? (primary metric, secondary, guardrails)

Success criteria:
- Ship if: [primary metric] improves by [X%] with [confidence level]
- Iterate if: [signal is positive but below threshold]
- Kill if: [no signal or guardrail metric degrades by Y%]

Guardrail metrics (must not degrade):
- [Metric 1]: [threshold]
- [Metric 2]: [threshold]

Step 3: Build the Experiment Brief (20 minutes)

Draft an experiment brief:

1. Background: Why we're running this experiment (1-2 sentences)
2. Hypothesis: Stated as testable prediction
3. Method: What changes, for whom, for how long
4. Primary metric: What we measure, target, baseline
5. Secondary metrics: Supporting signals
6. Guardrail metrics: What must not degrade
7. Sample size and duration: Minimum for statistical significance
8. Success criteria: Ship / iterate / kill thresholds
9. Decision date: When we read the results and decide
10. Owner: Who makes the call

Keep it to one page. If the brief is longer than the experiment, you're over-engineering it.

Step 4: Monitor Results (Ongoing)

Here are the current experiment results: [paste data or describe]

Analyze:
- Are we reaching statistical significance? How many more days at current rate?
- Is the primary metric trending toward the success threshold?
- Are any guardrail metrics showing degradation?
- Are there segment-level differences worth investigating?
- Any external factors that could be confounding results?

Flag:
- 🟢 On track — continue running
- 🟡 Mixed signals — may need more time or investigation
- 🔴 Guardrail breach — consider stopping early
- ✅ Reached significance — ready for decision

Step 5: Make the Call (15 minutes)

The experiment has concluded. Here are the final results: [data]

Summarize:
1. Did the primary metric hit the success threshold?
2. Did any guardrail metrics degrade?
3. Were there surprising segment-level differences?
4. What did we learn that we didn't expect?

Frame the decision:
- Ship: results met criteria, here's the rollout plan
- Iterate: signal is there but we need to adjust [what] and retest
- Kill: no meaningful signal, here's what we learned and what to try next

Either way: what's the validated learning? What do we now know that we didn't before?

Output

• Ranked assumption map with risk scores
• Experiment brief (hypothesis, method, metrics, success criteria)
• Monitoring summary with significance tracking
• Decision recommendation with validated learning

Common Pitfalls

1. Testing too many things at once. One hypothesis per experiment. If you're testing three things simultaneously, you won't know which one caused the result.
2. Calling it too early. Wait for statistical significance. Early reads are noise. Set the duration upfront and stick to it unless a guardrail breaches.
3. No kill criteria. If you only define "success," you'll always find a way to declare victory. Define the kill threshold before you start.
4. Over-engineering the test. The cheapest valid test wins. A fake door that takes a day beats an A/B test that takes a sprint. Match the test cost to the decision stakes.
5. Learning nothing from failures. A killed experiment isn't a failure — it's a validated learning that saved you from building the wrong thing. Document what you learned and feed it back into discovery.

Related Skills

• Discovery Process — when you need to explore the problem space before you can form a hypothesis
• Metrics Review — for ongoing monitoring of experiment results alongside product health
• Growth & Retention Diagnostics — when experiment results reveal a deeper retention or growth issue