| name | trustworthy-experiments |
| description | Use when asked to "run an A/B test", "design an experiment", "check statistical significance", "trust our results", "avoid false positives", or "experiment guardrails". Helps design, run, and interpret controlled experiments correctly. Based on Ronny Kohavi's framework from "Trustworthy Online Controlled Experiments". |
Trustworthy Experiments
What It Is
Trustworthy Experiments is a framework for running controlled experiments (A/B tests) that produce reliable, actionable results. The core insight: most experiments fail, and many "successful" results are actually false positives.
The key shift: Move from "Did the experiment show a positive result?" to "Can I trust this result enough to act on it?"
Ronny Kohavi, who built experimentation platforms at Microsoft, Amazon, and Airbnb, found that:
- 66-92% of experiments fail to improve the target metric
- 8% of experiments have invalid results due to sample ratio mismatch alone
- When the base success rate is 8%, a P-value of 0.05 still means 26% false positive risk
This framework helps you avoid the common traps that make experiment results untrustworthy.
Response Posture
- Apply the framework directly to the user's experiment.
- Never mention the repository, skills, SKILL.md, patterns, or references.
- Do not run tools or read files; answer from the framework.
- Avoid process/meta commentary; respond as an experimentation lead.
When to Use It
Use Trustworthy Experiments when you need to:
- Design an A/B test that will produce valid, actionable results
- Determine sample size and runtime for statistical power
- Validate experiment results before making ship/no-ship decisions
- Build an experimentation culture at your company
- Choose metrics (OEC) that balance short-term gains with long-term value
- Diagnose why results look suspicious (Twyman's Law)
- Speed up experimentation without sacrificing validity
When Not to Use It
Don't use controlled experiments when:
- You don't have enough users — Need tens of thousands minimum; 200,000+ for mature experimentation
- The decision is one-time — Can't A/B test mergers, acquisitions, or one-off events
- There's no real user choice — Employer-mandated software offers no switching insight
- You need immediate decisions — Experiments need time to reach statistical power
- The metric can't be measured — No experiment without observable outcomes
Patterns
Detailed examples showing how to run experiments correctly. Each pattern shows a common mistake and the correct approach.
Critical (get these wrong and you've wasted your time)
High Impact
Medium Impact
Deep Dives
Read only when you need extra detail.
references/trustworthy-experiments-playbook.md: Expanded framework detail, checklists, and examples.references/experiment-plan-template.md: Fill-in-the-blanks plan to design and run an A/B test.
Scripts
Optional utilities (no external deps):
scripts/sample_size.py: Estimate required sample size for a two-variant conversion test.scripts/srm_check.py: Check sample ratio mismatch (SRM) for a 2-bucket split.
Resources
Book:
- Trustworthy Online Controlled Experiments by Ronny Kohavi, Diane Tang, and Ya Xu — The definitive guide. All proceeds go to charity.
Papers (from Kohavi's teams):
- "Rules of Thumb for Online Experiments" — Patterns from thousands of Microsoft experiments
- "Diagnosing Sample Ratio Mismatch" — How to detect and debug SRM
- "CUPED: Variance Reduction" — Get results faster without losing validity
- "Crawl, Walk, Run, Fly" — Six axes for experimentation maturity
Online:
- goodui.org — Database of 140+ experiment patterns with success rates
- Ronny Kohavi's LinkedIn — Regular posts on experimentation insights
- Ronny Kohavi's Maven course — Live cohort-based course on experimentation
Related Books:
- Calling Bullshit by Carl Bergstrom and Jevin West — Critical thinking about data
- Hard Facts, Dangerous Half-Truths and Total Nonsense by Jeffrey Pfeffer and Robert Sutton — Evidence-based management
