tetlocks-10-commandments

# Tetlock's 10 Commandments for Superforecasting ## Overview Philip Tetlock's 20-year Good Judgment Project identified "superforecasters"—the top 2% of predictors who consistently beat experts, pundits, and prediction markets. His 10 Commandments distill their practices into actionable principles: think probabilistically, update incrementally, beware biases, aggregate perspectives, track accuracy, embrace uncertainty. ## When to Use - Making predictions about uncertain future events - Improving decision-making under uncertainty - Evaluating others' forecasts (consultants, analysts, pundits) - Building forecasting teams or processes - Learning from prediction failures - Competing in prediction markets or tournaments ## The 10 Commandments ### 1. Triage - Focus Effort Where It Matters Don't waste time on unpredictable questions ("Will aliens visit?") or trivially predictable ones ("Will sun rise tomorrow?"). Focus on questions in the "Goldilocks zone": difficult but not impossible. **Example:** Skip: "Bitcoin price in 2030" (too noisy). Focus: "Will Bitcoin ETF be approved by SEC within 18 months?" (resolvable, valuable). ### 2. Break Down Complex Questions (Fermi Estimation) Decompose vague questions into answerable sub-questions. Aggregate sub-forecasts to avoid "holistic" guessing. **Example:** "Will fusion energy be commercially viable by 2040?" → Break into: "Will ITER succeed?" "Will private fusion reach breakeven?" "Will costs drop below $0.10/kWh?" "Will regulations allow deployment?" ### 3. Strike the Right Balance Between Inside and Outside Views Start with base rate (outside view): "What happened in similar cases?" Then adjust for unique details (inside view). Don't skip base rate, don't ignore specifics. **Example:** "Will our startup succeed?" Outside view: 90% startups fail. Inside view: Strong team, proven traction. Adjusted forecast: 30% success (better than base, but not ignoring statistics). ### 4. Strike the Right Balance Between Under- and Overreacting to Evidence Update beliefs when new evidence arrives, but don't whipsaw. Bayesian updating: Weight new evidence proportional to its quality and relevance. **Example:** Forecast: 60% chance of product launch success. New evidence: One beta tester complains. Don't drop to 20% (overreacting). Update to 55% (slight adjustment for weak signal). ### 5. Look for the Clashing Causal Forces Map competing forces pushing in opposite directions. Strong forecasters consider both bullish and bearish factors simultaneously. **Example:** EV adoption forecast: Bullish forces (battery costs falling, climate policy). Bearish forces (charging infrastructure gaps, consumer range anxiety). Weigh both, don't cherry-pick. ### 6. Strive to Distinguish as Many Degrees of Doubt as Data Allow Avoid round numbers (50%, 75%). Use granular probabilities (63%, 72%) when evidence supports it. Precision forces you to process information deeply. **Example:** Don't say "probably" or "75%." Say "67% confident based on 3 comparable precedents showing 2/3 success rate." ### 7. Strike the Right Balance Between Under- and Overconfidence Humans are overconfident on hard questions, underconfident on easy ones. Calibrate: Track "70% forecasts"—do they happen 70% of the time? **Example:** "90% sure this feature ships on time." Track 50 such forecasts. Only 60% shipped on time. You're overconfident. Recalibrate "90%" to "60%." ### 8. Look for the Errors Behind Your Mistakes When wrong, diagnose: Was it poor process (didn't check base rate), bad luck (correctly assessed 80% odds, hit the 20%), or missing information? **Example:** Predicted 80% chance competitor wouldn't launch in Q1. They did. Diagnosis: Missed insider information (process error), not bad luck. Fix: Build better intelligence gathering. ### 9. Bring Out the Best in Others, Bring Out the Best in Yourself Form diverse teams. Encourage dissent. Reward accuracy over ego. Practice intellectual humility. **Example:** Amazon's "disagree and commit"—require contrary views before alignment. Surfaces blind spots. ### 10. Master the Error-Balancing Cycle Iterate: Forecast → Track → Analyze errors → Adjust process → Repeat. Superforecasters spend 50% more time than average forecasters on post-mortems. **Example:** Quarterly review: "I forecasted 8 events at 70%, only 4 happened (50%). I'm overconfident. Next quarter, my 70% threshold requires stronger evidence." ## Example Application **Situation:** Product manager forecasting whether new feature will increase retention >10% in 3 months. **Application**: - **Commandment 1 (Triage)**: Predictable timeframe (3 months), measurable (10%), worth effort. Good question. - **Commandment 2 (Break down)**: Sub-questions: Will users discover feature? Will they understand it? Will they find it valuable? Will it reduce churn drivers? - **Commandment 3 (Outside/inside)**: Outside view: Similar features increased retention 5-15% (median 8%). Inside view: Our user research shows strong demand. Forecast: 60% chance of >10%. - **Commandment 4-5 (Evidence/forces)**: Bullish: User interviews positive. Bearish: Feature is complex, onboarding weak. Balance: Lower to 55%. - **Commandment 6 (Precision)**: Not "probably" or "60%." Use 55% based on specific evidence weighting. - **Commandment 7 (Calibration)**: Check past "55% forecasts"—did they happen ~55% of time? Yes? Maintain. - **Commandment 8-10 (Tracking)**: Set reminder to analyze in 3 months, learn from outcome, update process. **Outcome:** Feature increased retention 12%. Forecast was calibrated (55% for a yes, outcome was yes, though close). Process validated. Improve calibration over 100 forecasts. ## Anti-Patterns - ❌ Skipping base rates (inside view only) - ❌ Never updating forecasts ("set it and forget it") - ❌ Using vague language ("might," "could") instead of numbers - ❌ Judging forecast quality by outcome alone (resulting bias) - ❌ Not tracking forecasts → no calibration feedback loop - ❌ Overconfidence on every forecast (98% sure about everything) - ❌ Treating forecasting as innate talent vs. trainable skill ## Related - thinking-in-bets - bayes-theorem - base-rate-analysis - calibration - superforecasting