superforecasting

# Superforecasting **Category:** Decision-Making & Strategic Thinking **Source:** Philip Tetlock & Dan Gardner - "Superforecasting: The Art and Science of Prediction" (2015) **Practitioner Score:** 46/50 (Tier 1 Canonical) ## Overview Superforecasting is a systematic methodology for making accurate probabilistic predictions, developed through Philip Tetlock's Good Judgment Project. The research identified "superforecasters" - individuals who consistently outperform experts, pundits, and even intelligence analysts by 30% or more. The framework codifies their techniques into 10 actionable commandments plus deliberate practice protocols. **Core Insight:** Prediction accuracy is a learnable skill. By combining rigorous process (break problems down, update beliefs incrementally, balance outside/inside views) with calibrated probabilistic thinking and error analysis, anyone can dramatically improve forecasting ability. **Evidence:** Good Judgment Project participants trained in these techniques beat CIA analysts with classified information access. ## When to Use - **Strategic decisions** - Market entry, product launches, competitive moves - **Resource allocation** - Investment decisions, hiring plans, capacity planning - **Risk assessment** - Project timelines, crisis likelihood, threat analysis - **Competitive intelligence** - Predicting competitor actions, market shifts - **Long-range planning** - Technology adoption curves, regulatory changes **Anti-patterns:** - Decisions already made (confirmation bias trap) - Binary yes/no thinking without probability ranges - One-shot predictions without learning feedback - High-emotion situations without cooling-off period ## How to Execute ### Step 1: Triage - Choose Forecast-Worthy Questions **Action:** Focus effort on questions where hard work pays off - **Skip "clocklike" questions:** Simple rules/trends suffice (e.g., "Will sun rise tomorrow?") - **Skip "cloud-like" questions:** Too random even for models (e.g., "Will World War III start?") - **Target Goldilocks zone:** Difficult but tractable with analysis - **Output:** Prioritized list of forecastable questions ### Step 2: Fermi-Style Decomposition **Action:** Break intractable problems into tractable sub-problems - **Identify knowable parts:** What can be researched or estimated? - **Expose unknowables:** Flush ignorance into the open - **Example:** "Will product X succeed?" → market size × conversion rate × pricing × competition - **Output:** Hierarchical breakdown with researchable components ### Step 3: Balance Outside View (Base Rates) and Inside View **Action:** Start with reference class, adjust for specifics - **Outside view first:** How often do things of this sort happen in situations of this sort? - **Find comparison class:** Similar products, markets, technologies - **Inside view adjustment:** What makes this case unique? - **Output:** Base rate probability + reasoned adjustments ### Step 4: Incremental Belief Updating **Action:** Update forecasts as new evidence arrives - not too much, not too little - **Avoid under-reacting:** Ignoring genuinely new information - **Avoid over-reacting:** Jumping to conclusions from noisy signals - **Bayesian mindset:** P(H|E) = P(E|H) × P(H) / P(E) - **Output:** Revised probability with explicit reasoning ### Step 5: Consider Clashing Causal Forces **Action:** Map arguments for AND against your thesis - **Steel-man opposition:** Understand counterarguments deeply - **Force interaction:** How do conflicting factors balance? - **Example:** "AI adoption" → Cost savings (pro) vs. Implementation complexity (con) - **Output:** Two-column list of forces with relative weights ### Step 6: Granular Probability Estimates **Action:** Translate vague hunches into numeric probabilities - **Avoid vague language:** "Likely" means what exactly? - **Use fine gradations:** 55% vs. 60% forces precision - **Calibration practice:** Track how often your 70% predictions come true - **Output:** Numeric probability (e.g., 68%) with confidence range ### Step 7: Balance Under/Overconfidence **Action:** Manage trade-off between decisiveness and humility - **Calibration:** Are your 80% predictions correct 80% of the time? - **Resolution:** Can you distinguish 60% from 80% events? - **Avoid extremes:** "Definitely" (99%+) and "No way" (1%-) rarely justified - **Output:** Calibrated probability that neither overstates nor understates certainty ### Step 8: Learn from Errors Without Hindsight Bias **Action:** Analyze mistakes while resisting "I knew it all along" - **Pre-mortem:** Before outcome, write why forecast might fail - **Post-mortem:** After outcome, compare to pre-mortem (not current knowledge) - **Brier score tracking:** Measure accuracy over time - **Output:** Error log with root cause analysis ### Step 9: Leverage Team Wisdom **Action:** Master collaborative forecasting dynamics - **Perspective-taking:** Reproduce others' arguments to their satisfaction - **Precision questioning:** Help clarify without judgment - **Constructive confrontation:** Disagree without being disagreeable - **Output:** Team forecast incorporating diverse viewpoints ### Step 10: Master the Error-Balancing Bicycle **Action:** Treat commandments as guidelines requiring constant judgment - **No rigid rules:** Every situation is unique - **Deliberate practice:** Forecasting is skill built through repetition - **Feedback loops:** Clear, unambiguous results inform learning - **Output:** Continuous improvement trajectory ## Real-World Examples **Good Judgment Project (2011-2015):** - Superforecasters beat intelligence analysts by 30% - Ordinary people trained in these methods outperformed experts - Result: Validated that forecasting is a learnable skill **Prediction Markets (Metaculus, Good Judgment Open):** - Calibrated forecasters consistently identify probability ranges - Aggregated predictions outperform individual experts - Result: Operational use in policy, business, research **Tech Industry Product Forecasting:** - Decompose adoption rates into addressable market × conversion × retention - Update predictions as beta data arrives - Result: Better resource allocation, realistic roadmaps ## Integration Points **Complements:** - **Brier Score:** Measures superforecasting accuracy quantitatively - **Fermi Estimation:** Powers Step 2 decomposition - **Bayes' Theorem:** Mathematical foundation for belief updating - **Calibration:** Essential skill for Step 7 confidence management - **Base Rate Analysis:** Core of Step 3 outside view **Contrasts with:** - **Expert Intuition:** Systematic process vs. gut feel - **Punditry:** Probabilistic humility vs. confident pronouncements - **Binary Thinking:** 65% vs. "yes/no" ## Common Pitfalls **Pitfall 1: Anchoring on Initial Estimate** - **Warning sign:** Forecast barely moves despite major news - **Fix:** Explicit belief updating protocol after each information update **Pitfall 2: Ignoring Base Rates** - **Warning sign:** "This time is different" without evidence - **Fix:** Always start with outside view reference class **Pitfall 3: Overconfidence in Extremes** - **Warning sign:** Many forecasts at 5% or 95% - **Fix:** Force justification for extreme probabilities, track calibration **Pitfall 4: Confirmation Bias in Research** - **Warning sign:** Only seeking evidence supporting initial view - **Fix:** Actively search for disconfirming evidence (Step 5) **Pitfall 5: No Feedback Loop** - **Warning sign:** Making predictions but never tracking outcomes - **Fix:** Maintain prediction log with dates, probabilities, and resolutions ## Validation Checklist - [ ] Question is in Goldilocks zone (neither trivial nor impossible) - [ ] Problem decomposed into researchable sub-components - [ ] Base rate identified from reference class - [ ] Both supporting and opposing forces mapped - [ ] Probability is numeric and granular (not vague language) - [ ] Calibration tracked over time (70% predictions = 70% accuracy) - [ ] Forecast updated as new information arrives - [ ] Pre-mortem written before outcome known - [ ] Team input incorporated through structured dialogue ## Key Metrics **Brier Score:** Primary accuracy measure (0 = perfect, 2 = worst) - Formula: (1/N) Σ(forecast - outcome)² - Target: < 0.20 for well-calibrated forecaster **Calibration:** Do your X% predictions happen X% of the time? - Plot predicted probability vs. observed frequency - Perfect calibration = diagonal line **Resolution:** Can you distinguish different probability levels? - Difference in outcomes between 60% and 80% forecasts - Higher resolution = better discrimination ## Further Reading - **"Superforecasting"** - Philip Tetlock & Dan Gardner (2015) - **"Expert Political Judgment"** - Philip Tetlock (2005) - **Good Judgment Open:** Free forecasting platform with training - **Metaculus:** Advanced forecasting community - **"The Signal and the Noise"** - Nate Silver (Bayesian thinking)