calibration

# Calibration ## Overview Calibration is the alignment between predicted probabilities and actual outcome frequencies - if you claim "70% confidence" and those predictions come true 70% of the time, you're well-calibrated. If they happen 40% or 90% of the time, you're miscalibrated (overconfident or underconfident). Developed through Philip Tetlock's decade-long forecasting research, calibration training transformed amateur "superforecasters" into predictors who outperformed professional intelligence analysts by 30%. The framework is metacognitive: you're not just predicting outcomes, you're learning to accurately assess the strength of your own beliefs. Most people are terrible at this - they confuse "feels very certain" with "is actually 95% likely" - leading to systematic overconfidence in business decisions, project timelines, and strategic bets. Calibration requires rigorous tracking: record predictions with explicit percentages, measure Brier scores (accuracy metric), identify systematic biases (always too high/low?), and adjust. Over hundreds of predictions, well-calibrated forecasters develop intuitive sense of uncertainty levels. ## When to Use - Improving forecast accuracy in business planning, sales projections, product timelines - Building prediction markets or forecasting tournaments within organizations - Evaluating expertise - are "industry experts" actually calibrated or just confidently wrong? - Training teams to express uncertainty honestly instead of false precision - Debugging overconfidence bias in founders, executives, product managers - Assessing risk - calibrated probabilities enable better expected value calculations ## The Process ### Step 1: Make Explicit Probability Predictions Transform vague language ("probably," "unlikely," "very confident") into precise percentages. Force yourself to commit to a number. **Vague → Calibrated:** - ❌ "We'll probably hit Q3 revenue target" - ✅ "I'm 65% confident we hit Q3 revenue target" - ❌ "This hire seems great" - ✅ "I'm 80% confident this hire succeeds in role for 18+ months" **Key technique:** Use 10% increments for granularity (50%, 60%, 70%) instead of just "likely/unlikely." ### Step 2: Track Predictions and Outcomes Systematically Create a prediction log with: (1) prediction statement, (2) probability assigned, (3) date made, (4) outcome (success/failure), (5) date resolved. **Tracking spreadsheet columns:** - Prediction text - Probability (%) - Date predicted - Actual outcome (1 = happened, 0 = didn't happen) - Date resolved - Category (hiring, product, revenue, etc.) **Volume matters:** You need 50-100+ predictions to measure calibration accurately. Single predictions don't reveal systematic bias. ### Step 3: Calculate Brier Score Brier score measures prediction accuracy on 0-2 scale, where 0 = perfect, 2 = worst possible. Lower scores = better calibration + resolution. **Formula:** Brier Score = (1/n) × Σ(forecast - outcome)² - Forecast = your probability (0.7 for 70%) - Outcome = 1 if happened, 0 if didn't - n = number of predictions **Example:** - Prediction: "70% chance we ship feature by Q3" → shipped on time - Brier: (0.7 - 1)² = 0.09 - Prediction: "90% chance candidate accepts offer" → they declined - Brier: (0.9 - 0)² = 0.81 (large penalty for overconfidence) **Benchmark:** Superforecasters achieve Brier scores of 0.15-0.25. Average forecasters: 0.35-0.50. ### Step 4: Analyze Calibration Curve Group predictions by probability bucket (60-70%, 70-80%, etc.) and compare predicted probability to actual frequency of outcomes. **Perfect calibration:** - 60% predictions → 60% actually happened - 80% predictions → 80% actually happened - 90% predictions → 90% actually happened **Overconfidence pattern:** - 80% predictions → only 60% happened (systematically too confident) **Underconfidence pattern:** - 70% predictions → 85% happened (leaving accuracy on table) **Visualization:** Plot predicted probability (x-axis) vs. actual frequency (y-axis). Perfect calibration = diagonal line. ### Step 5: Identify and Correct Systematic Biases Look for patterns in miscalibration - are you overconfident in certain domains? Underconfident in others? Do you cluster predictions at 50%, 80%, 90% instead of using full range? **Common biases to detect:** - **Domain overconfidence:** Great calibration in engineering estimates, terrible in hiring - **50% clustering:** Defaulting to 50% when uncertain (should use wider range) - **Overconfidence in rare events:** Saying 95% for things that happen 75% of time - **Outcome-blind anchoring:** Anchoring on initial view, not updating with new data **Correction strategy:** Apply domain-specific adjustments. If you're always 15 points too high on hiring, mentally subtract 15 points before recording prediction. ### Step 6: Practice Deliberate Calibration Training Calibration improves through feedback loops - predict, observe outcome, update mental model, repeat. Superforecasters practice daily. **Training exercises:** - **Trivia calibration:** Predict "% confident I know the answer" then check - forces you to distinguish 60% from 90% - **Same-day resolution:** Predict events with quick feedback (will meeting run over time? will build pass?) - **External benchmarks:** Compare your forecasts to prediction markets, other forecasters - **Pre-mortems:** Before launching, predict "% chance we encounter X problem" - revisit after launch **Key insight:** Calibration is a skill, not a trait. Even poor forecasters improve 20-30% with training. ## Example Application **Situation:** Product manager predicting feature ship dates over 6 months, tracking 25 predictions. **Application:** - **Initial predictions:** Average 85% confidence, actual delivery rate: 60% (severe overconfidence) - **Brier score:** 0.42 (below average) - **Pattern identified:** Always underestimate integration complexity, overestimate team velocity - **Adjustment:** Reduce confidence by 15 points on features requiring backend integration - **Month 4-6 results:** 72% confidence, 68% delivery rate (much better calibration) - **New Brier score:** 0.28 (approaching superforecaster level) **Outcome:** PM now expresses uncertainty honestly, sets realistic stakeholder expectations, and prioritizes better (focuses on high-confidence ships). ## Example Application 2 **Situation:** Executive team forecasting quarterly OKR achievement, historically claiming "80% confident" but only hitting 50%. **Application:** - **Intervention:** Implement prediction tracking across 12 OKRs per quarter - **Training:** Each exec predicts individual OKR success probability before quarter starts - **Q1 calibration:** 78% average prediction, 45% actual (massive overconfidence, Brier 0.48) - **Feedback session:** Review calibration curves, identify systematic biases (underestimate competitor responses, overestimate user adoption) - **Q2-Q3 adjustment:** Average predictions drop to 62%, actual outcomes rise to 58% (better planning + realistic confidence) - **Q3 Brier score:** 0.31 (significant improvement) **Result:** More honest uncertainty led to better resource allocation, contingency planning, and stakeholder trust. ## Anti-Patterns - ❌ Vague confidence language ("probably," "likely") instead of explicit percentages - can't measure calibration - ❌ Never tracking predictions - calibration requires feedback loops, not just guessing - ❌ Clustering predictions at 50%, 80%, 95% - use full granularity (55%, 72%, 88%) - ❌ Judging calibration on <20 predictions - need volume to detect systematic bias - ❌ Ignoring domain differences - calibration varies across areas of expertise - ❌ Confusing confidence with accuracy - you can be perfectly calibrated at 60% (not great accuracy) - ❌ Updating predictions without recording original - defeats feedback loop learning - ❌ Treating Brier score as only metric - calibration curve reveals different issues than resolution ## Related - thinking-in-bets (probabilistic decision-making that depends on calibrated probabilities) - superforecasting (broader forecasting framework where calibration is core skill) - confidence-intervals (statistical method for expressing uncertainty ranges) - bayesian-updating (how to update probabilities as new evidence emerges) - brier-score (mathematical measure of forecast accuracy and calibration)