brier-score

# Brier Score **Category:** Decision-Making & Strategic Thinking **Source:** Glenn W. Brier (1950) / Meteorology / Forecasting Science **Practitioner Score:** 42/50 (Tier 1) ## Overview The Brier Score is a strictly proper scoring rule that measures the accuracy of probabilistic predictions. It calculates the mean squared error between predicted probabilities and actual outcomes, providing a single numeric measure of forecast quality. Lower scores indicate better accuracy, with 0 being perfect and 2 being worst possible. **Core Insight:** You can't improve what you don't measure. The Brier Score converts vague notions of "good forecasting" into quantifiable performance, enabling systematic improvement through calibration feedback. **Formula:** BS = (1/N) Σ(fᵢ - oᵢ)² - fᵢ = forecasted probability (0-1) - oᵢ = actual outcome (1 if event occurred, 0 if not) - N = number of forecasts ## When to Use - **Calibration assessment** - Are your 70% predictions actually happening 70% of the time? - **Forecaster comparison** - Which analyst/model produces more accurate predictions? - **Training feedback** - Providing objective performance scores for improvement - **Decision validation** - Evaluating quality of past predictions over time - **Model selection** - Comparing machine learning models or forecasting systems **Anti-patterns:** - Single predictions (need 20+ for statistical validity) - Non-probabilistic forecasts (binary yes/no) - Events without clear resolution criteria - Immediate feedback needs (requires outcome data) ## How to Execute ### Step 1: Record Probabilistic Forecast **Action:** Document your prediction as a probability between 0% and 100% - **Precision:** Use granular probabilities (65%, not "likely") - **Timestamp:** Record when prediction was made - **Resolution criteria:** Define exactly what constitutes "event occurred" - **Output:** Logged forecast with probability, date, and outcome definition ### Step 2: Wait for Event Resolution **Action:** Allow sufficient time for outcome to be determined - **Clear endpoint:** Specify resolution date/trigger in advance - **Unambiguous outcome:** 1 (occurred) or 0 (did not occur) - **No gaming:** Outcome determination must be independent of forecaster - **Output:** Resolved outcome (1 or 0) ### Step 3: Calculate Individual Forecast Error **Action:** Compute squared difference between forecast and outcome - **If event occurred (oᵢ = 1):** Error = (1 - fᵢ)² - **If event did not occur (oᵢ = 0):** Error = (0 - fᵢ)² = fᵢ² - **Example:** Predicted 70% (0.7), event happened → (1 - 0.7)² = 0.09 - **Output:** Single forecast Brier score ### Step 4: Aggregate Across Multiple Forecasts **Action:** Average squared errors across N predictions - **Minimum sample:** 20+ forecasts for meaningful assessment - **Formula:** BS = (Error₁ + Error₂ + ... + Errorₙ) / N - **Output:** Overall Brier score for forecast set ### Step 5: Interpret Score Against Benchmarks **Action:** Compare your score to reference points - **Perfect accuracy:** 0.00 (impossible in practice) - **Excellent:** < 0.10 (superforecaster level) - **Good:** 0.10 - 0.20 (well-calibrated forecaster) - **Average:** 0.20 - 0.30 (typical expert) - **Poor:** > 0.30 (worse than random guessing) - **Output:** Performance classification ### Step 6: Decompose into Calibration vs. Resolution **Action:** Break Brier score into skill components - **Calibration:** Are X% forecasts correct X% of the time? - **Resolution:** Can you distinguish different probability levels? - **Formula:** BS = Reliability - Resolution + Uncertainty - **Output:** Diagnostic breakdown identifying improvement areas ### Step 7: Implement Calibration Improvements **Action:** Use insights to adjust forecasting behavior - **Overconfident (too many extremes):** Pull predictions toward 50% - **Underconfident (clustered near 50%):** Increase differentiation - **Systemic bias:** Adjust all forecasts by consistent offset - **Output:** Updated forecasting protocol ## Real-World Examples **Weather Forecasting:** - Meteorologists tracked with Brier scores for 50+ years - Led to dramatic improvements in precipitation forecasting - Result: Today's 5-day forecast as accurate as 1-day forecast in 1970s **Good Judgment Project:** - Superforecasters averaged 0.15-0.18 Brier scores - Regular forecasters averaged 0.25-0.30 - Intelligence analysts (with classified info) averaged 0.30+ - Result: Validated that systematic methodology beats expertise **Sports Betting Markets:** - Bookmakers use Brier scores to evaluate odds accuracy - Prediction markets (PredictIt, Polymarket) track participant scores - Result: Efficient markets reflect well-calibrated probabilities ## Integration Points **Complements:** - **Superforecasting:** Brier score measures effectiveness of 10 commandments - **Calibration:** Primary diagnostic tool for improving calibration - **Bayesian Updating:** Tracks whether belief updates improve accuracy - **Prediction Markets:** Aggregation mechanism with Brier-optimized incentives **Enables:** - **Performance tracking:** Quantitative measure for deliberate practice - **A/B testing:** Compare forecasting methodologies empirically - **Incentive design:** Reward accurate probabilistic predictions ## Common Pitfalls **Pitfall 1: Conflating Low Score with Skill** - **Warning sign:** 0.05 score from 10 predictions all at 51%/49% - **Fix:** Check resolution - can you distinguish probability levels? **Pitfall 2: Ignoring Calibration Components** - **Warning sign:** Good overall score masking systemic bias - **Fix:** Decompose into reliability, resolution, and uncertainty **Pitfall 3: Small Sample Sizes** - **Warning sign:** Declaring "good forecaster" from 5 predictions - **Fix:** Require minimum 20-50 forecasts before drawing conclusions **Pitfall 4: Cherry-Picking** - **Warning sign:** Only tracking predictions you feel confident about - **Fix:** Commit to scoring ALL forecasts in domain upfront **Pitfall 5: Resolution Ambiguity** - **Warning sign:** Disputes about whether event "really" occurred - **Fix:** Define resolution criteria precisely when making forecast ## Multi-Category Extension For events with more than 2 outcomes (e.g., election with 3+ candidates): **Formula:** BS = (1/N) Σ Σⱼ (fᵢⱼ - oᵢⱼ)² **Example:** Three-way race forecast of A=60%, B=10%, C=30% - If A wins: (1-0.6)² + (0-0.1)² + (0-0.3)² = 0.16 + 0.01 + 0.09 = 0.26 - If B wins: (0-0.6)² + (1-0.1)² + (0-0.3)² = 0.36 + 0.81 + 0.09 = 1.26 - If C wins: (0-0.6)² + (0-0.1)² + (1-0.3)² = 0.36 + 0.01 + 0.49 = 0.86 **Note:** Multi-category scores range from 0 (perfect) to 2 (worst possible). ## Validation Checklist - [ ] All forecasts are probabilistic (0-100%), not binary - [ ] Resolution criteria defined before outcome known - [ ] Outcomes recorded honestly (no retroactive "adjustments") - [ ] Sample size sufficient (20+ forecasts minimum) - [ ] Score decomposed into calibration and resolution components - [ ] Comparison made to baseline (random guessing, other forecasters) - [ ] Trend tracked over time to measure improvement ## Practical Tips **Tracking System:** - Spreadsheet columns: Date, Question, Forecast %, Outcome (0/1), Error² - Running average in final column - Monthly review of calibration plots **Calibration Plot:** - X-axis: Your forecasted probability (grouped into bins) - Y-axis: Actual frequency of occurrence - Perfect calibration = diagonal line **Improvement Signals:** - Score decreasing over time (learning) - Calibration plot approaching diagonal - Resolution increasing (more differentiation) ## Further Reading - **Original paper:** "Verification of Forecasts Expressed in Terms of Probability" - Glenn W. Brier (1950) - **"Superforecasting"** - Philip Tetlock (Brier score as key metric) - **Good Judgment Open:** Platform with automatic Brier score calculation - **Cultivate Labs:** Forecasting guide with Brier score tutorials