calibrate-probabilities

name	calibrate_probabilities
description	How to evaluate and fix probability calibration — holdout discipline, Brier/log-loss reading, Platt-style rescaling.
triggers	["calibrate","calibration","brier","log loss","log-loss","overconfident"]

Calibrating probabilities

A model that says 70% should be right ~70% of the time. Calibration is measured, never assumed.

Measure (always on holdout)

• calibration_metrics(pairs) where pairs = holdout {prob, outcome} rows.
• Brier: mean squared error. 0 = oracle; 0.25 = coin flip on a balanced set; beating the market baseline matters more than the absolute number.
• Log-loss: punishes confident wrongness. If log-loss looks much worse than Brier, the model is overconfident in its tails.

Fix overconfidence (in run_python)

• Shrink toward the base rate: p' = w * p + (1 - w) * base_rate, fit w on a validation slice (grid over w ∈ [0.5..1.0] minimising log-loss is fine).
• Platt-style: fit logistic regression of outcome on logit(p) — two parameters, hard to overfit; refuse fancier recalibration without more than ~200 samples.
• Re-run calibration_metrics AFTER rescaling and report both before/after.

Persist

save_model with the post-calibration metrics and the rescaling parameters in params — the next session must be able to reproduce the pipeline from the row.