| name | heater-appb-calibration-backtest |
| description | Use when proving a HEATER algorithm works — calibrating constants, backtesting projections or recommendations, validating win probabilities, adding uncertainty intervals, checking calibration drift — or before letting any new math drive user-facing recommendations. Also when asked "is this signal real?" or when a calibration run finds no clear winner. |
Calibration & Backtesting — proving the math before it advises anyone
Core principle: new decision math is gated, not vibes-checked. The shipped pattern: research signals were calibrated (slice 2) BEFORE the API exposed them (slice 3); the player model ships with its own validation gate (src/player_model/validation.py) run on real game logs (406 players / 3,837 player-weeks).
The toolbox
| Tool | Use | Notes |
|---|
python scripts/run_backtest.py --quick | replay historical MLB weeks, score optimizer recommendation accuracy | src/optimizer/backtest_runner.py (RMSE, Spearman, bust rate, lineup grading) |
run_historical_backtest(records_df) | grade against Section G targets: rank_mae < 1.5, cat_win_rmse < 8, trade_prediction_spearman > 0.4, projection MAE (HR < 5, AVG < 0.020) | data-agnostic — computes whatever the supplied columns support; no historical data ships with the repo (guard test_historical_backtest.py) |
python scripts/calibrate_sigmoid.py --full | grid-search urgency k-values → writes CONSTANTS_REGISTRY | consumers read the registry at runtime, so results apply without restart |
python scripts/calibrate_repl_baselines.py | refresh replacement levels from league standings | annual cadence |
python scripts/calibrate_research_signals.py | backtest the recency blend on ~58k game_logs rows | wrote research_w_l7/l14/l30 = 0.70/0.20/0.10 into the registry |
python scripts/compute_empirical_stats.py | compare correlation/CV defaults vs real MLB data | scenario-generator inputs |
Split-conformal (src/engine/projections/conformal.py, E10a + src/engine/output/backtest_calibration.py) | distribution-free projection intervals | the Phase-6 calibration foundation; tests tests/test_conformal.py |
| P6 calibration drift report | detect drifting constants | REPORT-ONLY by design — it never auto-tunes; a human reviews before any constant moves |
Rules
- Calibration must be able to lose. The research grid returns
best=None (abstains) on empty/degenerate data — a calibrator that always crowns argmax is fitting noise. Report effect sizes honestly even when weak: the recency signal won at ~0.06 next-week persistence, and that number went into the docs rather than being rounded up to "works".
- Write results into
CONSTANTS_REGISTRY, and patch the registry in tests. The sigmoid calibrator originally patched legacy module aliases while consumers read the registry — calibration silently no-opped. Guard: test_sigmoid_calibrator_patches_registry.py (calibrators + sensitivity analysis must target the runtime read-path).
- No data leakage. The survival calibrator SKIPS with a WARNING when ADP is missing rather than backfilling from
actual_pick (guard test_survival_calibrator_skips_on_missing_adp.py). If the feature wouldn't exist at prediction time, it doesn't exist at calibration time.
- Determinism: fixed seeds for calibration/backtest runs; date-relative logic patches
src.et_today._utcnow, never datetime.now — evening-ET flake otherwise.
- Backtests on ambient data don't survive CI. A leaders perf test needed the live 26MB DB → 0 rows + flaky in worktrees; calibration runs are operator scripts against the real local DB, while their GUARDS are count-based and DB-free.
- Replay honesty: point-in-time projections aren't stored, so historical replays score past dates with CURRENT projections — carry
proxy_caveat=True to the UI (streaming replay pattern); exact replay needs a scheduler-written snapshot table (owner-gated).
- Sensitivity before shipping a constant:
run_sensitivity_analysis perturbs registry constants (mock.patch) — a result that flips under small perturbation isn't a result.
When NOT to apply
Pure refactors with frozen-reference equivalence tests don't need recalibration — equivalence IS the proof. Trade-level / matchup-level outcome backtests need external week-by-week history the repo doesn't ship: owner-gated, note as follow-up rather than blocking. Don't run --full calibrations casually mid-feature; they're operator actions with registry-wide effect.
