| name | model_development |
| description | The general model-building method for ANY market — framing, sample-size discipline, feature selection (user priors + data-driven), baselines, leakage-safe validation, calibration, persistence. |
| triggers | ["build a model","train a model","model development","predictive model","new model","build me a model"] |
Model development — the general method
This is the method for every market (winners, totals, lines, props). Market recipes
(build_a_totals_model, build_a_h2h_model) are worked examples OF this method,
never replacements for it.
1. Frame the problem
- What exactly is being predicted? One binary probability per selection is the
default (P(home win), P(over)). Name the market convention (whose line, which
book) before any code.
- What is the decision the probability feeds? A backtest entry needs the prob
BEFORE the price moves — timestamps are part of the model contract
(
record_predictions takes predicted_at; never backdate dishonestly).
2. Sample size BEFORE cleverness
- Rule of thumb: 10–20 outcomes per model parameter as a floor. A logistic
regression with 5 features wants 100+ decided events; anything fancier wants
several times that.
- "Last 20 games" is rarely enough — it estimates a mean with ±20%+ noise and one
parameter eats half of it. Prefer multiple seasons with recency weighting
(exponential decay) over tiny recent windows.
- Sport cadence changes everything: an MLB team plays 162 games/season, an NFL team
17 — the same "two seasons of data" is 324 events in one sport and 34 in the
other. Count EVENTS, not calendar time.
- Watch regime changes: rule changes, roster turnover, venue moves. Old data is
only valuable while the process that generated it still operates — say so when
you truncate history and why.
- When the data cannot support the model requested, SAY THAT PLAINLY and build the
smaller model that it can support.
3. Features: ask AND measure
- When the request is open-ended, ask the user which stats they believe matter
for this market — domain priors are real information and it is their model.
- Independently measure importance from the data (univariate signal, simple
permutation importance in run_python) — then report where the user's priors and
the data disagree, with numbers. Do not silently drop either.
- Fewer features beat more: every feature is a parameter and §2 already priced
parameters. Justify each one in a sentence.
4. Baseline first
- The market's vig-removed probabilities ARE the baseline (
value_finder shows
them). A model that cannot beat the closing line has no edge regardless of its
Brier — report your model AND the market baseline side by side.
- The second baseline is the base rate (home teams win X%). Beat both or say so.
5. Validate without lying to yourself
- Time-ordered splits only: train on the past, test on the future. Never shuffle
time; walk-forward (rolling refit) is the gold standard when you have volume.
- Leakage checklist before quoting any holdout number: no post-game stats in
pre-game features, no closing prices as features for an entry-time model, no
target-derived columns, holdout events strictly after every training event.
- Report holdout Brier/log-loss via
calibration_metrics (see quant_concepts
for how to read them), then save_model — it refuses uncalibrated models by
design. Record forward picks with record_predictions.
6. Honesty rules
- Print every number in run_python; quote only printed numbers.
- Probabilities only — sizing belongs to the bankroll manager; no locks, no
guarantees, the user decides.
