measure-dont-believe

name	measure-dont-believe
description	Discipline for algorithm and ranking changes — hand-designed signals are usually wrong in direction, not just magnitude. Build an offline backtest first; the backtest script becomes the behavior spec; numeric gates decide mergeability.

Measure, Don't Believe

Real case: a recommendation system was to be fixed with a hand-designed "repurchase cycle" model — intuitive, elegant, plausible ("customers buy every N days; recommend when overdue"). The backtest showed the direction was inverted: P(rebuy) decreased monotonically with time since last purchase. Meanwhile the production vector-similarity ranking scored worse than recommending the same top-10 to everyone (4.5% vs 13.6% HitRate@10). A calibrated-from-data model hit 56%. The designer's intuition lost twice; counting won twice.

The discipline

1. Backtest before designing. Build a leave-last-event-out (or time-split) replay from historical data BEFORE proposing an algorithm. Include a dumb baseline (global popularity / majority class); anything that loses to it is disqualified regardless of how sophisticated it looks.
2. Never hand-tune signal directions or weights. If two scores must combine, either calibrate both to a common empirical unit (e.g. bucketed probability lookup tables with holdout calibration + Laplace smoothing) or compose by slots — don't invent blend constants. When asked "based on what?", "judgment" is not an answer.
3. The backtest script IS the behavior spec. The serving implementation must reproduce the script's ranking on a fixture; divergence is a bug in the implementation, by definition. Keep the script in the repo with its data-export commands and baseline numbers in the docstring.
4. Numeric merge gates. Any PR touching ranking/weights/scoring attaches fresh backtest numbers. Set explicit thresholds (e.g. "HitRate@10 ≥ X and ≥ +Y points over baseline, or no merge"). No numbers → no merge.
5. Check your inputs before trusting your model. Open the actual data: in the real case, 99.8% of "product embedding" texts contained leaked LLM prompt instructions and 100% embedded sales statistics — making all new products mutually similar. Garbage in, garbage out is a checkable condition, not a proverb.
6. Metrics have blind spots — name them. Hit-rate rewards predicting what users would buy anyway; the incremental value of discovery/exploration isn't captured. State what the metric can't see and which decisions therefore remain business judgment.

Cheap calibration recipe (no ML infra required)

Split training data again (hold out the last event per entity). Bucket each signal (recency bins, log-scale lift bins...). Compute empirical P(outcome | bucket) with Laplace smoothing. Serve by table lookup; rank by probability. Zero learned parameters, fully explainable, refreshed nightly by counting.