| name | hunting-leakage-and-validating |
| description | Finds label leakage and designs honest validation — leakage taxonomy, grouped and time-aware splits, base rates, and the attack-your-own-model checklist. Use before trusting any metric, when a score looks too good, or when features and labels share time windows or origins. |
Hunting leakage and validating
The sneakiest failure in applied ML: the model quietly reads the answer during training. Scores
look amazing; the work is worthless. Your job is to attack your own model before anyone else can.
The leakage taxonomy — three ways answers sneak in
1. Label-derived features. The label was computed FROM a column, and that column (or its
sibling) is in the features. Symptom: one feature towers over all others, and the score is
near-perfect. Test: train once WITH the suspect, once WITHOUT — a collapse from ~1.0 to ~0.7
is the confession.
2. Future/overlapping windows. A feature summed over a window that CONTAINS the label's
window already knows the outcome. Draw the timeline: every feature must be knowable at the
moment of prediction, labels strictly after. If a 90-day aggregate overlaps your last-30-day
label — only the earlier sub-window is a legal feature.
3. Decision-derived features (product flags). Scores and flags produced by an existing
system encode a decision someone already made. Using them as features means learning the old
rule, not the world — a circular result. They may serve as a BASELINE to beat, never as inputs.
Honest splits — random is rarely honest
- Grouped split (GroupKFold by the entity that repeats — client, site, user): rows from one
group share hidden character; a random split lets the model memorize the group and fake skill.
The honest question is "does it work on a group it never saw?"
- Time split (train on the past, test on the future): the only split that mimics deployment
for anything trend-like.
- Report the random-split number next to the honest-split number if you like — the GAP between
them is itself a finding about how much memorization was happening.
Always print the base rate
Accuracy of 71% on a label that is 62% positive is 9 points of skill, not 71. Every score sits
next to its naive baseline (majority class / random ranking), always.
Sealed holdouts leave receipts
If you claim a sealed or holdout evaluation ("touched once, blind"), commit the receipts: the
cell or script that builds the sealed frame, AND the metrics file it produced. A sealed claim
that can only be verified by digging through git history isn't sealed — it's trusted. And
check your population definition for future information: if the rows you keep depend on
anything from the outcome window (e.g. "clients still active in the label month"), say so in
your limitations — it's a choice, not a crime, but hiding it is.
The attack checklist (run it before you believe anything)
How to verify
- Deliberately ADD a leaky feature and watch the score jump toward 1.0 — if it doesn't, your
test harness itself is broken. Then remove it and keep the honest number.
- Swap your random split for a grouped split and report both numbers. If you can't explain the
gap, you're not done.
