An observational finding fails in two directions. One is boring ("this 'surprise' is just the
model's top feature restated"). This gate catches the other: a finding that is genuinely
surprising but wrong — a pooled rate that is really a mix, a "lift" that is funnel position,
a "do X → they convert" that is reverse-causation. These survive a novelty check (they look
new) and a single sleepy read (the number is real; the interpretation is the bug), and ship.
Each step below defends against one specific, recurring way that happens.
-
Leak-free cohort from immutable business-event dates — not today's snapshot. Reconstruct
each unit's state as of the relevant date from history/event tables, not the current
snapshot; a snapshot value used at a historical anchor is leakage unless the attribute is
provably static. Read an outcome gate off its real business date with no snapshot fallback
(a COALESCE to a snapshot silently re-introduces the leak).
-
Before ANY event-anchored design, probe outcome_date − anchor_date. If you will measure
"activity in a window after milestone M → advanced later," first plot the gap distribution.
If a large share is ≤ 0, the anchor is structurally wrong — the outcome largely happens
before the anchor is even stamped, so a forward-ordered design keeps only an unrepresentative
tail and the lift inverts as an artifact. (A re-writable milestone date also leaks post-outcome
activity into a "pre-milestone" window — anchor on an immutable-upstream event and verify it
precedes the outcome.)
-
Decompose every pooled rate by the dominant structural axis before believing a gradient. A
pooled rate is a weighted mix; a "gradient" across some other variable is often that variable
silently proxying the mix (Simpson). Break the pooled number down by the dominant axis (level /
segment / program / stage) and read within it. Report the sub-population mix per segment
so the composition is visible.
-
De-confound the survivors with a regression — adjusted OR/coef [CI], multiple-testing-corrected.
Fit outcome ~ segment + controls (at minimum the structural axes from step 3) and report the
adjusted effect with its CI, BH-corrected. A raw gradient that goes to ≈1 net of controls is
composition; one that survives is a real association (still not necessarily causal — step 5).
Handle perfect/quasi-separation (drop/collapse tiny cells, note it) and aliased predictors.
-
Marker vs lever — a surviving association is still a SELECTION MARKER until an A/B says otherwise.
Intent is usually a common cause of both the behavior and the outcome, so even a robust,
de-confounded gradient is a targeting/selection marker, not a proven lever — "make them do X →
they advance" needs a holdout. Say so explicitly. Flag downstream-conditioned features
separately: if the "predictor" is the outcome act, its lift is the act in disguise. Disposition
is three-way: lever (survives an A/B) · marker (survives controls, associational) ·
artifact (composition/leak — retire it).
-
Validate any coverage-limited join is UNBIASED before trusting it. When an outcome/attribute
is only observable on a subset (identity bridge, side table, partial instrumentation), compare
the bridged-subset rate to an independent full-coverage reference; match ⇒ report with a
coverage caveat, diverge ⇒ the estimate is biased. Distinguish "value 0 = none" from "value 0 =
not observed."
-
Triple-probe every headline via independent constructions. Reproduce the number at least a
second way (different source table, join path, or grain) — and for the highest-stakes headline,
a third — and confirm they agree before it ships. Independent constructions landing on the same
value is the evidence; a single construction is a hypothesis. Report SQL, denominators, coverage,
and the as-of date per headline.
-
Write the honest ledger — retire / reframe / reproduces. For each prior figure in
circulation, state a verdict (retire = artifact/leak; reframe = right direction, wrong
denominator/anchor, give the corrected value; reproduces = re-derived clean) with the
replacement. State every rate's as-of date; keep protected attributes aggregate-only.
-
Ship the correction to every RENDERED surface — the ledger isn't shipped until the surfaces match.
Steps 1–8 make the finding true; this makes the shipped surfaces true. Editing the sentence that
quotes a figure does not touch the other places it is rendered: the served/baked copy of the
doc (re-bake + confirm the deploy actually re-baked, not a cached payload), embedded chart images
(a caption edit does not regenerate the PNG — grep the chart generators for the stale figure AND its
retired causal title, regenerate, and view the render; don't assume "there is no chart pipeline"),
twin / mirrored copies (a partially-swept twin self-contradicts), and dashboard payloads / cache
tables (a separate baker holds the value). Match each corrected value to the TARGET cell's
construct (in-window@Nd vs cumulative vs @120d — same numeral, different metric; don't cross them).
When step 4/5 retires a "lift" to a null, the CHART must lead with the adjusted estimate + a CI whisker
on the no-lift line and subordinate the raw gap (muted, "raw · self-selected"); kill causal titles.
Each step generalizes into a focused skill in this bundle; load the matching one when a step gets hard.
