coverage-limited-join-validate-unbiased-before-trusting

name	coverage-limited-join-validate-unbiased-before-trusting
description	When you can only observe an outcome/attribute on a SUBSET of the population because the join/linkage/bridge is coverage-limited — imperfect identity resolution, a side child-table that only covers some rows, partial instrumentation, survey non-response — validate the restricted estimate is UNBIASED before reporting it. Use when: (1) a join covers only X% of rows and you're about to either treat unmatched rows as null/negative OR report the matched-subset rate as the population rate; (2) the primary record has no foreign key to the entity you need and you found a peripheral table that carries both keys as a bridge; (3) you're weighing "rebuild the full linkage" vs "use the partial one." The technique: find an INDEPENDENT full-coverage reference for the same quantity and compare the bridged-subset rate to it — match ⇒ missingness is ~ignorable (MAR), report with a coverage caveat; diverge ⇒ the gap is informative (MNAR), the restricted estimate is biased and must be recovered or bounded. Prevents silently shipping a biased rate off a partial join.
author	Claude Code
version	1.0.0
date	"2026-07-07T00:00:00.000Z"

Validate a coverage-limited join is unbiased before trusting its estimate

Problem

You need an outcome or attribute that lives on a related entity, but the link is coverage-limited — identity resolution is fuzzy, the primary record lacks a foreign key, a bridge table only covers some rows, or instrumentation/response is partial. You can only observe the quantity on the matched subset. Two instinctive moves are both wrong:

• Impute unmatched rows as null/negative/absent → undercounts the numerator (the unmatched rows had outcomes too; you just can't see them).
• Report the matched-subset rate as the population rate → silently biased if coverage is non-random (e.g. matched rows skew toward further-along / more-engaged / higher-income entities).

The coverage % alone doesn't tell you which. "76% coverage" could be perfectly unbiased or badly skewed — you have to check.

Context / Trigger conditions

• A join/LEFT JOIN/bridge that resolves only part of the population and you're about to compute a rate over the matched subset.
• The primary record has no FK to the entity you need (so you hunted for a peripheral child table that carries both keys — e.g. a supporting-documents / events / junction table).
• You're deciding whether to invest in rebuilding full linkage (expensive, error-prone) vs. accept the partial bridge.
• A cross-cohort / cross-cycle / cross-segment estimate where one slice has full coverage and another only has the partial bridge.

Solution

1. Find the peripheral bridge (if the FK is missing). When the primary record carries no FK to the target entity, scan sibling/child tables for one holding both keys (grep INFORMATION_SCHEMA for the two id columns co-occurring). A junction/child table is often a usable bridge even when the parent isn't linked — but it inherits that table's coverage (only rows that produced a child record).

2. Get an INDEPENDENT full-coverage reference for the same quantity. The check needs a benchmark you trust at ~100%. Sources, in order of preference:

• A different slice/cohort where full linkage does exist (e.g. a recent cycle covered by a complete identity graph while the older cycle only has the bridge).
• An aggregate control total from an authoritative source (a reported enrollment count, a billing total, a source-system COUNT).
• The same population via a second, independent linkage path.

3. Compare the bridged-subset rate to the full-coverage reference.

• Match (within noise) → the missingness is ~ignorable for this outcome (MAR). Report the restricted estimate, state the coverage explicitly ("N=X of Y, ~C% coverage, verified unbiased against "). Do NOT rebuild linkage you don't need.
• Diverge → coverage is informative (MNAR); the restricted rate is biased in the direction of who-gets-matched. Either recover the missing linkage, or report a bound (e.g. assume the unmatched are all-0 / all-1 for a worst/best case) rather than the point estimate.

4. Never let the coverage gap masquerade as a real signal. If you skipped the check and imputed unmatched-as-negative, a coverage difference between segments will look like an outcome difference. Segment A at 90% linkage and segment B at 50% linkage will show a fake gap purely from coverage.

Verification

The MAR check is the verification. Additionally: report the covered-N alongside the rate (not just the %); confirm the reference is genuinely independent of the bridge (not the same join relabeled); if no independent reference exists, mark the estimate coverage-limited / unverified rather than asserting it.

Example (a two-cycle funnel with a partial identity bridge)

The primary application record has no foreign key to the account that carries the outcome label, so the outcome can't be joined directly. A complete identity graph exists only for the current cycle. For the prior cycle the only bridge is a peripheral child table (supporting_documents) carrying both application_id and account_id — but it covers only part of the outcome records (say roughly three-quarters).

Rather than treat the uncovered remainder as "did not convert" (undercount) or trust the covered subset blindly, benchmark on the current cycle, where BOTH the bridge and the full graph exist: if the bridged-subset outcome rate matches the full-coverage rate within about a percentage point, the bridge is ~unbiased for this outcome, so the prior-cycle restricted rate is trustworthy with a stated coverage caveat — no need to rebuild the (expensive, fuzzy) identity graph. Had they diverged, the prior-cycle number would be reported as bounded/unverified instead.

Notes

• This is the descriptive-analysis sibling of feature-join-path coverage decisions — but the question is different: not "which path / take the union" but "is the only available partial link unbiased for my estimate."
• Works for any partial-observation setting: record linkage, tracking-pixel coverage, survey response, instrumented-vs-uninstrumented users, matched-cohort studies.
• The reference must be independent. Benchmarking a bridge against itself (or against a subset of the same join) proves nothing.

References / See also

• feature-join-path-choice-is-coverage-not-leakage-measure-before-building — sibling: choosing among multiple join paths for a model feature is a coverage (not leakage) decision; take the union.
• bq-identity-resolution-debug — mechanics of debugging identity resolution in BigQuery.
• snapshot-transient-state-cohort-censors-outcome-use-ever-entered-from-history — related snapshot-vs-history censoring trap.
• Missing-data taxonomy: MCAR / MAR / MNAR (Rubin 1976) — the check above distinguishes MAR-enough from MNAR empirically.