audit-er-results

name	audit-er-results
description	Audit entity resolution match results for quality — analyze coverage, false positives, and false negatives
user_invocable	true

Audit Entity Resolution Results

After an ER match run, audit the results for match quality. Work from the repo root directory. Default paths assume results/<entity-type>/ for outputs. The --entity-type flag selects the config (candidacy_stage or elected_official).

Run each step below sequentially. Read each output CSV after running the script, then interpret the results before moving to the next step.

Step 0: Ensure Results Exist

If results already exist in the results directory, skip to Step 1. Otherwise, run the match pipeline first. The --run-audit flag runs Steps 1-3 automatically after matching, but you should still interpret the outputs.

# Candidacy (from local CSV):
uv run python -m scripts.cli match --entity-type candidacy_stage --input data/input.csv

# Candidacy (from Databricks — the typical path):
uv run python -m scripts.cli match --entity-type candidacy_stage \
  --input goodparty_data_catalog.dbt.int__er_prematch_candidacy_stages

# Elected officials:
uv run python -m scripts.cli match --entity-type elected_official \
  --input goodparty_data_catalog.dbt.int__er_prematch_elected_officials

Requires DATABRICKS_HTTP_PATH env var for Databricks reads (see scripts/databricks_io.py for auth details).

Step 1: Summary Statistics

uv run python -m scripts.cli audit summary --entity-type candidacy_stage --results-dir results/candidacy_stage/

Read results/audit_summary.csv and interpret the terminal output. Flag:

• Any provider with a match rate below 50% (may indicate missing blocking rules or post-prediction filters that are too aggressive for that source's data format)
• Any provider with a match rate above 95% (may indicate over-matching)
• Cluster sizes > 4 (likely false-positive chains)
• Within-source duplicate clusters (should not exist in link_only mode)
• Significant match rate differences between provider pairs (e.g. if source A matches well with source B but poorly with source C, the issue is likely source C's data format)

Source-Specific Analysis

When a new source has been added or an existing source shows anomalies, drill into that source specifically:

• Compare its match rate against each other source individually
• Check if unmatched records are concentrated in specific states or election dates
• Sample unmatched (singleton) records and manually search for plausible matches in other sources to gauge false negative severity

Step 2: Low-Confidence Match Review

uv run python -m scripts.cli audit low-confidence --entity-type candidacy_stage --results-dir results/candidacy_stage/ --sample 20

This finds the 20 pairs closest to 0.5 match probability — the model's most ambiguous decisions. Read results/audit_low_confidence.csv. For each pair:

1. Look at the side-by-side display columns (name, office, state, etc.)
2. Look at the gamma values to understand which comparisons agreed/disagreed
3. Assess whether this looks like a true match or false positive
4. Note the pattern — what's driving the ambiguity?

Summarize your findings as:

• Common patterns in true matches that scored low (model is underweighting something)
• Common patterns in false positives that scored mid-range (model is overweighting something)
• Specific comparison columns that are noisy or unhelpful

Step 3: False Negative Review

uv run python -m scripts.cli audit false-negatives --entity-type candidacy_stage --results-dir results/candidacy_stage/ --sample 20

Read results/audit_false_negatives.csv. For each suspicious non-match:

1. Check the was_in_pairwise_predictions column. Note that the pairwise predictions CSV contains only pairs that survived BOTH blocking AND the post-prediction filters in scripts/pipeline.py. A pair that shows was_in_pairwise_predictions = False may have been:

   • Never generated by blocking rules — no blocking rule matched the pair
   • Generated but removed by post-prediction filters — the pair was blocked and scored, but the person identity filter or race-level filter dropped it

   To distinguish these two cases, temporarily relax or remove the post-prediction filters and re-run, then check if the pair appears.

2. Per-provider singleton drill-down: Check the unmatched singleton count for each provider. If a provider has a high proportion of unmatched singletons (e.g. >30% unmatched), drill into that provider specifically. If the absolute number of unmatched records is small enough to review exhaustively (e.g. <500), do so rather than sampling — cross-reference each singleton against all other providers by state + election_date + last_name to find plausible matches that were missed. This catches false negatives that random sampling across all providers would miss, especially for smaller sources where a handful of missed matches significantly impacts the match rate.

3. Look for systematic patterns:

   • Are all misses from one specific provider? A new source with different data conventions (e.g. office name formatting) will often show up as a cluster of false negatives from that source.
   • Do they share a common data quality issue (missing field, different formatting)?
   • Is there a name variant pattern the model isn't handling?
   • Do the office names have low JW similarity despite referring to the same race? Cross-source office name formatting is the most common cause of false negatives (e.g. "mayor of brodhead" vs "brodhead city mayor", JW = 0.557).

Diagnosing Post-Prediction Filter False Negatives

The post-prediction filters in pipeline.py are a common source of false negatives, especially for new data sources. The filters include:

• Person identity: gamma_last_name > 0 AND (gamma_first_name > 0 OR gamma_email > 0 OR gamma_phone > 0)
• Race-level office check: gamma_official_office_name > 0 OR locality-token-overlap — requires either JW >= 0.75 OR a shared meaningful locality token between office names. If a new source uses office name conventions that share neither JW similarity nor locality tokens with existing sources, this filter will drop those pairs.
• Race ID consistency (candidacy only): excludes pairs where both have br_race_id but they differ, unless office names are similar (JW >= 0.88). Note: elected officials config does NOT have the race ID filter.

When investigating filter-caused false negatives:

1. Back up current results (cp -r results results_baseline)
2. Relax the suspect filter and re-run
3. Compare the new pairs: are they true positives or false positives?
4. If mixed, find a more targeted fix (e.g. adding stop words to the locality token list, adjusting JW thresholds)
5. Verify no regression in existing matches (see Step 4)

Step 4: Regression Check

When proposing or testing changes to blocking rules, comparisons, or post-prediction filters, always verify that existing match quality is preserved:

# Compare pairwise predictions
baseline_pairs = set(zip(baseline_pw['unique_id_l'], baseline_pw['unique_id_r']))
new_pairs = set(zip(new_pw['unique_id_l'], new_pw['unique_id_r']))
lost = baseline_pairs - new_pairs  # should be empty or near-empty

Check specifically:

• Zero lost pairs between the dominant source pair (e.g. BR×TS)
• Zero broken clusters — records that were clustered together should remain together
• New pairs are predominantly true positives — manually review a sample
• No new within-source duplicates introduced

Step 5: Recommendations

Based on all steps above, compile actionable recommendations. Organize them as:

Blocking rule changes (pairs never generated):

• New blocking rules needed
• Existing rules that are too restrictive

Comparison/threshold changes (pairs generated but misscored):

• Comparison columns to add, remove, or re-threshold
• Term frequency adjustments that might help

Post-prediction filter changes (pairs scored but filtered out):

• Office name filter: locality token stop-word list updates, JW threshold adjustments
• Person identity filter adjustments
• Race ID filter adjustments

Data quality issues (fix upstream in the prematch dbt model):

• Missing fields for specific providers
• Office name formatting inconsistencies across providers
• Schema mapping issues

Present these recommendations to the user and ask before making any changes. If the user wants to proceed with changes, edit the entity config in scripts/configs/candidacy.py or scripts/configs/elected_official.py for comparisons, blocking rules, or post-prediction filters. Edit the dbt prematch model for upstream data quality fixes.