entity-resolution

name

entity-resolution

description

Multilingual entity resolution and record matching across cultures and languages. Use when deduplicating records, matching customer/company names across datasets, or resolving entities with cross-cultural name variants. Handles CJK names (Chinese/Japanese/Korean), Arabic transliterations, Hispanic dual surnames, patronymic naming (bin/binti/ibn), Korean romanization variants, and legal entity suffix normalization. Triggers: record matching, deduplication, entity resolution, fuzzy name matching, cross-language matching, merge duplicate records, KYC name matching, or any task involving "are these the same person/company?"

Quick Start

Follow the 3-step recipe: Normalize, Score, Decide.

from normalize import normalize
from similarity import composite_score

name_a = normalize("José García López", entity_type="person")
name_b = normalize("Jose Garcia Lopez", entity_type="person")
score = composite_score(name_a, name_b)
# score = 1.0 -> MATCH

Run the CLI for batch resolution:

python scripts/resolve.py --input-a records_a.csv --input-b records_b.csv --output results.json

Interpret the 3-tier output:

Tier	Condition	Action
MATCH	score >= T_HIGH	Auto-merge the records
REVIEW	T_LOW < score < T_HIGH	Flag for human review
NO_MATCH	score <= T_LOW	Auto-reject the pair

Default thresholds: T_HIGH=0.85, T_LOW=0.60.

Decision Tree

Route to the appropriate reference based on data characteristics:

• CJK names (Chinese/Japanese/Korean characters detected) Read references/cultural-patterns.md#cjk-names
• Arabic names (Mohammed variants, ibn/bin patterns) Read references/cultural-patterns.md#arabic-names
• Hispanic names (dual surnames, compound family names) Read references/cultural-patterns.md#hispanic-names
• Patronymic names (bin/binti, -son/-dottir patterns) Read references/cultural-patterns.md#patronymic-systems
• Korean names (romanization of 이/김/박 etc.) Read references/cultural-patterns.md#korean-romanization
• Company/legal entity matching (Ltd, GmbH, Inc suffixes) Read references/cultural-patterns.md#legal-entity-suffixes
• Algorithm tuning needed (adjusting weights or metrics) Read references/algorithms.md
• Large datasets (>1000 records, performance concerns) Read references/blocking-strategies.md

Pipeline Architecture

Input Records -> Normalize -> Block -> Score -> Classify -> Output

Normalize — Apply Unicode NFKD decomposition, case folding, whitespace normalization, and legal suffix stripping (for companies). Convert all text to a canonical form suitable for comparison.

Block — Partition records into candidate groups by blocking key (e.g., first N characters, phonetic code) to avoid O(n^2) pairwise comparisons. Essential for datasets with more than a few hundred records.

Score — Compute a composite similarity score from multiple metrics: Jaro-Winkler, Levenshtein ratio, token overlap, and phonetic matching. Each metric captures a different dimension of name similarity.

Classify — Apply threshold-based classification to the composite score:

• score >= T_HIGH (0.85) -> MATCH (auto-merge)
• score <= T_LOW (0.60) -> NO_MATCH (auto-reject)
• otherwise -> REVIEW (flag for human review)

Output — Emit results as JSON with pair IDs, scores, classifications, and per-metric breakdowns for auditability.

Running the Scripts

scripts/normalize.py

Import and call normalization functions directly:

from normalize import normalize, unicode_normalize, case_fold
from normalize import normalize_whitespace, normalize_legal_suffix
from normalize import transliterate

# Full normalization pipeline
clean = normalize("José García López", entity_type="person")

# Individual steps
text = unicode_normalize("Müller")       # NFKD decomposition
text = case_fold(text)                    # lowercase folding
text = normalize_whitespace(text)         # collapse whitespace
text = normalize_legal_suffix(text)       # strip Ltd/GmbH/Inc
text = transliterate(text)               # romanize non-Latin scripts

scripts/similarity.py

Import and call scoring functions:

from similarity import jaro_winkler, levenshtein_ratio
from similarity import token_similarity, soundex, phonetic_match
from similarity import composite_score

# Individual metrics
jw = jaro_winkler("smith", "smyth")       # 0.0-1.0
lr = levenshtein_ratio("smith", "smyth")  # 0.0-1.0
ts = token_similarity("John Smith", "Smith John")  # token overlap
sx = soundex("Smith")                     # phonetic code
pm = phonetic_match("Smith", "Smythe")    # boolean

# Combined scoring (recommended)
score = composite_score("John Smith", "Jon Smyth")

scripts/resolve.py

CLI usage:

python scripts/resolve.py \
  --input-a A.csv \
  --input-b B.csv \
  --t-high 0.85 \
  --t-low 0.60 \
  --output results.json

Argument	Description	Default
--input-a	Path to first dataset (CSV)	required
--input-b	Path to second dataset (CSV)	required
--output	Path for output results (JSON)	stdout
--t-high	Upper threshold for MATCH classification	0.85
--t-low	Lower threshold for NO_MATCH	0.60
--entity-type	"person" or "company"	"person"
--blocking	Blocking strategy name	"default"

Library usage:

from resolve import resolve

results = resolve("a.csv", "b.csv", t_high=0.85, t_low=0.60)

Threshold Guide

Default thresholds: T_HIGH=0.85, T_LOW=0.60.

Select a preset based on risk tolerance:

Preset	T_HIGH	T_LOW	Use Case
Conservative (KYC/AML)	0.90	0.70	Regulatory, high-cost errors
Balanced (data migration)	0.80	0.55	General deduplication
Aggressive (marketing dedup)	0.75	0.50	Broad matching, low-cost errors

For detailed threshold tuning, read references/algorithms.md#threshold-tuning.

When to Load References

Scenario	Load Reference
CJK, Arabic, Hispanic, or Korean names	references/cultural-patterns.md
Company or legal entity matching	references/cultural-patterns.md#legal-entity-suffixes
Tuning algorithm weights or metrics	references/algorithms.md
Large-scale matching (>1K records)	references/blocking-strategies.md