synonym-cluster

name	synonym-cluster
description	Identify tokens or phrases that refer to the same concept but appear in different forms — transcription variants from voice notes, spelling variants, acronyms vs expansions, aliases. Use on any voice-note or STT-derived corpus before frequency/NER/topic work, or to deduplicate entity lists.

Synonym Cluster

Collapse variant surface forms to a canonical concept.

When to use

• Voice notepad corpora: "GitHub" / "git hub" / "get hub" / "gitbub".
• Entity lists with aliases: "Daniel Rosehill" / "Daniel R." / "DR".
• Mixed acronym/expansion usage: "LLM" / "large language model".

Procedure

1. Candidate extraction: take unique tokens or NER spans from word-frequency / ner-extraction. Keep surface counts.
2. Pair-score candidates:
   • Lexical: RapidFuzz token_sort_ratio, Jaro-Winkler, or Levenshtein normalized by length. Fast, handles typos and transcription slips well.
   • Phonetic: Metaphone / Double Metaphone for STT variants ("get hub" ≈ "github" phonetically).
   • Embedding: sentence-transformer similarity for semantic aliases ("LLM" vs "large language model") — needed when lexical/phonetic fails.
3. Cluster: agglomerative clustering with a similarity threshold (lexical ≥0.85, embedding ≥0.88). Union-find is fine for sparse pair sets.
4. Pick canonical: the highest-frequency member of each cluster, or the longest well-formed member for acronym cases.
5. Review: surface clusters with borderline similarity (0.80-0.90) for manual confirmation — automated merging at that band causes false positives. For expert judgment on a specific cluster, a single LLM call is fine.
6. Output:

   [
     {"canonical": "GitHub", "variants": ["git hub", "get hub", "gitbub"], "total_count": 1423},
     {"canonical": "large language model", "variants": ["LLM", "LLMs"], "total_count": 892}
   ]
   

7. Apply: produce a replacement map and run over the corpus (re.sub with word boundaries), or keep as a lookup for downstream skills.

Cost control

No LLM needed for the clustering pass itself. Embeddings (if used) are a one-time cost on the unique-vocabulary set — typically 5-50k unique tokens, not N docs. Cheap.

Pitfalls

• Don't merge across semantic categories (person names ≠ place names) — keep NER labels during clustering.
• Acronyms collide ("MS" = Microsoft / Multiple Sclerosis / manuscript). Context required — escalate ambiguous clusters to an LLM.