| name | kb-dedupe |
| description | Find near-duplicate notes and merge them safely. Use when the user asks to "dedupe the KB", "find duplicate notes", "merge these notes", or after large research sessions that may have re-covered existing ground. |
| user_invocable | true |
| arguments | Optional: --kb <name>, --threshold X (default from config similarity.dedup_threshold), --limit N, or two slugs to merge directly |
/kb-dedupe
Find likely-duplicate note pairs and resolve them: merge, link, or dismiss. The engine handles mechanical rewiring; you make the editorial calls.
Direct merge: If $ARGUMENTS contains two slugs, skip discovery and go to step 4 with that pair.
Steps
-
Discover — Run python3 .kb/kb-index.py dedupe --json [--kb <name>] [--threshold X] [--limit N]. If empty, report and stop.
-
Classify each pair — Read BOTH notes fully before judging. High cosine similarity has three distinct causes; only one is a duplicate:
- True duplicate: same concept covered twice (usually from separate research sessions). → merge.
- Template siblings: same structure, different subject (two speaker profiles, two investor notes, two game profiles). The similarity is boilerplate, not content. → dismiss; optionally note if the shared boilerplate is bloating them.
- Intentional variants: drafts/versions kept on purpose (e.g. memo format variants), or a concept note + its deep-dive. → dismiss or link. If they're wired via
related: already (already_linked: true), lean toward intentional.
-
For true duplicates, pick the survivor — Prefer: the note with more inbound links; better sources (primary > secondary); higher epistemic status; better title/slug. When equal, keep the newer.
-
Merge — For each confirmed pair:
a. First move any unique content from the loser into the survivor (facts, sources, takeaways it lacks — respect the note-format sections). Update the survivor's updated: date.
b. Run python3 .kb/kb-index.py merge <loser> <survivor> — this backs up automatically (.kb/backup.sh), rewires every [[wikilink]]/related:/depends_on: reference across all KBs, unions sources, and marks the loser deprecated_by the survivor.
c. Use --dry-run first if the pair has many inbound links; sanity-check the rewire list.
d. Delete the loser stub only if the user asked for deletion; default is to keep the deprecated stub.
-
Verify — python3 .kb/kb-index.py build --incremental, then lint <survivor> for each merge. Confirm graph orphans didn't grow.
-
Log — Append a ## [YYYY-MM-DD] dedupe | <scope> entry to .kb/log.md: pairs reviewed, merges done (loser → survivor), pairs dismissed and why.
Report format
Table of pairs: score | verdict (merged / template siblings / intentional variants) | action taken. Lead with merges. Never merge without having read both notes — similarity scores cannot distinguish duplicates from templates.
