dedup

name	dedup
description	Find duplicate functions and eliminate them so they stay gone. Triage tl dupes output by tier, apply the right consolidation pattern, and lock the win with a duplication-ratchet test. Use when asked to dedupe, remove duplication, consolidate copy-paste, clean up repeated helpers, or act on tl dupes output.
compatibility	Requires tokenlean CLI tools (npm i -g tokenlean) and git

Dedup

tl dupes finds duplicates; removing them well is the skill. Most of the output is noise, the wins hide in two tiers, and the mitigation that actually sticks is non-obvious: you remove the second home, not "fix both copies."

Workflow

Scan → Triage → Mitigate → Verify → Ratchet

1. Scan

tl dupes --near        # structural + near surfaced first; tests excluded by default

Via the MCP tool, near is a number (the similarity threshold), not a flag: mcp__callmux__tl_dupes { near: 0.85 }. Passing near: true fails schema validation — the CLI's bare --near defaults that threshold for you.

Files copied verbatim into another tool (e.g. a plugin shipped into a user's config dir) are external contracts — their duplicate helpers are by-design, not cleanup targets. tl dupes excludes them by default. Mark your own in .tokenleanrc.json:

{ "externalContractFiles": ["src/my-copied-plugin.js"] }

Use tl dupes --near --include-contracts only when you deliberately want to audit those copies too.

2. Triage by tier — signal vs. noise

tl dupes returns four tiers. Treat them differently:

• Structural + near = signal. Renamed clones and copy-paste-with-edits — the drift-prone ones. Start here.
• Exact = quick wins, but check intent first. Two trivial pass-throughs colliding (return [...this.x.values()]) aren't always worth a shared home.
• Names = mostly noise. Idiomatic repeats (update, close, get, logger levels). Skim, don't mine — add --no-names once you've glanced.

3. The "real dupe?" test

The rule that separates worth-fixing from leave-it:

• Same rule encoded twice → collapse it. Validation, response unwrapping, a path-containment check, a magic constant, a normalization step. These are the drift hazards: one copy gets a fix, the other doesn't, and a phantom bug surfaces weeks later.
• Same shape by coincidence → leave it. Logger debug/info/warn/error, SSE emit* helpers, store thunks, thin HTTP wrappers. Merging hurts readability for zero safety gain.

Propose a verdict per group; let the operator confirm. Dedup needs the shape-vs-rule judgment — never auto-merge.

4. Mitigate: delete the home, don't fix the copies

The principle agents get wrong: don't "find and fix both copies" — you'll miss the third one next time. Remove the second home so there's nothing left to drift.

What's duplicated?
  ├─ Two classes, same except one method
  │   → abstract base class with one abstract seam (the part that differs)
  ├─ Two components with the same stateful logic
  │   → extract a hook / shared function
  ├─ A method defined but never called
  │   → delete it — verify dead first:  tl impact <file>  +  repo-wide call grep
  └─ A constant/type repeated
      → one exported definition, import everywhere

Diff the copies before merging — the diff is often a latent bug. When two copies merge, their differences surface and force a decision. (Real case: two response-unwrap helpers disagreed on whether an {error} field meant failure; unifying them forced — and fixed — that decision.) Before collapsing:

tl snippet <fnA> <fileA>   # then diff against
tl snippet <fnB> <fileB>

5. Verify

tl parallel "test=tl run '<test-command>'" "guard=tl guard" "dupes=tl dupes --near"

The re-run of tl dupes confirms the group is gone (and didn't spawn a new one).

6. Ratchet — lock the win

This is the half that makes dedup stick. A duplication ratchet is a test that counts the definition sites of a primitive that should have exactly one home, and fails CI if the count exceeds a baseline — a ceiling that can only move down:

• New copy added → count exceeds baseline → hard fail, message points at the canonical home.
• You consolidate → count drops → test passes but prints "lower the baseline", locking the win.
• Eventually baseline === 1 and the primitive is permanently single-homed.

The scan finds today's dupes; the ratchet stops tomorrow's. It turns "we thought we fixed this already" from a recurring debug session into a test failure at PR time.

If a ratchet already exists: after consolidating, lower its baselines for anything you merged.

If absent, offer to scaffold one. Keep it tiny and language-agnostic — match definition sites, never call sites. Node example:

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';

// Down-only ceiling. Lower each number as you consolidate; never raise it.
const BASELINES = {
  'unwrapResponse': 1,   // canonical: src/transport/unwrap.js
  'assertInRoot':   1,   // canonical: src/fs/path-guard.js
};

for (const [name, max] of Object.entries(BASELINES)) {
  test(`duplication ratchet: ${name} ≤ ${max} definition(s)`, () => {
    // Definition sites only — `function name`, `const name =`, `name(` as a method.
    const out = execFileSync('rg', [
      '-c', '--no-filename',
      `(function\\s+${name}\\b|\\b${name}\\s*[=(:])`,
      'src',
    ], { encoding: 'utf8' });
    const count = out.split('\n').filter(Boolean)
      .reduce((n, line) => n + Number(line), 0);
    assert.ok(count <= max,
      `${name} defined ${count}× (baseline ${max}). Reuse the canonical home or lower the baseline if you intentionally consolidated.`);
  });
}

Tune the regex to the language; the philosophy is constant: a ceiling that only ratchets down.

Tips

• Sequence it: structural/near first (real wins), exact second (quick), names last (skim). Don't burn the operator's attention on the names tier.
• Make incremental commits per consolidated group — if a merge breaks tests you can bisect, and each ratchet-baseline drop is its own clean diff.
• After collapsing, tl unused catches the now-dead exports you forgot to remove.
• Per-project only. Don't try to dedup across repos.