dedup

name	dedup
description	Find duplicate functions in Codex and remove them so they stay gone — triage tl dupes by tier, consolidate the second home (not both copies), and add a duplication-ratchet test to block regressions. Use when deduping, removing copy-paste, consolidating repeated helpers, or acting on tl dupes output.
compatibility	Codex CLI with terminal access, git, tokenlean CLI (npm i -g tokenlean)

Dedup (Codex)

tl dupes finds duplicates; removing them well is the skill. Most tiers are noise, the wins hide in two of them, and the mitigation that sticks is non-obvious: remove the second home, don't "fix both copies."

Workflow

Scan -> Triage -> Mitigate -> Verify -> Ratchet

1. Scan

tl dupes --near        # structural + near first; tests excluded by default

Via the MCP tool, near is a number (similarity threshold), not a flag: mcp__callmux__tl_dupes { near: 0.85 }. near: true fails schema validation — the CLI's bare --near defaults that threshold for you.

Files copied verbatim into another tool (a plugin shipped into a user's config dir) are external contracts — their duplicate helpers are by-design. tl dupes excludes them by default. Mark your own in .tokenleanrc.json:

{ "externalContractFiles": ["src/my-copied-plugin.js"] }

tl dupes --near --include-contracts audits those copies too, when you mean to.

2. Triage by tier

Four tiers, treated differently:

• Structural + near = signal. Renamed clones, copy-paste-with-edits — the drift-prone ones. Start here.
• Exact = quick wins, but check intent. Trivial pass-throughs that collide aren't always worth a shared home.
• Names = mostly noise. Idiomatic repeats (update, close, get). Skim; add --no-names after a glance.

3. The "real dupe?" test

• Same rule encoded twice → collapse it. Validation, response unwrapping, path-containment checks, magic constants, normalization. Drift hazards: one copy gets fixed, the other doesn't, phantom bug later.
• Same shape by coincidence → leave it. Logger levels, SSE emit* helpers, store thunks, thin HTTP wrappers. Merging costs readability for no safety gain.

Propose a verdict per group; the operator confirms. Never auto-merge — the shape-vs-rule call needs judgment.

4. Mitigate: delete the home

Don't "find and fix both copies" — you'll miss the third next time. Remove the second home.

What's duplicated?
  ├─ Two classes, same except one method
  │   → abstract base class with one abstract seam (the differing part)
  ├─ Two components with the same stateful logic
  │   → extract a hook / shared function
  ├─ A method defined but never called
  │   → delete it — verify dead:  tl impact <file>  +  repo-wide call grep
  └─ A constant/type repeated
      → one exported definition, import everywhere

Diff the copies before merging — the diff is often a latent bug. Merging surfaces their differences and forces a decision (e.g. two unwrap helpers disagreeing on whether {error} means failure — unifying fixed it).

tl snippet <fnA> <fileA>
tl snippet <fnB> <fileB>     # diff the two before collapsing

5. Verify

tl parallel \
  "test=tl run '<test-command>'" \
  "guard=tl guard" \
  "dupes=tl dupes --near"

The tl dupes re-run confirms the group is gone and no new one appeared.

6. Ratchet — lock the win

A duplication ratchet is a test that counts the definition sites of a primitive that should have one home and fails CI if the count exceeds a baseline — a ceiling that only moves down:

• New copy → exceeds baseline → hard fail, message names the canonical home.
• You consolidate → count drops → passes, prints "lower the baseline."
• Eventually baseline === 1: permanently single-homed.

The scan finds today's dupes; the ratchet stops tomorrow's. If a ratchet already exists, lower its baselines after consolidating. If absent, offer to scaffold one — tiny, language-agnostic, matching definition sites, never call sites:

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';

// Down-only ceiling. Lower as you consolidate; never raise.
const BASELINES = { unwrapResponse: 1, assertInRoot: 1 };

for (const [name, max] of Object.entries(BASELINES)) {
  test(`duplication ratchet: ${name} <= ${max}`, () => {
    const out = execFileSync('rg', [
      '-c', '--no-filename',
      `(function\\s+${name}\\b|\\b${name}\\s*[=(:])`, 'src',
    ], { encoding: 'utf8' });
    const count = out.split('\n').filter(Boolean)
      .reduce((n, l) => n + Number(l), 0);
    assert.ok(count <= max,
      `${name} defined ${count}x (baseline ${max}). Reuse the canonical home or lower the baseline.`);
  });
}

Tune the regex per language; the philosophy is constant: a ceiling that only ratchets down.

Tips

• Sequence: structural/near (real wins) → exact (quick) → names (skim). Don't spend attention on the names tier.
• Commit per consolidated group — bisectable, and each baseline drop is a clean diff.
• tl unused after collapsing catches now-dead exports.
• Per-project only; don't dedup across repos.