semantic-duplication-scan

# Semantic Duplication Scan ## Overview Detect business logic that has been reimplemented multiple times across different architectural layers. Unlike linters (which detect syntactic similarity) or `domain-review` (which catches single-instance layer violations), this skill detects semantic equivalence — the same domain calculation independently appearing in domain services, client adapters, and presentation components with different variable names and structure. ## Annotation Prompt Version **promptVersion**: `1.0` When this version changes, any register entry with a different `promptVersion` is treated as stale and re-annotated on the next scan pass that touches that file. --- ## Pre-Filter Rules **Apply before any LLM call. No model invocation at this stage.** ### Trivial Function Definition A function is **trivial** — and must be excluded from the register — if it meets ALL of the following: - Contains **no arithmetic operators**: `+`, `-`, `*`, `/`, `%`, `**` - Contains **no boolean logic operators**: `&&`, `||`, `!`, `not`, `and`, `or` - Contains **no branching constructs**: `if`, `else`, `switch`, `case`, `ternary` (`?:`), `match` - Contains **no assignments to variables outside its own scope** (no external state mutation) - Contains **no calls to higher-order collection operations**: `map`, `filter`, `reduce`, `flatMap`, `forEach`, `find`, `some`, `every`, or language equivalents Trivial patterns (always excluded): - Getters: read and return a field with no transformation - Pass-through delegators: call one function with the same arguments, return the result unchanged - Identity functions: return the input unchanged - Constructors / initializers that only assign parameters to instance fields If a file contains **only trivial functions**, output: ``` No computation units found to analyze ``` and do not create or modify the register. ### File Exclusion Patterns Exclude the following from annotation regardless of content: ``` *.test.* *.spec.* __tests__/ *.test-d.* *.generated.* *.pb.* *.d.ts dist/ build/ .next/ coverage/ ``` Also exclude any path matching a pattern listed in `.semanticscanignore` (one glob per line) if that file exists in the project root. --- ## Process Flow ### Step 1 — Mode Detection Check for `computation-register.json` in the project root: - **Absent** → full-scan mode - **Present** → incremental mode ### Step 2 — Pre-Flight (Incremental Mode Only) Run: `git rev-parse --is-shallow-repository` If output is `true`: - Output the exact string: `Shallow clone detected — semantic-scan requires full history for incremental mode. Run with --full to override.` - Exit non-zero If `--full` flag was passed: skip this check and force full-scan mode. If `lastScanCommit` in the register is not found in git history: - Output: `lastScanCommit not found in history — running full scan` - Switch to full-scan mode ### Step 3 — Scope Resolution 1. If a path argument was provided (e.g., `/semantic-scan src/pricing`), use it as a prefix filter: only consider files whose paths start with the argument 2. Apply `.semanticscanignore` patterns: exclude any file matching a listed glob 3. Apply file exclusion patterns from the Pre-Filter Rules above ### Step 4 — File Selection **Full-scan mode**: Glob all source files in the resolved scope. **Incremental mode**: Run `git diff <lastScanCommit> HEAD --name-only`, then filter to files in the resolved scope. If the git diff result is empty (no files changed since `lastScanCommit`): 1. Update `lastScanCommit` to HEAD in the register and write the updated register 2. Output: `No changes since last scan — register up to date` 3. Exit 0 — do not proceed to annotation or clustering **Graph-assisted file/function discovery.** Prefer CodeGraph/Repowise over raw `Glob`/`Read` for locating candidate computation functions across layers — Repowise's semantic search (`search_codebase`) is particularly well suited here, since it's built for exactly the kind of similarity search this skill needs to find the same domain concept reimplemented in different layers. See [`knowledge/codegraph-vs-graphify.md`](../../knowledge/codegraph-vs-graphify.md) for tool selection and the fallback contract. ### Step 5 — Pre-Filter For each selected file, identify non-trivial computation functions using the Trivial Function Definition above. Apply without an LLM call — use structural heuristics (presence of operators, branches, higher-order calls). If no non-trivial functions remain after filtering: - **First run**: `No computation units found to analyze` → exit 0, no register created - **Incremental run**: `No new computation units found in changed files — register unchanged` → exit 0, register not modified ### Step 6 — Annotation (Haiku, file-level batching) For each file with non-trivial functions: 1. Emit progress to stderr: `Annotating [N/total] <filename>` 2. Send all non-trivial functions from the file in a single Haiku call using the pinned prompt below 3. If the call fails, record `{file, error}` in `scanErrors` and continue — do not abort **Annotation prompt (pinned — do not paraphrase):** ``` You are a semantic annotation assistant. For each function below, produce a JSON object describing what it computes in pure domain business terms. Use this schema: { "function": "<function name>", "layer": "<inferred layer — see rules below>", "semanticDescription": { "verb": "<lowercase infinitive verb>", "domainConcept": "<lowercase, no articles, normalized>", "inputs": ["<domain term>", ...], "outputConcept": "<domain term>" } } Layer inference rules — infer from what the function imports and uses: - "infrastructure": imports DB clients, ORMs, HTTP clients, message brokers (pg, redis, axios, fetch, prisma, mongoose, etc.) - "presentation": imports rendering primitives, formats for display, accesses DOM or templates (React, Vue, Svelte, JSX, HTML templates, etc.) - "domain": depends only on domain types and pure functions, no external imports - "application": orchestrates domain and infrastructure without owning business rules - "unknown": cannot be determined from available context domainConcept rules: - Use lowercase - Remove articles: a, an, the - Normalize the verb to infinitive form - Example: "calculates the discounted price" → domainConcept: "discounted price", verb: "calculate" Describe only what the function computes in domain terms. Do not reference the implementation language, variable names, or data structure types. Functions: <paste function source here> ``` **Canonicalize `domainConcept` after receiving the response:** 1. Lowercase 2. Strip leading/trailing articles: `a `, `an `, `the ` 3. Normalize verb in the `verb` field to infinitive (e.g., "calculates" → "calculate", "computing" → "compute") ### Step 7 — Register Update Build a register entry for each annotated function: ```json { "file": "<relative path from project root>", "function": "<function name>", "layer": "<inferred layer>", "semanticDescription": { "verb": "<canonicalized>", "domainConcept": "<canonicalized>", "inputs": ["<domain term>", ...], "outputConcept": "<domain term>" }, "promptVersion": "1.0", "commitHash": "<HEAD commit hash>", "line": <first line of function definition> } ``` **Merge strategy:** - Replace all entries whose `file` matches a re-annotated file - Remove entries whose `file` no longer exists on disk - Remove entries for files matching `.semanticscanignore` patterns - Preserve all other entries unchanged **Idempotency:** Sort the full entry list by `file` ascending, then `function` ascending before writing. **Write the register.** If the write fails (permissions, disk full): - Output the exact path that could not be written and the OS error - Exit non-zero **Update `lastScanCommit`** to the current HEAD commit hash after successful write. **Report partial failures.** After writing (only if `scanErrors` is non-empty): - N=1: `Warning: 1 file could not be annotated. Re-run /semantic-scan to retry.` - N>1: `Warning: N files could not be annotated. Re-run /semantic-scan to retry.` Exit code 0 — partial success is not a failure. --- ## Clustering ### Token Budget and Partitioning The full register may be too large to cluster in a single call. Use layer-pair sharding: Shard the register by layer pair before sending to Sonnet: - `domain × presentation` - `domain × infrastructure` - `application × presentation` - `application × infrastructure` - `domain × application` For each shard: if the shard exceeds **50,000 tokens**, further shard by the first normalized token of `domainConcept` (split alphabetically into sub-groups). After per-shard clustering, run a **cross-shard reconciliation pass**: send only the cluster representatives (one entry per cluster, the top-ranked candidate) back to Sonnet to catch equivalents that landed in different shards. Keep this merge input under 50,000 tokens. ### Clustering Call (Sonnet) For each shard, emit progress: `Clustering [layer-pair]: <layer-a> × <layer-b>...` **Clustering prompt (pinned — do not paraphrase):** ``` Group these register entries by semantic equivalence — entries that compute the same domain concept regardless of implementation differences. Return clusters as JSON arrays of entry IDs (use "file:function" as the ID). Two entries belong in the same cluster ONLY IF both would need to change if the underlying business rule changed. Input entries: <paste shard entries as JSON array> Return only: { "clusters": [["file1:funcA", "file2:funcB"], ...] } Entries that are semantically unique should not appear in any cluster. ``` --- ## Canonical Scoring For each cluster with 2 or more entries: **Layer rank** (higher = more canonical): 1. `domain` 2. `application` 3. `presentation` 4. `infrastructure` 5. `unknown` **Within the same layer rank:** count infrastructure imports in the function's file; fewer imports = higher rank. **Ambiguity predicate:** If the top two candidates tie on layer rank AND differ by ≤1 infrastructure import → escalate to Opus. Before the Opus call, emit: `Resolving ambiguous canonical for cluster: <domainConcept>...` **Opus prompt:** ``` Given these N entries that all compute the same domain concept, which one is the most appropriate canonical location for the business rule? Consider: domain purity, reusability, and least coupling to delivery mechanism. Return the file:function ID of the preferred canonical and a one-sentence rationale. Entries: <paste cluster entries> ``` If `--no-opus` flag was passed: skip Opus escalation. Report ambiguous clusters as: ``` canonical: ambiguous — human review required ``` ### Canonical Verdict Output - Clear winner (domain or application layer, no tie): `canonical: suggested <file:line> — requires human confirmation` - No winner (all infrastructure/unknown, or tie unresolved): `canonical: none — a new domain-layer implementation may be required` **Cross-scope notice** (scoped runs only): If a cluster contains entries outside the scoped path: ``` Note: this cluster includes <N> entry/entries outside the scoped path — run without scope argument to see full context ``` (Use "entry" when N=1, "entries" when N>1.) --- ## Report After clustering is complete: **Duplicates found:** ``` ## Semantic Duplication Report ### Cluster: <domainConcept> - <file>:<line> [<layer>] ← inferred canonical (or all entries if no canonical) - <file>:<line> [<layer>] canonical: suggested <file:line> — requires human confirmation [Note: this cluster includes N entry/entries outside the scoped path ...] --- ``` **No duplicates:** ``` No semantic duplication detected ``` **`file:line` accuracy:** Line numbers point to the first line of the function definition. If the file has been modified since annotation, append: `(line may have shifted — re-run scan to refresh)`