find-duplicates

name	find-duplicates
source	botcore
description	Detects code duplication using hybrid analysis combining algorithmic (JSCPD) and semantic (AI) techniques. Covers Type-1 through Type-4 clone detection, safe refactoring workflows, and duplication prevention strategies. Use when scanning for copy-paste code, reducing code bloat, consolidating duplicate utilities, or improving DRY compliance. Triggers: duplicates, duplication, copy paste, code clone, DRY, refactor.
version	1.0.0
triggers	["duplicates","duplication","copy paste","code clone","DRY","refactor","shared utility","jscpd","code smell"]
portable	true

Finding Duplicates

Detect and eliminate code duplication using hybrid analysis -- algorithmic detection for syntactic clones, AI analysis for semantic clones.

Why This Exists

AI agents create flat, duplicative codebases because they do not know existing utilities unless they are in context, they implement first rather than searching for existing code, and they lack refactoring instinct when patterns emerge. The result is code bloat, inconsistency, and harder maintenance. This skill provides the detection and remediation workflow.

Capabilities

1. Algorithmic detection -- Run JSCPD to find Type-1/2/3 clones (exact copies, renamed variables, minor structural changes)
2. Semantic detection -- Use AI analysis to find Type-4 clones (same behavior, different implementation)
3. Prevention -- Search for existing utilities before writing new code to stop duplication at the source
4. Prioritization -- Triage duplicate findings by occurrence count and block size into actionable categories
5. Safe refactoring -- Guide Red-Green-Refactor consolidation of duplicates with test coverage

Routing Logic

Request type	Load reference
Refactoring workflow, output format, cross-repo detection	references/refactoring-guide.md

Core Principles

1. Detect Before You Generate

The best duplication is the one that never happens. Before writing any new utility function, search the codebase for existing implementations. Use project-level search, grep, or vector-based discovery when available.

2. Three-Tier Detection

Tier	Clone Types	Method	Speed
Syntactic	Type-1, 2, 3	JSCPD (token-based)	Fast (~3s)
Semantic	Type-4	AI / vector similarity	Slower (~30s)
Preventive	All	Search before generating	Real-time

Always start with Tier 1 (fast, cheap). Escalate to Tier 2 when algorithmic detection alone is insufficient. Apply Tier 3 as a habit for every new utility.

3. Clone Type Awareness

Type	Description	Example
Type-1	Exact copy	Identical code in two files
Type-2	Renamed identifiers	Same logic, different variable names
Type-3	Structural variation	Added/removed statements, reordered blocks
Type-4	Semantic equivalent	Different implementation, same behavior

JSCPD catches Types 1-3. Only AI analysis reliably catches Type-4.

4. Prioritize by Value, Not Volume

Not all duplication warrants refactoring. Use this triage:

Priority	Criteria	Action
High	3+ occurrences, >20 lines each	Extract to shared utility immediately
Medium	2 occurrences, >10 lines each	Extract if touched again
Low	Small blocks, edge cases	Document and defer

5. Safe Refactoring Only

Never consolidate duplicates without test coverage. Follow the Red-Green-Refactor pattern: write tests covering all duplicate implementations first, verify they pass, create the unified utility, update call sites, then confirm tests still pass.

Quick Reference

JSCPD Commands

# Scan current directory
lush discover duplicates

# Scan specific path
lush discover duplicates --path src/

# With custom threshold
lush discover duplicates --threshold 3

JSCPD Configuration (.jscpd.json)

{
  "threshold": 5,
  "reporters": ["json", "console"],
  "ignore": [
    "**/node_modules/**",
    "**/.git/**",
    "**/dist/**",
    "**/__pycache__/**"
  ],
  "mode": "mild"
}

Modes: strict (exact only), mild (ignores whitespace/formatting), weak (ignores comments and variable names -- finds Type-2).

AI-Based Detection

# Run semantic analysis via subagent
lush-subagent run role='duplicates'

Workflow

Detection Phase

1. Run algorithmic scan -- Execute lush discover duplicates --path <target> for fast Type-1/2/3 results
2. Review findings -- Examine reported clone pairs with file locations and similarity percentages
3. Run semantic scan (optional) -- Execute lush-subagent run role='duplicates' for Type-4 clones
4. Triage results -- Categorize findings into High / Medium / Low priority using the table above

Remediation Phase

5. Write tests first -- Create tests covering every duplicate implementation before changing anything
6. Verify tests pass -- Run the test suite to confirm tests pass against current (duplicated) code
7. Create unified utility -- Extract the shared logic into a single well-named function or module
8. Update call sites -- Replace all duplicate usages with imports from the new shared utility
9. Verify tests still pass -- Run the same tests against the unified implementation
10. Document the extraction -- Note what was consolidated and where the shared utility lives

Checklist

• Algorithmic scan completed (JSCPD or equivalent)
• High-priority duplicates identified (3+ occurrences, >20 lines)
• Tests written covering all duplicate implementations before refactoring
• Unified utility created with clear naming and documentation
• All call sites updated to use the shared utility
• Test suite passes after consolidation
• No new duplication introduced during refactoring

When to Escalate

• Cross-repo duplication requiring multi-repository analysis or shared packages
• Type-4 semantic clones where AI analysis produces uncertain or conflicting results
• Duplication embedded in generated code (protobuf, OpenAPI codegen) that cannot be refactored
• Consolidation would require breaking public API contracts or changing package boundaries
• Duplicate patterns are intentional (e.g., platform-specific implementations that look similar but must diverge)