// After writing code, detect and clean up duplication at three levels — copy-paste blocks, cross-package shared code, unnecessary wrappers, and concept-level SoT violations. Detect with indexion, fix, and verify.
| name | indexion-refactor |
| description | After writing code, detect and clean up duplication at three levels — copy-paste blocks, cross-package shared code, unnecessary wrappers, and concept-level SoT violations. Detect with indexion, fix, and verify. |
Detect and eliminate duplication at three levels — textual, structural, and conceptual — using indexion's analysis commands, then verify SoT is enforced.
opendir, ENOENT, or similar filesystem errors appear from unexpected paths| Level | What it is | Tool | Example |
|---|---|---|---|
| Textual | Copy-pasted code blocks, identical functions | plan refactor | is_whitespace copied across 5 modules |
| Structural | Same logic structure with different names | plan solid, plan unwrap | cross-package extraction candidates, trivial wrappers |
| Conceptual | Same domain concept implemented independently | explore + manual analysis | Three modules each deciding "is this file an archive?" |
Textual duplication is easy to find and fix. Conceptual duplication is the hardest and most dangerous — it produces no copy-paste matches but means changing one concept requires updating every scattered implementation.
plan refactor)Start with high-confidence matches and work down.
# Step 1: Find 90%+ duplicates (high confidence)
indexion plan refactor --threshold=0.9 \
--include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' \
cmd/indexion/
indexion plan refactor --threshold=0.9 \
--include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' \
src/
Read the output in three sections:
| Section | What it finds | Action |
|---|---|---|
| Similar Files | Files with high overall similarity | Investigate for structural consolidation |
| Duplicate Code Blocks | Line-level identical code between files | Extract to @common or shared module |
| Function-Level Duplicates | Structurally similar functions (TF-IDF on bodies) | Unify into single SoT function |
Same-file duplicates (functions within one file at 90%+) are the highest-value
targets — easiest to fix, clearest wins. Example: get_global_data_dir and
get_global_cache_dir share 95% structure, extracted into resolve_os_dir.
# Step 2: Use grep to trace references before consolidating
indexion grep "TypeIdent:TfidfEmbeddingProvider" src/
indexion grep --semantic=name:is_whitespace src/
# Step 3: Fix, then re-run to confirm duplicates are gone
indexion plan refactor --threshold=0.9 --include='*.mbt' ...
# Step 4: Lower threshold and iterate
indexion plan refactor --threshold=0.85 --include='*.mbt' ...
plan refactor options:
| Option | Default | Description |
|---|---|---|
--threshold=FLOAT | 0.7 | Minimum similarity threshold |
--strategy=NAME | hybrid | Similarity: hybrid, tfidf, bm25, jsd, ncd |
--fdr=FLOAT | 0 | FDR correction (0=disabled) |
--style=STYLE | raw | Output: raw, structured |
--format=FORMAT | md | Output: md, json, text, github-issue |
--name=NAME | -- | Project name (for structured style) |
--include=PATTERN | -- | Include pattern (repeatable) |
--exclude=PATTERN | -- | Exclude pattern (repeatable) |
-o, --output=FILE | stdout | Output file path |
--specs-dir=DIR | kgfs | KGF specs directory |
What remains after cleanup (stop signals):
native.mbt / stub.mbt) — intentional platform branchingto_string on different types) — different types, same patterncommand() functions) — @argparse API pattern, not duplicationis_disqualifying_keyword vs is_skip_token) — different purposeplan solid)After cleaning within each directory, find code that should be shared across packages.
# Find overlap between two packages
indexion plan solid --from=src/a,src/b
# Specify extraction target
indexion plan solid --from=src/a,src/b --to=src/common
# Use tree edit distance for precise function-level matching
indexion plan solid --from=src/a,src/b --strategy=apted
# Higher threshold for stricter matching
indexion plan solid --from=src/a,src/b --threshold=0.95
# Filter files
indexion plan solid --from=src/a,src/b --include='*.mbt' --exclude='*_test.mbt'
plan solid differs from plan refactor:
plan refactor | plan solid | |
|---|---|---|
| Scope | Internal duplication within directories | Cross-directory overlap |
| Goal | Consolidate within a codebase | Extract shared code into a new package |
| Input | <path> | --from=dirA,dirB |
plan solid options:
| Option | Default | Description |
|---|---|---|
--from=DIRS | (required) | Source directories (comma-separated or repeatable) |
--to=DIR | -- | Target directory for extraction |
--rules=FILE | -- | Rules file (.solidrc) |
--rule=RULE | -- | Inline rule (repeatable) |
--threshold=FLOAT | 0.9 | Minimum similarity threshold |
--strategy=NAME | tfidf | Similarity: tfidf, apted, tsed |
--include=PATTERN | -- | Include pattern (repeatable) |
--exclude=PATTERN | -- | Exclude pattern (repeatable) |
--format=FORMAT | md | Output: md, json, github-issue |
-o, --output=FILE | stdout | Output file path |
--specs-dir=DIR | kgfs | KGF specs directory |
Workflow:
plan refactor on each directory individually first to clean internal duplicationplan solid --from=dirA,dirB to find cross-directory extraction candidatesindexion grep "TypeIdent:SharedType" to verify all references are updatedplan unwrap)After consolidation, clean up trivial delegation functions that add indirection without value.
# Step 1: Quick check
indexion grep --semantic=proxy src/
# Step 2: Detailed report
indexion plan unwrap --include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' src/
# Step 3: Preview changes (safe — no files modified)
indexion plan unwrap --dry-run --include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' src/
# Step 4: Apply fixes
indexion plan unwrap --fix --include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' src/
# Step 5: Run tests
moon test --target native
What gets detected: Functions whose body is a single function call with all arguments forwarded as simple identifiers — no control flow, no transforms.
// Detected (default) — trivial delegation
fn matches_pattern(text : String, pat : String) -> Bool {
@glob.glob_match(text, pat)
}
// Excluded by default (use --all to include)
fn length(self : MyList) -> Int {
self.items.length() // self-delegation (encapsulation)
}
fn emit(value : String) -> Action {
Emit(value) // bare constructor
}
plan unwrap modes:
| Mode | Flag | Description |
|---|---|---|
| Report | (default) | List wrappers found |
| Preview | --dry-run | Show all edits without modifying files |
| Fix | --fix | Apply edits to files |
plan unwrap options:
| Option | Default | Description |
|---|---|---|
--dry-run | -- | Preview edits |
--fix | -- | Apply edits |
--all | -- | Include self-delegation and bare constructor wrappers |
--include-self | -- | Include self.field.method patterns |
--include-bare | -- | Include bare constructor wrappers |
--include=PATTERN | -- | Include pattern (repeatable) |
--exclude=PATTERN | -- | Exclude pattern (repeatable) |
--format=FORMAT | md | Output: md, json, text |
-o, --output=FILE | stdout | Output file path |
--specs-dir=DIR | kgfs | KGF specs directory |
Review before removing:
@osenv_path) are abstraction layers, not accidental indirection--dry-run firstexplore + analysis)This is the hardest level. Textual and structural tools won't find it because the code is different — but the concept is the same.
# Find which files share vocabulary (= work in the same concept domain)
indexion explore --threshold=0.4 \
--include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' \
src/ cmd/
Files at 40-60% similarity without structural duplication are concept neighbors — they use the same terms because they deal with the same domain.
For each high-similarity pair, ask: "What concept do they share, and who owns it?"
# Inspect shared vocabulary with tree structure comparison
indexion explore file_a.mbt file_b.mbt --threshold=0 --strategy=apted
Common patterns of concept leakage:
| Symptom | Concept leaked | Fix |
|---|---|---|
Both files call is_X(spec) then Y::from_spec(spec) | "Determine if X and configure Y" | Extract try_do_X(path, spec) into the module that owns X |
Both files @fs.read_file_to_string(path) when content is already loaded | "Read file content" | Pass content as argument, don't re-read |
Both files parent_dir(path) then @fs.read_dir(dir) | "List sibling files" | Centralize directory walking into pipeline |
Multiple if is_virtual_path(x) { skip } guards | "Real vs virtual path" | Make the type system prevent virtual paths from reaching here |
Both files buf.write_string("\n"); buf.write_string(x) | "Join text entries" | Extract join_text_entries() into the owning module |
The module that defines the concept should be the only one that implements the logic.
Rules:
vfs.mbt, discover.mbt, and args.mbt, it belongs in vfs.mbt only.is_archive_spec + ArchiveSpec::from_spec + expand_archive separately. Export try_extract_archive_text(path, spec) -> String?.if is_virtual_path(x) { skip } means virtual paths shouldn't reach here at all. Fix the source, not the sink.SupportedFile.content holds the text, no downstream code should call @fs.read_file_to_string(file.path).# Confirm textual duplication is gone
indexion plan refactor --threshold=0.9 \
--include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' \
src/ cmd/indexion/
# Confirm concept similarity is reduced
indexion explore file_a.mbt file_b.mbt --threshold=0
# Confirm wrappers are cleaned up
indexion plan unwrap --include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' src/
# Run tests
moon test --target native
After SoT consolidation:
Write a test that structurally prevents the old pattern from recurring:
test "SoT: SupportedFile.path is always a real filesystem path" {
// Create an archive, run load_supported_file_info
// Assert: no path contains "!/"
// Assert: every path passes @fs.path_exists
}
The test doesn't check behavior — it checks the SoT invariant.
If you're adding if is_special_case(x) { skip } to a function that shouldn't receive
special cases, the problem is upstream. The function's caller should never pass that value.
Stderr messages from C runtime (opendir: No such file or directory) mean invalid data
reached a system call. catch absorbs the error, but perror() already printed.
The only fix is preventing invalid data from reaching the call.
If the same fix is needed in explore, search, grep, reconcile, plan documentation... the fix belongs in the shared pipeline, not in each command.
40-60% TF-IDF similarity between modules that aren't supposed to share concepts is a warning. The vocabulary match IS the signal.
| Question | Command |
|---|---|
| "What files are similar?" | explore --format=list |
| "What exactly is duplicated?" | plan refactor --threshold=0.9 |
| "What code overlaps between packages A and B?" | plan solid --from=A,B |
| "Which functions are trivial wrappers?" | plan unwrap or grep --semantic=proxy |
| "What concept do these files share?" | explore file_a file_b --threshold=0 --strategy=apted |
| "Has the duplication been fixed?" | Re-run plan refactor with same threshold |
README construction — initialize template structure, generate per-package READMEs from doc comments, plan writing tasks, assemble root README from docs/ and package READMEs via doc.json config, and verify edits with `plan drift`.
Detect and review name/content drift in code using `indexion identity audit`, then plan verified renames, moves, folder changes, or splits.
Generate and use a pre-edit structure brief so coding agents learn likely owners, consumer surfaces, and unsafe edit locations before implementing.
Generate SDD requirements from RFCs/specs and quantitatively verify implementation conformance. spec draft → spec align → spec verify → automated validation loop with codex/claude. Operate spec-to-impl drift gates as CI checks.
Documentation analysis — assess coverage, detect code-to-doc drift with plan reconcile, visualize dependencies with doc graph. Answers "what needs docs?" and "are docs still accurate?"
Project wiki lifecycle — create pages, track source changes with ingest, update stale pages, lint structural integrity, detect code-to-doc drift with reconcile, and manage search indexes. A runbook for navigating and maintaining the wiki as an agent.