project-decomposition

# Project Decomposition: Recon, Grouping & Validation ## What this skill does This is the single entry point for measuring a codebase scope and (when needed) decomposing it into topology groups. Two modes: 1. **Lightweight recon (JSON)** — `scripts/decompose.py ... --profile-json` produces a structured `ProjectProfile` JSON with LOC / file counts / module list / dep graph summary / skip patterns. Used by upstream planners to decide pipeline variant and grouping needs. 2. **Full grouping + validation** — `scripts/decompose.py` directly, for L0→L1→L2 grouping protocol and L3 quality validation. Used when the workflow plan declares `grouping.needed: true`. Both modes are the same `decompose.py` engine — single source of truth for LOC counting (`--lang` defines the LOC semantic boundary; `--exclude` defines the directory boundary), graph extraction, and validation scoring. ### Which mode to run | Caller intent | Use | Why | |---------------|-----|-----| | Decide whether to fan out at all (sizing, dispatch) | `--profile-json` | Returns structured ProjectProfile (LOC, module count, depth, hub-ratio, sizing hints) without grouping output. Cheap, machine-readable. | | Caller already decided to fan out and needs L1 numbers + L2 grouping + L3 validation | run without `--profile-json`; then again with `--validate` for L3 | Human-readable Graph Statistics + Quality Scores. | | Caller wants both | run twice: once with `--profile-json` for the planner, once without for human L1 | The flags are mutually exclusive in one invocation; do not combine `--profile-json` with `--validate`. | `--profile-json` and `--validate` cannot be combined in a single invocation (the script rejects this). ## When to Apply **Scale threshold**: Codebase is large enough that no single agent reading the source can hold any one group's source in context, or has enough modules that dependency-aware grouping adds value. The unit being sized is **one group's codebase scope** (which many agents in the pipeline must read), not a "one-agent workload" — every group still runs a full plan→execute→validate pipeline with many agent sessions. **Scope check** (after scale threshold is met, applied by the caller — see Workflow § input contract for the field this maps to): 1. The caller determines the **rewrite scope** from the user request and passes it via the `Scope:` dispatch field: - **Full rewrite** — all modules in scope → `scope = full` - **Partial rewrite** — user specifies modules/layers/features → `scope = scoped` - **Cross-layer rewrite** — rewrite targets a different architectural layer (e.g. Razor → Angular while backend stays) → `scope = scoped` (only the target layer is in scope; source-layer modules are `context-only`) 2. Apply the scope (classify groups using the dispatched value): - `full` — all groups are `in-scope` - `scoped` — analyze all modules for the dependency graph, but classify each group: - **`in-scope`** — being rewritten → generates DAG tasks - **`context-only`** — depended on by in-scope but NOT being rewritten → no DAG tasks, dependency context only 3. **Skip entirely** if all of the following hold for the in-scope set: - DAG depth ≤ 2 AND module count ≤ 3, AND - total in-scope LOC < `ideal_group_loc` (from `topology_hints`) Report skip rationale back to the caller (plain text reply, not the artifact); do not produce a topology artifact. ## Architecture: 3 Layers | Layer | Output | |-------|--------| | **L1 Extract** (`scripts/decompose.py <path> --lang '<lang>' --exclude '<patterns>'`) | Modules, edges, SCCs, DAG layers, LOC, oversized flags. No grouping. | | **L2 Group** (caller agent) | Module groups using L1 data + topology-aware judgment. See `references/grouping.md`. | | **L3 Validate** (`scripts/decompose.py <path> --lang '<lang>' --exclude '<patterns>' --validate '<spec>'`) | 4 raw Quality Scores (Outlier ratio, SCC integrity, Group cycles, Coverage). Definitions and judgment protocol live in `references/grouping.md` §4 and `references/topology-thresholds.md`. | Do NOT skip any layer. L3 is the only authoritative quality check — hand-computed metrics are not accepted. ## Files (load on demand) | File | Load when | |------|-----------| | `references/grouping.md` | L2 step — topology-aware grouping rules + L3 metric definitions (authoritative source for thresholds) | | `templates/output-template.md` | Output step — project topology artifact format and fixed path | | `references/consume.md` | Downstream consumer contract — how the artifact is read and accepted | | `references/manual-extraction.md` | Only if L1 reports `Unsupported language` (rare — C#/Java/Python/JS-TS supported) | **Python is mandatory for topology grouping.** `decompose.py` is the only authoritative LOC counter for grouping/L3 metrics — external counters drift across runs and inflate by 5–15% on comment-heavy languages, which destabilizes outlier-ratio and SCC-integrity scoring. `scripts/decompose.py` invocation surface: - `<path> --lang '<lang>' --exclude '<merged>'` — extract mode - `--validate '<spec>'` — L3 quality gate - `--module-loc <Module>` — sub-dir LOC for a specific module - `--json out.json` — machine-readable dump - `--profile-json` — `ProjectProfile` JSON to stdout (see **JSON Profile Mode** below) `--lang` and `--exclude` are required. `--lang` accepts `python|java|csharp|javascript` (or aliases: `py`, `cs`/`c#`/`dotnet`, `js`/`ts`/`node`). For mixed projects pass comma-separated, e.g. `--lang csharp,javascript`. `--exclude` takes the **merged** list (see Workflow § effective_exclude). ## Workflow The caller dispatches with a fixed input contract — these fields MUST be present, do not infer them yourself: | Field | Type | Example | Used by | |-------|------|---------|---------| | `Source path:` | absolute path to repo root | `/path/to/repo` | every L1/L3 invocation as `<source-path>` | | `Language(s):` | comma-separated `decompose.py --lang` value | `csharp` or `csharp,javascript` | `--lang '<lang>'` | | `Skip patterns:` | comma-separated path segments (may be empty) | `tests,docs,samples` | exclude-merge input | | `Scope:` | `full` \| `scoped` | `scoped` | Scope check classification | | `Mode:` | `topology` \| `profile-json` | `topology` | output shape — `profile-json` returns `ProjectProfile` JSON to stdout instead of producing the artifact (see **JSON Profile Mode** below) | All five fields are required. If any is missing, halt and request it from the caller — do **not** guess or default. Then build the **effective exclude list** by union-ing four sources: ``` effective_exclude = caller_skip_patterns ∪ gitignore_dirs(<source-path>/.gitignore) ∪ auto_gen_dirs(<source-path>) ← see below ∪ BASELINE ``` **`auto_gen_dirs(path)`** — pre-L1 sniff for framework-generated code that the caller cannot anticipate (EF Core `Migrations/`, protobuf-generated stubs, OpenAPI/GraphQL codegen, ANTLR `.g4`-output dirs, etc.). These dirs may contain tens of thousands of lines that are not authored code; they distort outlier ratio and force spurious sub-splits. Procedure: 1. Walk top 2 levels under `<source-path>` (skipping anything already in BASELINE / gitignore_dirs / caller_skip_patterns). 2. For each directory whose **source-file LOC** (matching `--lang` extensions) ≥ 5% of project total, sample up to 3 source files (largest first) and read the **top 100 lines** of each. 3. If ≥ 1 sampled file contains any of these markers (case-insensitive), add the directory's segment name to `auto_gen_dirs`: - `<auto-generated>` - `DO NOT EDIT` - `This file was generated` - `@generated` - `Code generated by` (Go convention) - `auto-generated by` 4. Cap at 8 directories. Log each accepted dir and the marker that triggered it; this list is reproducible across runs. Do NOT use this mechanism to skip hand-written code that happens to be large — only marker-confirmed generated code qualifies. If marker matching is ambiguous, leave the dir in and let the outlier-ratio band trigger normal remediation. **BASELINE** (language-neutral, fixed list, always applied — fallback when caller/gitignore miss something): ``` .git, .svn, .hg, .idea, .vscode, .vs, .DS_Store, node_modules, bower_components, vendor, packages, __pycache__, .venv, venv, .tox, .pytest_cache, .cache, bin, obj, target, build, dist, out, .gradle, .mvn ``` BASELINE is fixed and language-neutral. Project-type-specific extras (framework directories, generated folders) are the **caller**'s responsibility and must be passed via `caller_skip_patterns`; do not extend BASELINE per project. **`gitignore_dirs(path)`** — parse repo's `.gitignore` and extract directory-shaped entries: - keep: lines ending in `/`, or bare path segments without glob chars (e.g. `wwwroot/lib/`, `App_Data`) - drop: blanks, comments (`#…`), negations (`!…`), file-level globs (any line containing `*` `?` `[`) - strip leading/trailing `/`; dedupe across all three sources - if `.gitignore` is missing, this set is empty Pass the merged list verbatim in every `decompose.py` invocation as `--exclude '<merged>'`. Rationale: build output, vendored deps, and generated assets the repo itself ignores must not inflate LOC or group sizes — relying on the caller alone is fragile. 1. **Extract (L1)** — Invoke `scripts/decompose.py <source-path> --lang '<lang>' --exclude '<merged>'`. Treat output as source of truth; do NOT re-extract module/edge/LOC data from raw source. Apply the caller's `scope` to classify modules. 2. **Group (L2)** — Load `references/grouping.md`. Use L1 data + topology rules to form groups. 3. **Validate (L3, MANDATORY)** — Invoke `scripts/decompose.py <source-path> --lang '<lang>' --exclude '<merged>' --validate 'G1:M1,M2|G2:M3,M4|...'`. Classify path (Path A vs Path B) and judge Quality Scores per `references/grouping.md` § Scored Validation Protocol & Convergence Loop. Paste full output into Provenance. 4. **Output** — Load `templates/output-template.md`. Produce the artifact at the fixed path. 5. **Consume** — Downstream consumers load `references/consume.md`. The caller runs an additional acceptance check (see consume.md § Acceptance Check). --- ## JSON Profile Mode (`--profile-json`) For upstream planners that just need a structured snapshot of the scope — no grouping, no L2/L3 — invoke `decompose.py` with `--profile-json` instead of running the full L1→L2→L3 workflow. ``` scripts/decompose.py <scope_path> --lang '<lang>' --exclude '<merged>' --profile-json ``` - `<scope_path>`: project root, module dir, or any walkable sub-folder. - `--lang`: required. Same canonical names + aliases as the extract mode. Multi-lang via comma. - `--exclude`: same semantics as extract mode (see Workflow § effective_exclude). The script ships with NO defaults beyond `.git`. - `--profile-json`: emits `ProjectProfile` JSON to stdout and suppresses normal text output. > **Note on `.gitignore`**: `.gitignore` uses glob/negation/anchored semantics that don't map cleanly to the script's segment-substring exclusion. The caller assembles the merged exclude (see Workflow § effective_exclude) and passes it explicitly via `--exclude`. The script does NOT parse `.gitignore` itself. Outputs `ProjectProfile` JSON to stdout. Schema: ```json { "scope_path": "/abs/path", "languages": ["csharp"], "skip_patterns": ["bin", "obj", "..."], "total_loc": 314217, "total_files": 4321, "module_count": 36, "edge_count": 40, "scc_count": 36, "scc_nontrivial": 0, "dag_depth": 6, "topology_hints": { "target_group_count": 6, "ideal_group_loc": 52369, "median_module_loc": 1182, "trivial_module_test": {"abs_loc": 500, "pct_total": 0.01, "op": "or"}, "oversized_threshold": 50000 }, "modules": [ {"name": "Nop.Core", "loc": 7143, "files": 355, "layer": 5, "in_deg": 4, "out_deg": 0, "flags": []} ], "dag_layers": [ {"layer": 0, "module_count": 31, "loc": 69913} ], "warnings": ["LOW COVERAGE: 27% of project code is outside discovered modules."] } ``` `topology_hints` field meanings: - `target_group_count` — recommended number of topology groups: `ceil(total_loc / 80_000)`, clamped to `[max(2, ceil(module_count / 12)), max(2, ceil(module_count / 4))]`. Computed by L1. - `ideal_group_loc` — `total_loc / target_group_count`. Used by grouping rules to size merges and judge group-LOC distribution. - `trivial_module_test` — module is trivial iff `loc < abs_loc OR loc/total_loc < pct_total` (note: OR, not AND). **Single source of truth**: profile mode and extract/validate mode share the same in-memory analysis pass — LOC counting, `--lang` semantics, and `--exclude` semantics are identical by construction. If you need grouping or `--validate`, omit `--profile-json` and invoke `decompose.py` directly. --- ## Anti-patterns - ❌ Hand-computing quality scores instead of running L3 `--validate` - ❌ Re-extracting modules/edges/LOC from raw source instead of using L1 output - ❌ Grouping by name or functional domain instead of dependency affinity (e.g. "payment plugins" — group by topology, not by what modules do) - ❌ Splitting or moving modules between groups to equalize LOC sizes (corrupts dependency graph — size rebalancing belongs upstream). Note: merging trivial modules per `grouping.md` §1 is topology-driven, not LOC rebalancing. - ❌ Skipping L3 validation because the artifact "looks reasonable"