mantis-structural-index

# Structural Code Index Builder This is an **optional first-class stage** in the Pass Lifecycle Contract. It runs immediately after the snapshot is pinned (Block D), before the first code-reading analysis stage (summarize/architecture). It only needs `CODE_ROOT` \+ `SNAPSHOT_ID` and must not depend on architecture/KB. ## System Goal Structural Code Index Builder. Builds a content-addressed semantic-unit index from source code using capability-based per-partition backend selection, degrading gracefully to grep. Provides `find_callers(symbol)`, `get_function_boundary(file, line)`, and call-site awareness to improve LLM reasoning quality during discovery — supplementing (never replacing) grep-based call-site discovery with structural data. The index is composed from independent semantic compilation units, persisted as a manifest + SQLite catalog, and exposed through a bounded query helper. ## Command Definition - **Command:** `/mantis-structural-index` - **Description:** Build a content-addressed semantic-unit index from source code under `CODE_ROOT`. - **Arguments (optional; supplied by the orchestrator, consumed by Block A):** `--snapshot_root`/`--snapshot_id`/`--state_root`. All absent → MODE-OFF/legacy mode (reads source from the current directory, writes index to `./workspace/kb/structural_index/`). ## Input/Output Contract - **Reads**: - `workspace/.mantis_state.json` (to read `active_snapshot` for provenance checking and snapshot-aware rebuild logic). - `workspace/kb/structural_index/manifest.json` (to check `snapshot_id` for reuse-on-match idempotency — primary check). - `workspace/kb/structural_index.jsonl` (backward-compat provenance check if manifest absent). - `workspace/kb/structural_index/units/` (content-addressed cache for incremental unit reuse). - `workspace/kb/structural_index/native/` and sidecar `provenance.json` files (prebuilt index attachments and metadata manifests). - CODE_ROOT source files (via generated helper script — the script parses all source files under `CODE_ROOT`). - **Writes**: - `workspace/kb/structural_index/manifest.json` (STATE-RELATIVE — atomic commit point, written LAST). - `workspace/kb/structural_index/catalog.sqlite` (STATE-RELATIVE — query-optimized serving store). - `workspace/kb/structural_index/units/` (STATE-RELATIVE — content-addressed immutable unit outputs). - `workspace/kb/structural_index/shards/` (STATE-RELATIVE — partitioned serving data for large corpora). - `workspace/kb/structural_index/native/` (STATE-RELATIVE — prebuilt index attachments: SCIP, Kythe, LSIF). - `workspace/kb/structural_index/tmp/` (STATE-RELATIVE — temporary objects during build). - `workspace/kb/structural_index.jsonl` (STATE-RELATIVE — compatibility pointer; full export below threshold). - `workspace/helpers/build_structural_index.py` (STATE-RELATIVE — the builder helper script). - `workspace/helpers/query_structural_index.py` (STATE-RELATIVE — the query helper script). - **Preconditions**: - Source files must exist under `CODE_ROOT`. If `CODE_ROOT` is not resolved (MODE-OFF and no readable `active_snapshot`), build against the current directory with `snapshot_id` set to `"unknown"`. Do NOT skip — this is the standalone-efficiency case. - **Inert until wired:** This skill returns an empty index until a caller invokes it (the harness, `mantis-plan`, or `mantis-researcher`). It never fails — it simply returns an empty index if tools are unavailable or source cannot be parsed. - **Idempotency Guarantee**: - Read-only on `CODE_ROOT`. Writes only to STATE-RELATIVE paths. Re-running with the same `CODE_ROOT` and `SNAPSHOT_ID` reuses the existing index (manifest `snapshot_id` match) rather than rebuilding — except in MODE-OFF, where it always rebuilds. Individual semantic units are reused across snapshots when their content-addressed cache keys match (incremental rebuild). ## Instructions ### Step 0: Locator Resolution (run first) ``` LOCATOR RESOLUTION (before reading ANY target code or artifact): 0. ROLE: If this skill NEVER reads target source (report, calibrate, reflect), you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from state for provenance/annotation; NEVER stop merely because a code root is unset. 1. Determine CODE_ROOT, in this priority order: a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root. It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback (used when a caller hands you a prepared tree, e.g. a patched shadow). b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it. c. Else read state_root/workspace/.mantis_state.json (state_root from --state_root if passed, else ./workspace/... relative to the current dir) -> active_snapshot.root / .snapshot_id / .snapshot_pinned. d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory, treat snapshot_pinned = false (MODE-OFF). Do NOT stop. 2. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a): verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is deliberately mutated and is sentinel-EXEMPT.) 3. PATH FIELDS: - SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is NOT of the form <existing-path>:<integer> is a non-source LOCATOR (symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL line-range and line-existence logic. - STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT): kb_references, repro_file_path, reattack_file_path, helper scripts, report files, and all state/findings JSON. 4. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy (mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may cd into CODE_ROOT. 5. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT (the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT lacks .git/.hg/.repo. 6. Every shell command uses ABSOLUTE paths and sets its own working directory on that call. Do NOT assume the working directory persists between calls. ``` This is a CODE-READING stage — it reads target source files under CODE_ROOT via the helper script. Block A step 0's findings-only skip does NOT apply. ### Step 1: Idempotency / Freshness Check (MANDATORY FIRST) **MANDATORY FIRST STEP:** Before writing any build scripts, probing backends, or extracting symbols, check for MODE-OFF or an existing manifest: 1. **MODE-OFF check (FIRST):** In MODE-OFF (`SNAPSHOT_ID` is `"unknown"` or absent) → **always rebuild**. Do NOT reuse a previous `"unknown"` index, because the live tree is mutable and `"unknown"` is a constant (not a freshness signal). Skip directly to Step 2 (individual units whose content has not changed may still hit the content-addressed cache). 2. If `workspace/kb/structural_index/manifest.json` exists, read its `snapshot_id` field. 3. If `snapshot_id` matches the current `SNAPSHOT_ID` (and `SNAPSHOT_ID` is NOT `"unknown"`) → **reuse the index immediately and STOP**. Do NOT invoke Step 2, probe backends, or write build scripts. This bounds cost across retries/crash-resume. 4. If `manifest.json` is absent but `workspace/kb/structural_index.jsonl` exists (backward compat), read its provenance header (`_provenance`, `snapshot_id` keys). If `snapshot_id` matches (and is NOT `"unknown"`) → **reuse and STOP**. Otherwise proceed to rebuild. 5. If `SNAPSHOT_ID` differs → proceed to rebuild (Steps 2–5). For incremental reuse: before rebuilding a semantic unit, check `units/` for a content-addressed cache hit (see Content-Addressed Cache Key below). A cache hit reuses the unit output without re-extraction. ### Step 2: Select Backend per Partition Partition the codebase into semantic units (see Per-Language Semantic Units below). For each unit, select the most precise backend available in this environment using a **capability-based, per-partition decision** — not a single global ladder. The following are **examples, not an exhaustive enum**. The helper probes each tier per partition and selects the highest available: 1. **Snapshot-matched SCIP / LSIF / Kythe / clangd-static / remote index** (precision: `semantic`) — if a pre-built index matching the current `snapshot_id` or `root_fingerprint` is available. Most precise: full type-aware cross-reference, call hierarchy, and hover/signature data. 2. **Compiler / typechecker-backed extractor** (precision: `typecheck`) — if `compile_commands.json`, build context, or typechecker is available. Type-accurate symbol resolution and call edges. 3. **Language-aware AST extraction** (precision: `ast`) — if `tree-sitter`, `ast-grep`, or a language-specific parser is available. Full AST parsing: function boundaries, call expressions, signatures. 4. **Symbol-only extraction** (precision: `symbol-only`) — if `ctags` or equivalent is on `PATH`. Symbol table only (function definitions, locations — no call graph). Call-site extraction uses a lightweight regex pass within known function boundaries. 5. **Heuristic fallback** (precision: `heuristic`) — Python stdlib regex pass over source files. Identifies function definitions and call patterns using language-agnostic heuristics. Less precise but zero-dependency. 6. **Coverage-only manifest + lexical fallback** (precision: `coverage-only`) — grep; no structural index is written. Manifest records `status: "empty"`. Consumers fall back to grep-based discovery (today's behavior byte-for-byte). **Native Index Probing & Resolution Rules**: - **Probe instruction**: Before evaluating per-partition backends, probe `workspace/kb/structural_index/native/` and subdirectories `native/{scip,lsif,kythe}/` for prebuilt index files. - **Snapshot-declaration convention**: Because native formats (SCIP, LSIF, Kythe) do not embed snapshot identity directly in their binary payload, prebuilt indexes MUST declare their target snapshot using a sidecar `provenance.json` manifest located at `workspace/kb/structural_index/native/provenance.json` or `native/<kind>/provenance.json`. The manifest contains an array of attachments: `[{"kind": "scip|lsif|kythe", "path": "...", "snapshot_id": "...", "root_fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}]`. A native index is matched if its declared `snapshot_id` equals `SNAPSHOT_ID` (when `SNAPSHOT_ID != "unknown"`) or its `root_fingerprint` matches the workspace's calculated root fingerprint. If the `files` array is absent or empty, treat the native index as covering no individual files directly (record in `manifest.native_indexes` but do not update `coverage` rows; fall through to lower tiers for all files). - **Record-and-Defer Ingestion Rule**: Parsing raw binary native indexes (e.g. SCIP protobuf) in pure Python without dependencies is costly and complex. Option A builder scripts MUST detect matching prebuilt native indexes, record their entries in `manifest.native_indexes`, and set the `coverage` table `backend` (e.g., `"scip"` or `"scip-clangd"`) for all files listed in the provenance manifest. If `catalog.sqlite` is NOT populated with symbols from the native index (raw binary deferred to harness/MCP readers), set `coverage.status = "deferred"` and `precision = "deferred"` (with `indexed_files = 0`). This prevents the query helper from claiming an un-ingested partition is "authoritative empty" at `semantic` precision, ensuring consumers run the mandatory grep fallback. When `catalog.sqlite` IS populated (e.g., via Option B pre-ingestion or `scip-to-sqlite`), set `precision = "semantic"` and `status = "indexed"`. **LSP is NOT equivalent to SCIP / LSIF.** LSP is an interactive protocol whose workspace state may be partial or mutable. Use it only when the server can demonstrate snapshot identity AND complete workspace coverage. A running language server does not automatically qualify as a `semantic` backend. **SCIP merging.** SCIP explicitly permits merging complementary information from indexers with different precision levels. Its format records indexer version and per-document language metadata. The builder MAY merge results from multiple indexers (e.g., a SCIP index for Go + tree-sitter for Python) within a single catalog. The determinism lives in a runtime-generated versioned helper (`build_structural_index.py`, `# MANTIS_HELPER_VERSION = 5`, grep-and-regenerate on reuse) that probes and selects backends per partition. No shipped binaries; air-gapped-safe. **Deterministic partial coverage.** On very large source trees, a full rebuild can dominate stage-0 wall-clock. Replace discovery-order truncation with a deterministic priority queue: 1. **Explicit target files and symbols** (from `plan.json`, if available). 2. **Changed units and known reverse dependencies** (from Block E diff + reverse-dependency edges in the existing catalog). 3. **Containing packages/modules and direct imports** of tier 1+2 units. 4. **Remaining units in normalized path order**. Apply deterministic `max_units` (default: 10000) or `max_source_bytes` (default: 500MB) bounds to that queue. Persist deferred units in the manifest so another invocation can resume. Statuses: `complete`, `partial`, `empty`, `failed`. A consumer should only interpret "no callers" strongly when the relevant partition is complete, snapshot-matched, covered, and semantically precise. Otherwise the answer is "no indexed callers", followed by the existing lexical fallback. The index is HINT-only, so partial coverage is safe (grep remains authoritative). ### Step 3: Write and Run Helper Scripts Two runtime-generated helpers are used. Both follow the grep-and-regenerate pattern: before reuse, grep the first line for the version marker; if absent or a different integer, REGENERATE. #### Builder: `build_structural_index.py` 1. Write the builder to `workspace/helpers/build_structural_index.py`. The FIRST LINE MUST be exactly `# MANTIS_HELPER_VERSION = 5`. Before reusing an existing helper, grep its first lines for `MANTIS_HELPER_VERSION = 5`; if that marker is absent or a different integer, REGENERATE the helper. 2. The builder partitions the codebase into semantic units (see Per-Language Semantic Units), computes content-addressed cache keys, checks `units/` for reuse, selects a backend per partition (Step 2), extracts symbols + edges, and writes results to `catalog.sqlite`. 3. The builder must extract: - **Symbols**: `symbol_id`, `name`, `qualified_name`, `namespace`, `language`, `file_path`, `start_line`, `end_line`, `kind`, `signature`, `backend`, `precision`. - **Call edges**: `caller_id`, `callee_id`, `callee_name`, `file_path`, `line`, `edge_kind`. - **Function boundaries**: `symbol_id`, `file_path`, `start_line`, `end_line`, `signature`, `language`. - **Coverage**: `file_path`, `indexed`, `backend`, `precision`, `unit_cache_key`, `status`. 4. The builder writes the manifest LAST (Step 4) and the compatibility pointer. 5. The builder MUST use ABSOLUTE paths and set its own working directory (Block A step 6). It MUST NOT write anything under `CODE_ROOT` when `snapshot_pinned` is true (Block A step 4). Backends that produce sidecar files (ctags `tags`, cscope `cscope.out`, clangd cache) MUST be redirected to STATE-RELATIVE paths: `ctags -f <state>/helpers/tags`, `cscope -f <state>/helpers/cscope.out`, `CLANGD_INDEX_STORAGE=<state>/helpers/`. Read-only LSP/SCIP queries to a running server need no redirect. #### Query helper: `query_structural_index.py` 1. Write the query helper to `workspace/helpers/query_structural_index.py`. The FIRST LINE MUST be exactly `# MANTIS_HELPER_VERSION = 5`. Before reusing an existing helper, grep its first lines for `MANTIS_HELPER_VERSION = 5`; if absent or a different integer, REGENERATE. 2. The query helper provides bounded, paginated operations against `catalog.sqlite` (or a remote endpoint — identical API). It IS the consumption contract (see Query Interface below).