index-inputs

name	index-inputs
description	Index Input Policy Documents

Index Input Policy Documents

Scan a domain's input/policy_docs/ directory and produce:

• policy_facets/input-index.yaml — metadata only: the files: block (SHAs, md_quality scores). Always small. Fully rewritten on every run.
• policy_facets/compressed/<rel>.md (one file per source doc) — caveman-compressed (token-reduced) mirrors of input/policy_docs/<rel>.md. Generated by /compress-input per file as part of the unified per-file batch and consumed by downstream skills (/extract-ruleset, /update-ruleset).
• policy_facets/computations/<rel>.md.yaml (one file per source doc) — per-file YAML map {sections} of {heading, summary, tags, stage?, stage_source?, computations} blocks. Each computations[] entry carries description:, optional preconditions:, and an optional expr_hint: of the form output_name = <expression>. Generated by /extract-computations per file as part of the unified per-file batch.

Per-file work is dispatched as a bounded fan-out of subagent workers — each worker reads one source file and runs all applicable per-file actions for that file (compress + extract) in the same context. Action outcomes are recorded as per-file marker files under policy_facets/.compress-plan.d/ and policy_facets/.extract-plan.d/; the matching xlator <tool> --finalize collates markers and updates the manifests. Concurrency is bounded by the XLATOR_AI_CONCURRENCY environment variable (default 3); the analyst can raise it once their Anthropic-tier rate-limit headroom is confirmed.

On first run (CREATE), all files are dispatched and every per-file output is generated. On subsequent runs (UPDATE), only changed or new files are re-processed using git SHA comparison; unchanged files are left alone via the per-action manifests.

Input

/index-inputs <domain>

If <domain> is not provided, list all $DOMAINS_DIR/*/input/policy_docs/ directories as a numbered menu, prompt the user to choose, await their response, and use it as <domain> before continuing.

Read ../../core/output-fencing.md now.

Pre-flight

Run these checks before doing anything else:

1. Domain argument provided?

   • NO → List all directories matching $DOMAINS_DIR/*/input/policy_docs/ as a numbered menu and prompt: :::user_input Available domains:
     1. snap
     2. example_domain Which domain? Enter a number or domain name: ::: Await the user's response and use it as <domain>. Then continue.

2. Domain folder exists?

   • NO → :::error Domain not found: $DOMAINS_DIR// ::: Then stop.

3. Input docs present?

   • $DOMAINS_DIR/<domain>/input/policy_docs/ missing or contains no .md files → :::error No input documents found. Add .md files to $DOMAINS_DIR//input/policy_docs/ and re-run. ::: Then stop.

4. Input files committed to git? Check for uncommitted changes (modified, staged, untracked) under $DOMAINS_DIR/<domain>/input/:

   git -C "$DOMAINS_DIR/<domain>" status --porcelain -- input/
   

   • If output is empty, continue.
   • If output is non-empty, list the affected paths and ask: :::user_input The following files under input/ are not committed to git: <list of paths from git status --porcelain> Commit them now? [y/n] :::
     • If y: stage and commit only those paths with message chore(<domain>): commit input files for indexing, e.g.:

       git -C "$DOMAINS_DIR/<domain>" add -- input/
       git -C "$DOMAINS_DIR/<domain>" commit -m "chore(<domain>): commit input files for indexing"
       

       Then continue.
     • If n: continue without committing. SHA detection via git hash-object still picks up working-tree edits, but the source state is not preserved in history.

Pre-step A: Install policy_facets/CLAUDE.md notice

Ensure policy_facets/ exists and overwrite its CLAUDE.md from the static template so the folder identifies itself as derived (not original) for any agent that walks into it:

mkdir -p "$DOMAINS_DIR/<domain>/policy_facets/"
cp "$CLAUDE_PLUGIN_ROOT/core/policy_facets_claude.md" "$DOMAINS_DIR/<domain>/policy_facets/CLAUDE.md"

The template is overwritten on every run so edits to xl-plugin/core/policy_facets_claude.md propagate without an extra command.

Pre-step B: Plan all per-file work

Run BOTH per-action plans up-front:

xlator compress-inputs <domain> --plan
xlator extract-computations <domain> --plan

Each emits its own work plan as JSON. The shapes are:

// xlator compress-inputs --plan
{
  "to_compress": [ {"src": "...", "dst": "...", "source_sha": "..."}, ... ],
  "to_delete":   [ "policy_facets/compressed/<rel>.md", ... ],
  "noop":        [ {"src": "...", "reason": "unchanged"}, ... ],
  "skipped":     [ {"src": "...", "reason": "not_allowed"}, ... ]
}

// xlator extract-computations --plan
{
  "to_extract":  [ {"src": "...", "dst": "...", "source_sha": "..."}, ... ],
  "to_delete":   [ "policy_facets/computations/<rel>.md.yaml", ... ],
  "noop":        [ {"src": "...", "reason": "unchanged"}, ... ],
  "skipped":     [ {"src": "...", "reason": "not_allowed"}, ... ]
}

Capture both JSON outputs. Build the unified per-file action map: for each source rel that appears in compress.to_compress ∪ extract.to_extract, record the action set as the subset that applies — ["compress"], ["extract"], or ["compress", "extract"]. Files appearing in neither plan's to_* list (i.e., both manifests classify them as noop) are not dispatched in Step 5 — they are already in sync.

Each --plan invocation also creates an empty .compress-plan.d/ and .extract-plan.d/ directory under policy_facets/ (clearing any stale markers from a prior crashed run). Workers in Step 5 write their per-file outcome markers there.

/index-inputs itself continues to scan input/policy_docs/ for SHA and md_quality scoring — its index reflects the source files, not the compressed copies and not the computations files.

Mode Detection

ls $DOMAINS_DIR/<domain>/policy_facets/input-index.yaml 2>/dev/null

Result	Mode
File absent	CREATE — index all files from scratch
File present	UPDATE — re-index changed/new files only

Quality Threshold

The minimum acceptable quality score is 40 (out of 100). Files scoring below this threshold are REJECTED: they are moved to $DOMAINS_DIR/<domain>/input/rejected/ (preserving subdirectory structure) and no per-file output is generated for them. The md_quality: field in the files: block explains the rejection.

Concurrency

Read XLATOR_AI_CONCURRENCY from the environment; default to 3 if unset. This is the number of subagent workers that run in parallel per batch in Step 5.

K="${XLATOR_AI_CONCURRENCY:-3}"

A value of 3 is conservative across Anthropic tiers and stays well within typical analyst rate-limit headroom. The analyst can raise it (e.g., XLATOR_AI_CONCURRENCY=8) once their tier confirms more capacity. There is no explicit 429 retry/backoff in v1 — if rate-limit errors surface, lower the value or set it to 1.

---

Process — CREATE Mode

Step 1: Discover all input documents

Glob all .md files recursively under $DOMAINS_DIR/<domain>/input/policy_docs/. Sort alphabetically by path.

:::progress Found document(s). Indexing all... :::

Step 2: Get git SHA and quality score for each file

For each file, run:

git hash-object $DOMAINS_DIR/<domain>/<domain-relative-path>

git hash-object returns the blob SHA of the file's current content (working-tree state), so the SHA changes whenever the file's bytes change — independent of whether the file is tracked or committed. This is the comparison key the per-action manifests use to decide SKIP vs REINDEX.

If git hash-object itself fails (e.g., git unavailable), use "untracked" and the file will be re-indexed on every run.

After obtaining each file's SHA, score it:

xlator rate-md-quality "$DOMAINS_DIR/<domain>/<domain-relative-path>"

Store the JSON result {"score": N} or {"score": N, "flags": [...]} alongside the SHA.

Step 3: Write the files block to input-index.yaml

Ensure $DOMAINS_DIR/<domain>/policy_facets/ exists (Pre-step B's --plan calls already created policy_facets/computations/ and policy_facets/compressed/). Write the following to $DOMAINS_DIR/<domain>/policy_facets/input-index.yaml (this creates or overwrites the file):

# Auto-generated by /index-inputs — do not edit manually
files:
  input/policy_docs/<file1>.md:
    sha: "<sha>"
    md_quality:
      score: <N>
      flags: [<flag1>, <flag2>]
  input/policy_docs/<file2>.md:
    sha: "<sha>"
    md_quality:
      score: <N>
  "input/policy_docs/<file with spaces>.md":
    sha: "<sha>"
    md_quality:
      score: <N>

YAML conventions for the files: block:

• Two-space indentation throughout
• Quote any path key that contains spaces (e.g., "input/policy_docs/441-1 EARNED INCOME.md":)
• sha: values as quoted strings
• md_quality: sub-block with score: as an unquoted integer
• flags: as an inline bracket list (e.g., flags: [no_headings, repeated_page_headers]) — omit the flags: line entirely when there are no flags (do not emit flags: [])

Important: Write each file's entry to the files: block one at a time (not all files in a single output block) to avoid hitting the 32k output token limit.

Step 4: Move rejected files

Run the rejection script, passing the domain directory and the quality threshold:

xlator reject-low-quality-inputs "$DOMAINS_DIR/<domain>" 40

The script reads the files: block just written, moves every file whose md_quality.score is below 40 from input/policy_docs/ to input/rejected/ (preserving subdirectory structure), and returns a JSON result:

{"moved": <R>, "files": ["input/policy_docs/<file>.md", ...]}

Capture <R> (the moved count) for the completion message.

Step 5: Dispatch unified per-file workers

Filter the unified action map (built in Pre-step B) to remove REJECTED sources — they have been moved to input/rejected/ by Step 4 and are no longer at their input/policy_docs/ paths. For each REJECTED file, emit:

:::progress Skipped (low quality, score=): input/policy_docs/.md :::

For the remaining sources, sort by source path for deterministic batch composition. Dispatch in batches of K (the concurrency cap). Each batch is a single assistant turn with K parallel Agent-tool calls using subagent_type: index-inputs-worker (defined at xl-plugin/agents/index-inputs-worker.agent.md); wait for the batch to return before issuing the next.

Each Agent call's prompt is the per-invocation context block the worker expects:

source_path: <abs path under input/policy_docs/<rel>.md>
domain_dir:  <abs path under $DOMAINS_DIR/<domain>>
source_sha:  "<sha>" or "untracked"
actions: <subset of [
  {name: "compress", skill: "/compress-input",       marker_path: "<domain_dir>/policy_facets/.compress-plan.d/<rel>.md.outcome.json", dst: "<domain_dir>/policy_facets/compressed/<rel>.md"},
  {name: "extract",  skill: "/extract-computations", marker_path: "<domain_dir>/policy_facets/.extract-plan.d/<rel>.md.outcome.json",  dst: "<domain_dir>/policy_facets/computations/<rel>.md.yaml"}
]>

The agent body owns the per-action loop (write in_progress marker → invoke child skill skipping pre-flight → atomically update marker to succeeded/failed → return one succeeded: or failed: line per action) and the path-discipline guarantee — see xl-plugin/agents/index-inputs-worker.agent.md for the full worker contract.

After each batch returns:

1. Verify markers on disk for each (src, action) the batch dispatched. If a worker reported succeeded: but the marker is missing or still in_progress, emit: :::progress ⚠ Worker reported success for but no completed marker — treating as aborted. ::: The action will surface as aborted in the corresponding --finalize summary.

2. Emit a :::progress line per per-action outcome (parsed from the worker's return text), e.g.: :::progress Compressed: input/policy_docs/.md Extracted: input/policy_docs/.md :::

3. Continue to the next batch.

On any per-action failure, do NOT abort the run — failures are surfaced in Step 6's --finalize summaries.

Step 6: Finalize per-action work

Run BOTH --finalize commands. They are independent of each other; either order works, but emit each summary as it completes so the analyst sees per-action outcomes incrementally:

xlator compress-inputs <domain> --finalize

Relay its summary inside :::important:

:::important ✓ Compress: compressed, failed, aborted, deleted, unchanged. :::

(Use the actual numbers from the tool's stdout. If <M> or <A> is non-zero, list each failed/aborted source path on subsequent lines.)

xlator extract-computations <domain> --finalize

Relay its summary inside a separate :::important:

:::important ✓ Extract: extracted, failed, aborted, deleted, unchanged. :::

Each --finalize:

• collates per-file outcome markers from its .compress-plan.d/ / .extract-plan.d/ directory,
• treats missing-or-in_progress markers for to_* entries as aborted (cleans up the dst since action state is indeterminate),
• writes the manifest atomically (tmp + os.replace),
• removes its .compress-plan.tmp / .extract-plan.tmp and its .d/ marker dir.

If either finalize reports any failed or aborted count > 0, exit non-zero so any caller can decide whether to proceed.

Step 7: Completion

After Step 6 completes, print:

:::important ✓ policy_facets/input-index.yaml written (files block). policy_facets/compressed/ written ( file(s) compressed). policy_facets/computations/ written ( file(s) extracted). file(s) rejected (low quality) and moved to input/rejected/ — re-run after reformatting. :::

Omit the "rejected" line if R = 0.

:::next_step Next step: /refine-guidance <domain> to set extraction goals and ruleset guidance :::

---

Process — UPDATE Mode

Step 1: Read the existing files block

Read $DOMAINS_DIR/<domain>/policy_facets/input-index.yaml. Extract the current files: SHA map (including any existing md_quality: values).

Step 2: Discover current input documents

Glob all .md files recursively under $DOMAINS_DIR/<domain>/input/policy_docs/. Sort alphabetically by path.

Step 3: Get current git SHA and quality score for each file

For each current file, run:

git hash-object $DOMAINS_DIR/<domain>/<domain-relative-path>

git hash-object returns the blob SHA of the file's current bytes — modified-but-uncommitted edits produce a new SHA, so changes are detected without requiring a commit. Use "untracked" only if git hash-object itself fails.

For files that will be classified as REINDEX (see Step 4), also score them now:

xlator rate-md-quality "$DOMAINS_DIR/<domain>/<domain-relative-path>"

Store the JSON result alongside the SHA.

For files that will be classified as SKIP (SHA unchanged), use the md_quality: value already stored in the prior index. If the prior index has no md_quality: for a SKIP file (e.g., it was indexed before this feature was added), score it now using the same command.

Step 4: Classify files

Compare current files + SHAs against the stored files: block:

Condition	Classification
File in index, SHA unchanged	SKIP — preserve existing per-file outputs and md_quality
File in index, SHA changed	REINDEX — re-run any per-file action whose manifest is stale
File not in index (new)	REINDEX — run all applicable per-file actions
REINDEX file with score < 40	REJECTED — move to input/rejected/; no per-file outputs
File in index but not on disk (deleted)	REMOVE — drop from files:; the per-action manifests' to_delete entries already handled the mirror-deletes via Pre-step B

:::progress Found document(s): changed/new, unchanged, deleted. Re-indexing file(s)... :::

Step 5: Dispatch unified per-file workers

Same shape as CREATE Step 5. Filter the unified action map (built in Pre-step B) to remove REJECTED sources, sort by source path, dispatch K parallel subagent workers per batch, verify markers post-batch, emit per-action :::progress lines.

REJECTED files (score < 40) are skipped — they produce no per-file outputs. SKIP files are not dispatched — Pre-step B's per-action manifests already classified them as noop for both compress and extract, and their existing per-file outputs remain untouched.

For each REJECTED file, emit:

:::progress Skipped (low quality, score=): input/policy_docs/.md :::

Step 6: Write the updated input-index.yaml (files block only)

Write the following to $DOMAINS_DIR/<domain>/policy_facets/input-index.yaml (overwrites the existing file):

# Auto-generated by /index-inputs — do not edit manually
# Domain: <domain>
files:
  <all current files with their current SHAs and md_quality>

Apply the same YAML conventions as CREATE mode Step 3:

• Quote paths with spaces; sha: as quoted strings; md_quality: sub-block with unquoted score:; flags: omitted when empty
• Write each file's entry one at a time (incremental write)

Step 7: Move rejected files

Run the same rejection script used in CREATE mode:

xlator reject-low-quality-inputs "$DOMAINS_DIR/<domain>" 40

Capture <R> from the JSON result for the completion message.

Step 8: Finalize per-action work

Same shape as CREATE Step 6: run BOTH xlator compress-inputs <domain> --finalize and xlator extract-computations <domain> --finalize, emit each summary in its own :::important block as it completes, list any failed/aborted source paths. Exit non-zero if either finalize reports any failure or aborted entry.

Step 9: Completion

After Step 8 completes, print:

:::important ✓ policy_facets/input-index.yaml updated (files block). policy_facets/compressed/ updated ( file(s) re-compressed). policy_facets/computations/ updated ( file(s) re-extracted, skipped, removed). file(s) rejected (low quality) and moved to input/rejected/ — re-run after reformatting. :::

Omit the "rejected" line if R = 0.

:::next_step Next step: /refine-guidance <domain> to set extraction goals and ruleset guidance :::

---

Common Mistakes to Avoid

• Don't manually edit policy_facets/computations/<rel>.md.yaml or policy_facets/compressed/<rel>.md — they are derived. Edit the source under input/policy_docs/ and re-run /index-inputs.
• Don't skip xlator <tool> --finalize — without it, the per-action manifest is not updated and the next run re-runs every per-file action.
• Don't use absolute paths in the index — files: keys must start with input/policy_docs/, not /Users/... or $DOMAINS_DIR/<domain>/.
• Don't run the rejection script before the files block is fully written — reject-low-quality-inputs reads policy_facets/input-index.yaml for scores; run it only after CREATE Step 3 / UPDATE Step 6 is complete.
• Don't re-dispatch unchanged files in UPDATE mode — Pre-step B's per-action manifests already classify SKIP files as noop; touching their per-file outputs burns AI tokens for no gain.
• Do quote files: keys that contain spaces — YAML keys with spaces must be quoted strings.
• Don't emit flags: [] for files with no quality flags — omit the flags: line entirely; score: alone is sufficient.
• Don't skip md_quality scoring — every file in the files: block must have an md_quality: entry.
• Don't dispatch workers for REJECTED files — they're moved to input/rejected/ before Step 5; no per-file outputs are generated for them.
• Don't abort the whole run on a per-action failure — workers continue past per-action errors within a single file (the in_progress → failed marker captures the failure), and the orchestrator continues to the next batch. Surface the failures in the per-action --finalize summaries and exit non-zero so the analyst can address them.
• Don't forget the XLATOR_AI_CONCURRENCY env var — it's the only rate-limit defense in v1. If 429 errors surface, lower it (or set to 1) and re-run.
• Don't dispatch all sources in one giant turn — the worker prompt is designed for bounded batches of K parallel workers. A single-turn dispatch of N=100 workers risks Anthropic-tier limits AND parallel-tool-call ceilings; bounded batches are documented and predictable.
• Don't rely on the worker writing the marker without verifying — after each batch returns, check the marker dir for each (src, action) it dispatched. A missing or in_progress marker after the worker reported succeeded: is the worker-LLM-obedience failure mode, surfaced as an aborted action by --finalize.