docs-grounding-verifier

# docs-grounding-verifier CLAIM-LEVEL grounding verification. Adapts the RAGAS faithfulness-eval pattern (proven in RAG literature) to docs/code instead of generated- answers/retrieved-context. Source code is the ground truth; docs paragraphs are the candidate text under audit. [python-architect persona](../../personas/python-architect.persona.md) [doc-writer persona](../../personas/doc-writer.persona.md) ## Sibling contract This skill is a SIBLING of `docs-corpus-audit` and `docs-sync`. The boundary is load-bearing: | Skill | Trigger | Scope | Granularity | | ---------------------- | -------------------------------------- | ---------------- | --------------- | | docs-sync | PR opened/synchronized | PR diff only | Page-level | | docs-corpus-audit | Maintainer asks for whole-corpus pass | Entire corpus | Page-level | | **docs-grounding-verifier** | **Verify specific pages factually** | **1..N pages** | **CLAIM-level** | `docs-corpus-audit` invokes this skill in its VERIFY phase on the highest-risk pages of each wave. `docs-sync` can invoke it on the specific pages in a PR diff. The skill is also runnable standalone. ## When to activate - Maintainer says "verify <page> against the code". - An audit wave wants per-claim grounding scores for its highest-risk pages. - A PR review wants to confirm that prose changes are not just plausible but actually consistent with the implementation. - A "fact-check" or "grounding" or "drift hunt" request. ## When NOT to activate - Whole-corpus sweep with no specific page list -> use `docs-corpus-audit`. - PR review with mixed code+docs diff -> use `docs-sync`. - Editorial / tone review -> use `editorial-owner` persona directly. ## Architecture (PIPELINE-of-PANELS) ``` PARENT -> [Stage 1: EXTRACT claims, fan-out PANEL] per page -> LLM extracts atomic factual claims as JSON script: scripts/extract-claims.py -> [Stage 2: RETRIEVE evidence, deterministic S7] per claim -> grep over src/ via keywords + hints script: scripts/retrieve-evidence.sh (NO LLM) -> [Stage 3: JUDGE grounding, adversarial A7] per (claim, evidence) -> LLM rules GROUNDED|PARTIAL|CONTRADICTED|UNSUPPORTED asset: assets/judge-prompt.md -> [Stage 4: SYNTHESIZE] aggregate ungrounded -> doc-writer for fix re-verify after fix (A8 ALIGNMENT LOOP) ``` Stage 2 is the load-bearing design choice: evidence retrieval is DETERMINISTIC (grep + AST hints), not LLM. The judge in Stage 3 can only rule on evidence it actually receives -- it cannot hallucinate support that the retriever did not find. This is the structural guard against the failure mode "the LLM convinces itself the docs match the code." ## Phase 1: SCOPE Input: list of page paths to verify (1..N). If a `risk_class` is attached (e.g. "high-stakes"), prefer it; otherwise treat all as equal. Out-of-scope: - Pages outside `docs/src/content/docs/` or `packages/apm-guide/.apm/skills/apm-usage/`. - Pages with no factual claims (pure editorial / landing). Skip rather than force-extract. ## Phase 2: EXTRACT (parallel) For each page, dispatch ONE claim-extractor agent: - Prompt template: `scripts/extract-claims.py <page>` produces the prompt and embeds the page content. - Returns: JSON `{"page", "claims":[{"id","text","section","keywords", "expected_source_areas"}]}` capped at 15 claims per page. Parallel safe; no shared state between extractors. ## Phase 3: RETRIEVE (deterministic, batched) For each claim, pipe to `scripts/retrieve-evidence.sh`: - Uses keywords + expected_source_areas to grep src/. - Returns one-line JSON: `{"claim_id","claim_text","evidence":[...], "evidence_count"}`. Sequential is fine (grep is fast). No LLM. Diagnostics on stderr, data on stdout. ## Phase 4: JUDGE (parallel) For each (claim, evidence) tuple, dispatch ONE grounding-judge agent: - Load `assets/judge-prompt.md`. - Send the prompt + the tuple. - Returns: JSON verdict per the schema in `judge-prompt.md`. Batching across claims-of-one-page into a single judge call is fine (prompt with all tuples at once). Across pages, fan out. ## Phase 5: SYNTHESIZE Aggregate verdicts. Materialize the report: ``` { "summary": { "pages_verified": N, "claims_total": N, "grounded": N, "partial": N, "contradicted": N, "unsupported": N, "grounding_rate": N/total }, "actionable": [ {"page", "claim", "verdict", "evidence_cited", "fix_suggestion"} ] } ``` CONTRADICTED and PARTIAL are doc-writer work items. UNSUPPORTED is split: if `retrieval_fix_suggestion` is plausible, retry retrieval with the suggested keywords; if still empty, treat as CONTRADICTED. ## Phase 6: ALIGNMENT LOOP (A8) Hand actionable items to doc-writer (one subagent per page). After edits, RE-RUN the pipeline on the same pages. The grounding_rate must MONOTONICALLY INCREASE between iterations or the loop has diverged -- stop and escalate to the operator. ## Ship gate - grounding_rate >= 0.9 on each verified page after the alignment loop. - Every CONTRADICTED claim cited a specific code file:line that disproves it -- not vague "the code doesn't say that". - The eval-runner (see `evals/`) passes on the trigger evals and the content evals before the skill is treated as production-ready. ## Bundled assets - `scripts/extract-claims.py` -- Stage 1 prompt builder. `--help`, `--schema`. - `scripts/retrieve-evidence.sh` -- Stage 2 retriever. Deterministic. `--help`. - `scripts/verify-page.sh` -- end-to-end orchestrator. `--help`. - `assets/judge-prompt.md` -- Stage 3 adversarial judge prompt. - `evals/trigger-evals.json` -- 20 dispatch queries (10 should, 10 shouldn't). - `evals/content-evals.json` -- seeded-drift recall scenarios. - `evals/run-evals.sh` -- the eval-runner that turns JSON into metrics. ## Failure modes guarded against - **Hallucinated grounding**: Stage 2 is deterministic; judge sees only real evidence. - **Adversarial weakness**: Stage 3 prompt defaults to SKEPTICAL. - **Page-level averaging**: claim-level granularity surfaces partials. - **Bundle leakage**: design notes / one-time scripts stay in session state, never in `references/`. - **Phantom dependency**: SKILL.md links its persona deps via relative paths; A9 PROBE before invoking docs-corpus-audit's substrate. - **Dispatch collision** with sibling skills: trigger-eval validation split is the ship gate (must distinguish from docs-sync / docs-corpus-audit triggers).