docs-corpus-audit

name

docs-corpus-audit

description

Use this skill to run a holistic regrounding pass on the entire microsoft/apm documentation corpus against current source code, page-by-page, and emit surgical fixes for stale claims. Activate when the maintainer wants a WHOLE-CORPUS audit (not per-PR review) -- typical triggers include "audit the docs", "reground the corpus", "check every page against code", "pre-release docs sweep", "the docs have drifted everywhere", or "we just reshaped the TOC, find dead links". Wave-batched and S7-verified; scales to the full ~112-page corpus in ~10 minutes wall-time. This is a SIBLING to docs-sync, not a replacement: docs-sync is per-PR (triggered by a diff); this skill is per-corpus (triggered by a maintainer ask). They share agent personas, schemas, and the docs index, but their triggers MUST NOT collide. Does NOT auto-merge, does NOT push without maintainer review, and does NOT replace per-PR drift detection.

docs-corpus-audit -- whole-corpus regrounding pass

The docs corpus drifts silently between releases. docs-sync catches drift introduced by individual PRs at PR-open time. This skill catches the accumulated drift that slips past per-PR review -- stale flag names, dead nav links from past IA reshuffles, deprecation banners that outlived their version targets, factual claims whose source-side truth has moved.

The pattern is A1 PANEL + WAVE EXECUTION + S7 DETERMINISTIC TOOL BRIDGE + A8 ALIGNMENT LOOP + A9 SUPERVISED EXECUTION. The corpus is split into disjoint page scopes; one verifier subagent owns each scope; agents extract factual claims, S7-verify against source, apply surgical fixes inline. The orchestrator then runs an alignment-loop pass to re-verify that applied edits actually ground out true.

This skill is ADVISORY but ACTIONABLE: agents apply edits inline on a working branch. The orchestrator is the sole writer to git -- stages, commits, pushes. Maintainer reviews the resulting PR.

Sibling contract with docs-sync

These two skills share substrate. Be explicit:

Shared resource	Owner	Both use
.apm/docs-index.yml (corpus map)	docs-sync	yes
doc-writer persona	shared	yes (per-page edits)
python-architect persona	shared	yes (S7 verification)
editorial-owner persona	shared	optional (voice pass at scale)
cdo persona	shared	yes (final synthesis)
assets/panelist-return-schema.json	docs-sync (mirrored)	yes

Trigger boundary (avoid DISPATCH COLLISION):

• docs-sync triggers on a PR event ("PR opened/synchronized", source-diff-driven).
• docs-corpus-audit triggers on a maintainer ask for a WHOLE-CORPUS pass ("audit the corpus", "reground", "pre-release sweep") -- no PR required, no diff required, the whole corpus is the input.

If a maintainer asks "review this PR's doc impact", route to docs-sync. If they ask "audit all our docs" or "the docs feel stale everywhere", route here.

Architecture invariants

• Wave-batched, not flat. Pages are partitioned into 6-8 disjoint scopes; each scope is one verifier subagent. Cost scales with wave size, not corpus size. A wave of 6 agents on ~10 pages each is the canonical shape.
• Disjoint page ownership. Each subagent has EDIT AUTHORITY on its scope only. No two agents touch the same file -- guarantees no merge conflicts during fan-in.
• S7 verification is mandatory. Every factual claim is verified against deterministic source: uv run apm <verb> --help for CLI, grep -n src/apm_cli/ for symbols, python -c "import ..." for module shape, file-existence checks for nav links. Never assert from LLM recall.
• Surgical edits only. 1-3 line patches per drift, preserving voice. Restructuring is deferred to the orchestrator post-pass, never auto-applied by per-scope agents.
• Single-writer interlock for git. Subagents NEVER run git commit, git push, or gh pr <write>. Orchestrator commits per wave; pushes once per session.
• Alignment loop (A8). After waves return, orchestrator re-greps the corpus for the patterns the agents claimed to fix. Any residue triggers a targeted re-dispatch (max 2 redrafts) or is escalated to maintainer.

Roster (composition, not invention)

Reuse docs-sync's personas. Do NOT invent a one-off "grounding- verifier" role; that's R3 EXTRACT in reverse.

Role	Persona	Always active?
Per-scope verifier+editor	python-architect (S7) and doc-writer (edits), bundled into one subagent prompt per scope	Yes -- one per page scope, parallel fan-out
Cross-corpus post-pass	orchestrator (deterministic greps via scripts/scan-cross-corpus-drift.sh)	Yes -- once after waves return
Alignment-loop checker	orchestrator (deterministic re-grep + targeted re-dispatch)	Yes -- once after post-pass
Voice pass (optional)	editorial-owner	Only when >20 edits to keep tone coherent
Final synthesis	cdo	Once, for the PR summary comment

The per-scope subagent prompt that composes python-architect + doc-writer is in assets/subagent-prompt-template.md -- the orchestrator substitutes scope + working dir + branch and dispatches via the task tool.

Process

1. PROBE (A9 SUPERVISED EXECUTION)
   - Check working tree: docs/src/content/docs/ exists?
   - Check working tree: packages/apm-guide/.apm/skills/apm-usage/
     exists? (Rule-4 backfill target. If missing, the audit cannot
     close Rule 4; ask maintainer before continuing.)
   - Check `.apm/docs-index.yml` reachable.
   - Verify on a working branch (not main).

2. RISK-TRIAGE (orchestrator, ~1 LLM call)
   - Read .apm/docs-index.yml only (NOT the corpus body).
   - Bucket pages by drift risk: HIGH (CLI ref, schemas, consumer
     flows), MEDIUM (producer, enterprise policy), LOW (concepts,
     contributing, troubleshooting, integrations).
   - Decide wave order: HIGH first, MEDIUM next, LOW last.

3. WAVE-PLANNER (orchestrator, deterministic)
   - Partition pages into 6-8 disjoint scopes per wave.
   - Each agent gets ~9 pages, mixed surface types.

4. WAVE EXECUTION (parallel, one subagent per scope)
   - Orchestrator dispatches one task per scope using the prompt
     template in assets/subagent-prompt-template.md.
   - Subagents read pages, extract claims, S7-verify, apply
     surgical edits, return JSON per the docs-sync panelist
     schema (mirrored at assets/panelist-return-schema.json).
   - Validate every return against the schema; reject malformed
     JSON.

5. CROSS-CORPUS POST-PASS (orchestrator, deterministic)
   - Run scripts/scan-cross-corpus-drift.sh to grep for patterns
     a per-scope agent cannot see (IA-reshuffle dead links, stale
     deprecation version targets, phantom flag references).
   - Patch residue inline.

6. ALIGNMENT LOOP (orchestrator, deterministic)
   - Re-run scripts/scan-cross-corpus-drift.sh.
   - Re-grep for claims the agents marked DRIFTED-FIXED.
   - If residue: targeted re-dispatch to the owning agent
     (bounded: max 2 redrafts per wave).

7. COMMIT + PUSH (orchestrator, single writer)
   - One commit per wave; structured message naming closed items.
   - Push to working branch.

8. PR + SUMMARY COMMENT (orchestrator)
   - If no PR exists: open one with the [pr-description-skill]
     (../pr-description-skill/SKILL.md).
   - Post per-wave summary comment: pages audited, drift caught,
     fixes applied, items deferred, alignment-loop residue.

Bundled assets

• assets/subagent-prompt-template.md -- the per-scope prompt the orchestrator substitutes and dispatches. Composes python-architect (S7) + doc-writer (surgical edit). Loaded once per scope.
• assets/panelist-return-schema.json -- subagent return schema, mirrored from docs-sync. Loaded once at wave start; validated against every return.
• scripts/scan-cross-corpus-drift.sh -- deterministic grep sweep for cross-corpus patterns (IA dead links, stale deprecation targets, phantom flags). Non-interactive; emits structured matches on stdout, diagnostics on stderr. Run --help for pattern list. Update this script after each major IA reshuffle.

Cost model

Wave size	Pages	Subagents	LLM dispatches	Wall time
Small	~30	4	~5	~3 min
Medium (default)	~55	6	~7	~5 min
Large	~110 (full corpus)	12 (two medium waves)	~14	~10 min

Compared to docs-sync (15-call flat ceiling), this skill scales as O(waves), not O(claims), because per-agent work fits in one context window. S7 verification dominates wall-time, not LLM cost.

Boundary (what this skill does NOT do)

• Per-PR doc-impact review -- use docs-sync.
• Single-page typo or copy edit -- direct edit is faster.
• Writing docs for a brand-new feature -- use docs-impact-architect and doc-writer directly.
• Auto-merging or pushing without maintainer review.
• Reviewing code quality, security, or test coverage (out of scope).

Evals

See evals/:

• evals/content-evals.json -- 3 corpus snapshots with seeded drift (stale CLI flag, dead nav link, expired deprecation target); expected behavior is that the skill catches all three and applies surgical fixes that ground out true on re-verification.
• evals/trigger-evals.json -- 10 should-trigger + 10 should-NOT- trigger queries, 60/40 train/val. The val split is the ship gate (>=0.5 should-trigger AND <0.5 should-not-trigger).
• evals/README.md -- how to run.

Provenance

This skill was extracted from a real session that audited the microsoft/apm corpus across 3 waves (PR #1511, 2026-05-27): 112/112 pages audited, 49 surgical fixes, ~25 LLM dispatches, ~30 min wall-time. The session design artifact (genesis hand-off packet) lives in session state, not in this bundle (maintainer- scope, not runtime-loaded).