| name | harness-audit |
| description | Use when auditing an AI-agent harness (CLAUDE.md / AGENTS.md / SKILL.md / rules files) for drift — dead links, stale model names, tool mismatches, and cross-document conflicts — then repairing only independently verified findings. Deadline-bounded, parallel-census based, with built-in guards against audit overrun. Especially useful right after a new model release, when official prompting guidance changes underneath your harness. |
/harness-audit — census, verify, repair your harness documents
Origin: a real overnight audit (2026-07-25; a single run, one harness, 252 documents) — full census → 52 suspected defects → independent re-verification → 49 confirmed → all repaired (one commit spanning 47 files + 2 sibling fixes). These aggregates carry their scope label deliberately — per-document verdicts are the reliable unit (STEP 2). The seven guards below were reverse-engineered from that night's inefficiencies (over-verification, purpose drift, no deadline), so that running this skill never costs what the first run did.
When to run this
- A new model just shipped. Model vendors now publish model-specific prompting guidance (e.g. Anthropic's Prompting Claude Opus 5, released with the model — see the dated source in this repo's README). Every release quietly invalidates some of your harness: model names pinned in docs, thresholds benchmarked on the old generation, instructions the new model no longer needs — or newly needs. This skill is the systematic "does my harness still tell the truth?" pass.
- Quarterly maintenance, or whenever a doc's claim visibly disagrees with reality.
- Before publishing/porting your harness anywhere else.
Fixed deliverables (declared at start — guard ②)
Exactly three outputs. Declare them up front; wanting to produce anything else mid-run is your drift alarm.
- Census table — every document × 4 axes, verdict
OK / FINDING / UNREADABLE
- Repair commit(s) — CONFIRMED findings only, applied centrally
- Disposition list — one line each for what a text edit can't fix (structural calls, files you don't own, deferred judgment)
The seven guards
| # | Guard | Rule |
|---|
| ① | Deadline required | Ask for an end time before starting (refuse to start open-ended). Reaching it = report what exists. Extensions are the user's call, not yours |
| ② | Deliverable frame is fixed | The three outputs above, nothing else — "while I'm at it" is how audits eat the night |
| ③ | Verification rounds capped at 2 | Re-verifying or re-wording the same target more than twice = escalate to the user instead |
| ④ | Label fixes only when behavior changes | Wording/label defects in files you don't own: intervene only if an agent would act differently; otherwise record and move on |
| ⑤ | Parallel by default, workers judge-only | 30+ documents = parallel worker census (not sequential reading). Workers never edit anything |
| ⑥ | Measure capacity before fan-out | Check the machine's concurrent capacity (panes/process headroom) before spawning workers; verify leftover cleanup after |
| ⑦ | Runtime checks are opt-in | Document truthfulness is the default scope. Executing skills to verify them is a separate, costlier stage — offer it as an upgrade question and run it only if the user says yes |
Procedure
STEP 0 — Deadline & frame (guards ①②)
Ask for: the deadline, and the target scope (default: project + user-home CLAUDE.md / AGENTS.md / SKILL.md / rules files). Declare the three deliverables.
STEP 1 — Fix the denominator first
Build an independent file list with find before scanning — never count while scanning (if existence becomes the entry condition, absences silently fall out of the denominator). Save the list as a JSON array.
STEP 2 — Four-axis census (guards ⑤⑥)
Parallel workers (~12 docs each) judge every document on four axes:
- tool-mismatch — do the tools/paths/servers the doc references exist in the current environment?
- stale — are model names, versions, dates, and retired-tool references current?
- dead-link — do relative references (
reference/, scripts/, examples/) resolve?
- conflict-dup — does it contradict another loaded document?
Worker schema: {path, verdict: OK|FINDING|UNREADABLE, axis, finding}. No minimizers in the prompt ("only report serious ones" shrinks output literally) — collect everything, filter downstream.
Include a unit-definition example in the worker prompt. Without "here is what counts as one finding", extraction criteria diverge per worker and aggregate counts don't reproduce (measured: verdict agreement 86–88%, extraction agreement 26–41%). Trust per-document verdicts; never quote aggregate counts standalone.
STEP 3 — Independent re-verification + fix proposals (two-stage)
Feed FINDINGs to different workers who re-verify from scratch: {verdict: CONFIRMED|REFUTED|UNCERTAIN, evidence (one measured line), fix_type: edit|policy|none, old_string?, new_string?}. Workers stay read-only; old_string must be verbatim and uniquely matchable. Don't punish UNCERTAIN — a wrong fix is worse than no fix. (In the origin run this stage overturned 3 of 52 first-pass findings.)
STEP 4 — Central batch apply
Apply CONFIRMED+edit through one applier (replace only when the match count is exactly 1; report failures) → stage files by name → commit. policy items go to the disposition list (banners, retirement notices, owner notifications — prefer reversible moves).
STEP 5 — Report
Census table (per-group accounting + findings) + repair commit hashes + disposition list + THREATS (why not to over-trust this result: skim-level verdicts, sample verification rate, denominator scope). If the deadline arrives first, report what exists.
STEP 6 (opt-in, guard ⑦) — Runtime verification of top skills
Only with explicit user consent: pick the N most-used skills and run non-destructive smokes (--help, --dry-run, status modes, dependency version checks) into a PASS/FAIL table. Never execute destructive paths (paid APIs, outbound sends, deletions).
Safety boundaries
- Workers never modify files — all edits are applied centrally by the main agent.
- Files owned by someone else (another agent, another person): notify via the disposition list instead of editing.
- In shared trees,
git add by explicit filename only (never bulk-add).
- Process cleanup only after per-process tree inspection — never kill by name pattern (the origin run nearly killed three live agent sessions that matched a worker-looking pattern).
Composes with
- The gates in this repo (
hooks/): the audit repairs what documents say; the gates enforce what agents do. A repaired rule that never fires is decoration — wire it; a firing gate built on a stale rule enforces yesterday's truth — audit it. Run both.
- A planning pass first (Superpowers brainstorming, Ouroboros interview, or any spec tool): decide scope/deadline/exclusions before touching ~250 documents — the audit's guards assume a plan exists.
/doctor (Claude Code built-in): doctor checks the installation (settings parse, duplicate installs, unused extensions); harness-audit checks the content (whether your documents tell the truth). Different layers, same morning.
