| name | harness-audit |
| description | Agent-driven, read-only self-audit of the harness — run the machine integrity layer once as a dry-run, present a per-check pass/fail table, cite the P1-1 doc-reality result, and for any failure give root-cause + fix + backlog follow-up. Consumes the machine gates; it does not reimplement them. NOT for auditing a consumer project's own codebase (use that project's test suite), and NOT for applying fixes — this skill observes and reports only. |
| when_to_use | A periodic or on-demand health check of the harness itself — "audit the harness", "/harness-audit", "is the harness passing its own gates", or when you want one interpreted report over the CI/gate layer before a release or after a structural change. |
| tools | Bash, Read, Grep, Glob |
/harness-audit
Goal
Produce one interpreted health report of the harness in a single read-only
dry-run. The machine integrity layer (P1-1 doc-reality.sh, the CI jobs, and the
verify-all.sh runner) is the enforcement; this skill is the consumer that
runs it once, reads off a per-check pass/fail table, and turns each failure into a
diagnosis a maintainer can act on.
Concretely the report answers three questions with evidence:
- Is the harness passing its own gates right now? — a per-check PASS / FAIL /
SKIP table, taken verbatim from
verify-all.sh.
- What does the doc-reality gate (P1-1) say? — cited explicitly, because it is
the gate that catches the harness lying about itself (phantom paths + drifted
counts), and it is the one item most easily missed in a wall of green.
- For every non-PASS: why, and what next? — the check's purpose, the likely
root cause, the recommended fix, and whether it warrants a new backlog row.
This skill supplies the interpretation; it does not re-implement any check.
The checks live in core/tests/ and CI; here you run them and read the results.
Why this is a layer, not a duplicate of P1-1
P1-1 (core/tests/doc-reality.sh) is a machine gate: it runs in CI, exits 0/1,
and mechanically blocks doc drift. This skill is the agent consumer that sits on
top of that gate — it invokes the whole machine layer in one dry-run and interprets
the results (root-cause, fix, backlog follow-up) the way a maintainer would. It reads
doc-reality's verdict; it never re-derives phantom-path or count logic. Deleting this
skill would not weaken any gate (CI still enforces them); deleting doc-reality would.
That asymmetry is the layer relationship: the gate enforces, the audit interprets.
This mirrors the machine-gate-vs-agent-consumer split the plan draws for H-2.
Steps
1. Run the machine integrity layer (one dry-run)
bash core/tests/verify-all.sh
verify-all.sh is the single-command runner. It dynamically discovers every
core/tests/*.sh check (the runner and its own self-test aside, to avoid recursion —
that self-test runs as its own CI step) so no harness check is silently omitted, and
runs, per check, PASS / FAIL / loud SKIP with a final tally. Its discovered set
already includes the machine layer this audit reports on:
doc-reality.sh (P1-1) — the doc-drift gate. It also cross-checks the §7
artifact counts, which include the hook file count, so the historical "hook
count" audit item is covered here (there is no separate hook-count step to run).sanitize-audit.sh — domain-neutrality (no prior-project taint).supply-chain-scan.sh — no injection-style directives in shipped files.adapter-parity.sh — the three adapters normalize one event identically.registry-drift.sh — the registry/manifest gate (plugin/marketplace fields,
hooks.json → executable core-hook resolution, agent name: frontmatter, and
registry↔agent model agreement).- every
*-test.sh battery — the per-hook and per-gate fixture tests.
evals:deterministic and evals:semantic — the two eval layers (same
invocation CI uses).gitleaks — the secret scan when present; a loud SKIP (never a silent pass)
when the binary is absent.
Capture the runner's per-check output and its final === verify-all: … === tally.
This is entirely read-only: the runner and every check it dispatches only read the
repo — no files are written, nothing is committed. Note the runner's exit status: a
non-zero exit means the harness is failing its own gates and the report must say
so plainly.
To enumerate the checks without executing them (useful for a coverage sanity-check
of the report), bash core/tests/verify-all.sh --list prints the labels only.
2. Run the runtime layer (doctor + telemetry digest)
verify-all.sh covers the repo; two further read-only probes cover the
runtime this repo is installed into and the live gate behavior:
bash setup.sh --doctor
bash core/infra/telemetry-digest.sh
setup.sh --doctor — environment diagnosis (tool availability, exec bits,
registry/hooks sanity) plus the drift observers: plugin-cache single-version
(a stale cached version re-exposing retired agents/skills), declared-hook
manifest vs live settings reconciliation, and the phantom-command scan (a
runtime commands/*.md invoking a script that does not exist on this
machine). Doctor WARNs are observations, never blockers — report them in the
same diagnosis format as step 4.telemetry-digest.sh — per-gate fire-rate from the local event sinks;
surface any DEAD (never fires) or FATIGUE (high-frequency ask) candidates as
calibration follow-ups, not as failures.
Append the doctor summary line (doctor: N pass, M warn, K fail) and any
DEAD/FATIGUE candidates to the report. If either probe's input is absent on
this machine (no runtime install, empty sinks), say so — an unmeasured layer is
reported as unmeasured, not as green.
3. Present the table (cite P1-1 explicitly)
Read the runner's output and render a per-check table:
check result note
----------------------- ------- -------------------------------------------
doc-reality.sh (P1-1) PASS docs match repo; §7 counts + hook count OK
sanitize-audit.sh PASS no prior-project taint
supply-chain-scan.sh PASS no injection-style directives
adapter-parity.sh PASS 3 adapters agree
registry-drift.sh PASS manifests/hooks/agents/registry agree
… (batteries) … PASS fixture tests green
evals:deterministic PASS labeled dataset, Pass^3
evals:semantic PASS test-meaningfulness judge, Pass^3
gitleaks SKIP binary not installed on this host
Call out the doc-reality (P1-1) line by name — its verdict is the headline of
this audit. State the overall tally (N passed / M failed / K skipped) and the
runner's exit status.
4. Diagnose each non-PASS
For every FAIL, every doctor WARN, and every SKIP that matters (e.g. a skipped
security scan), write:
- Purpose — what the check guards (one line).
- Root cause — the most likely reason it is failing, read from the check's own
failure output (each gate prints the offending item; quote it).
- Fix — the concrete corrective step (edit which doc / ship which file / restore
which exec bit), matching the guidance the gate itself prints.
- Backlog follow-up — whether this warrants a new row in
docs/harness-improvement-plan.md (a recurring or structural failure does; a
one-off local fix does not). Suggest the row; do not add it here.
A SKIP is not a PASS: an absent gitleaks is a coverage gap to flag (install it or
rely on the dedicated CI secret-scan job), not a silent green.
5. Summarize health + next actions
Close with an overall verdict — healthy (all gates green) or failing (one or more
gates red) — and an ordered list of recommended next actions drawn from step 4. If
the harness is failing its own gates, that is the top-line finding.
Completion condition
The audit is complete when a single dry-run has produced (a) a per-check
pass/fail table and (b) an explicit citation of the P1-1 (doc-reality) result.
Anything less (a table with no doc-reality line, or a doc-reality verdict with no
table) is an incomplete audit.
Hard rules
- Read-only. This audit observes; it does not edit, fix, or commit. Diagnosis
and fix suggestions are the output — applying them is a separate, human-directed
step.
- Do not reimplement the checks. The gates in
core/tests/ and CI are the
source of truth. This skill runs them and interprets their output; if a check's
logic is wrong, fix the check (and its battery), not this skill.
- The P1-1 citation is mandatory. doc-reality is the gate that catches the
harness lying about itself; its verdict is always named in the report.
- A non-zero
verify-all.sh exit is a failing harness. Report it as such —
never round a red run up to "mostly green".
