| name | meta-audit |
| description | Evaluates the accuracy and scope coverage of the other audit skills (perf-audit, test-audit, doc-audit, wiki-audit, dependency-audit) by re-verifying past findings against source and checking for scope drift, then safely improves their SKILL.md files. Use when asked to evaluate audit skill performance, or auto-triggered by a domain audit skill after its 5th run since the last evaluation. |
Meta-Audit: Evaluating the Audit Skills
Re-verifies past findings from the five domain audit skills against source (as of the
commit each finding was made against) and checks whether each skill's declared scope has
fallen behind the current codebase. Auto-fixes safe improvements; flags anything that
would change audit strategy.
When to Use
- Auto-triggered: a domain audit skill's
## Log This Run step reports TRIGGER from
scripts/audit-eval-check.sh <skill-name> (5 runs accumulated since the last evaluation)
- On-demand: "evaluate wiki-audit", "how are the audits doing" (runs all five independently)
Inputs
One or more target skill names: perf-audit, test-audit, doc-audit, wiki-audit,
dependency-audit. When invoked for "all five," repeat the full flow below independently
per skill (they don't share state or need to run in a particular order).
Phase 1: Load History
Read .claude/skills/meta-audit/history/<skill-name>.jsonl. Take every "type": "run"
entry after the last "type": "eval_marker" entry (or every run entry if no marker
exists yet). If there are zero run entries to evaluate, stop and report "nothing to
evaluate yet."
Phase 2: Adversarial Re-Verification (Parallel Agents, Read-Only)
Collect every finding across the loaded runs (cap at the 20 most recent if there are
more — note in the final report how many were skipped and why).
For each finding, spawn one fresh Explore agent given only its location and claim —
never its logged correct_value. If location names multiple files, the agent should verify the claim against all of them. Task: independently derive what the correct value
actually was as of that finding's commit_sha, reading source read-only via
git show <commit_sha>:<path> only. Never run git checkout, git switch, git reset,
git stash, or edit any file — nothing in this phase may change what's in the working tree.
location identifies where the wrong claim was written, not necessarily where the truth lives. For wiki-audit findings, location is a page inside the quest.wiki submodule — a separate git history not reachable via git show <commit_sha>:<path> against the main repo (confirmed: this always fails with "path does not exist"). In that case, and generally whenever location isn't a path that resolves in the main repo, derive the correct value from whatever main-repo source file the claim or correct_value text names or implies — search for it if it isn't named explicitly.
The agent reports its own independently-derived value plus the source location it based that on.
Diff the agent's answer against the logged correct_value. A mismatch is a confirmed
inaccuracy in the original audit — record it as {finding, original_correct_value, re_derived_value, agent_citation}.
Phase 3: Scope-Staleness Check
Only for skills with a <!-- meta-audit scope-universe: ... --> comment near the top of
their SKILL.md (currently: wiki-audit, doc-audit, test-audit, perf-audit —
dependency-audit has none, and this phase is a no-op for it).
- Read the scope-universe comment and enumerate what it currently matches in the repo
(e.g.
ls quest.wiki/*.md, ls src/*/, find tests -maxdepth 1 -type d -name '*_tests').
- Read the target skill's
## Phase 1 section and collect every Scope: line across
all its agents into one set.
- Diff the two sets:
- Scope gap — in the universe but not covered by any agent's
Scope: line.
- Scope rot — named in a
Scope: line but doesn't exist in the universe anymore.
Phase 4: Synthesize
Group confirmed inaccuracies (Phase 2) by category. A category that recurs more than
once across the evaluated runs is a stronger signal than a one-off — call this out
explicitly in the report, since it justifies adding a guardrail rather than shrugging
off a single mistake.
Rank scope gaps/rot (Phase 3) — a gap that's existed across multiple backfilled runs
(the file/module existed before several of the evaluated runs and was still never
picked up) ranks higher than one that just appeared.
Phase 5: Fix
Auto-fix (no user approval needed)
- Add a missing path to the correct agent's
Scope: line (scope gap)
- Remove a path from a
Scope: line that no longer exists (scope rot)
- Append one row to the skill's anti-patterns/findings table documenting a confirmed,
recurring false-positive category, with a one-line guardrail (e.g. "verify a
mechanic from its source implementation, not its name — see history for a past miss")
Flag for user review
- Any change to a skill's agent count or Phase structure
- Any change to what a skill treats as auto-fixable vs. flag-for-review
- A confirmed inaccuracy that isn't part of a recurring pattern (a one-off mistake
doesn't justify rewriting the skill — report it, but don't act on it)
Phase 6: Ship
The eval_marker needs the PR's own URL, which doesn't exist until the PR is created —
so commit the fix(es) and open the PR first, then append the marker as a follow-up
commit onto that same still-open branch before it merges:
- Branch, commit the
SKILL.md fix(es) and the updated history log, open a PR (same
pattern as every other audit skill — no direct commits to main). PR body includes
the full report: confirmed inaccuracies (with citations), scope gaps/rot, what was
auto-fixed, what's flagged for review.
- Now that the PR exists, append the
eval_marker entry with its real URL:
echo '{"type":"eval_marker","date":"<today>","pr_url":"<the PR URL from step 1>","runs_covered":<n>}' > /tmp/marker.json
scripts/audit-eval-log.sh <skill-name> /tmp/marker.json
- Commit and push this marker addition as one more commit on the same PR branch
(multiple commits on one PR are normal — squash-merge collapses them at merge time).
Output
Report the PR URL and final status when done (use /ship skill).
