| name | pressure-testing-skills |
| description | Use when validating or hardening a SKILL.md, prompt, rule, or any agent-followed document — when you need proof it actually works, not just that it reads well. Runs a fresh subagent that follows the doc literally on a known-answer fixture, captures friction, and iterates to determinism before minimizing. Use after writing-skills produces a draft, or whenever a skill 'looks right' but hasn't been proven under an agent. |
Pressure-Testing Skills — amd-smi
Empirically prove a skill (or prompt/rule/checklist) works by making a fresh
subagent follow it literally on a fixture whose answers you already know, then
iterate on the failures until the run is deterministic. Reading a skill tells you
it's plausible. Only a literal run by a naive agent tells you it's correct.
Iron Law: A SKILL IS NOT DONE UNTIL A FRESH SUBAGENT, FOLLOWING IT VERBATIM ON
A KNOWN-ANSWER FIXTURE, CATCHES EVERY PLANTED ISSUE WITH ZERO GUESSING AND NO
BLOCKING FRICTION — WITHOUT INVENTING STEPS THE SKILL DIDN'T GIVE.
If the agent got the right answer by adding its own check, the skill failed —
that check belongs in the skill. If it "ran clean" but missed a planted issue,
the skill failed. If it had to guess which value to use, the skill failed.
REQUIRED BACKGROUND: you must understand writing-skills (structure, CSO,
token budget) and dispatching-parallel-agents (how to run subagents) first.
Why Literal + Naive + Known-Answer
The whole method rests on three constraints — drop any one and the test lies:
| Constraint | Drop it and… |
|---|
| Literal — agent runs the documented commands verbatim, invents nothing | A clever agent papers over gaps with its own reasoning; you ship a skill only experts can follow |
| Naive — fresh subagent, no prior context, no domain memory | Your own knowledge leaks in and the skill looks clearer than it is |
| Known-answer — fixture with a pre-written answer key | You can't tell "caught everything" from "got lucky / ran clean" |
The Loop
0. Build the fixture + answer key (the hardest and most important step)
1. Snapshot the skill to a stable temp path (vN)
2. Dispatch a FRESH, correctly-tooled subagent — follow vN verbatim, invent nothing
3. Collect: issues-caught-vs-answer-key AND friction log
4. Fix the ROOT CAUSE of each miss / friction → snapshot v(N+1)
5. Repeat 2–4 until GREEN (stop condition below)
6. Minimize: cut redundancy, re-test after EACH cut, revert any regression
0. Fixture + answer key
You cannot grade a test with no answer key. Before iterating, write down every
issue the fixture contains and the exact evidence that should surface it.
- Best fixture: a real artifact with known defects (e.g. a
CHANGELOG.md
on develop you already audited by hand). Real fixtures expose real ambiguity.
- Answer key: a list like "F1: entry X is misfiled → commit
abc merged after
pin def; F2: ### Fixed is a disallowed heading; F3: deprecation under the
wrong section." Grade every run against it.
- A skill with no fixture cannot be pressure-tested. If you can't build one, the
skill is probably too vague to be useful — fix that first.
- Prefer the LIVE state; historical replays carry time-anchored traps. If the
skill resolves any bound from "now" (current tags,
origin/develop, HEAD), a
checked-out past state gives meaningless results — the bound is anchored to the
present, not the fixture's era. In one run, auditing a pre-release commit bounded
a section by a pin that didn't exist yet, and the untagged-release fallback
(origin/develop) pointed at today's tip, so the "too new" check could never
fire and the audit ran clean while proving nothing. Use the live artifact as the
fixture, or freeze the time-relative inputs too — otherwise you test a mirage.
- Grade coverage against what the real task touches, not what the skill looks
at. If the fixture's defects span the whole file but the skill only inspects
the top section, a literal run reports "clean" and is structurally blind to most
of the answer key. A scope mismatch between skill and task is itself a finding.
1. Snapshot to a stable path
Copy the skill under test to a temp path (/tmp/<skill>-test/SKILL-vN.md) and
point the subagent at ONLY that path. Never point it at the file you're editing
or an installed copy — a stale .claude/skills/ copy will silently override your
work. Tell the agent explicitly: "ignore any installed copy; use only this path."
2. Dispatch a correctly-tooled, fresh subagent
- Tools must match what the skill's commands need. A skill full of
git/gh
shell commands tested by a no-terminal agent proves nothing — it will only catch
surface issues (headings, wording) and give false confidence on the real logic.
Use a terminal-capable general subagent for shell-based skills.
- Fresh context every iteration so prior runs don't inflate apparent clarity.
- Give it the fixture setup, the snapshot path, and the literal-follow rules.
3. Collect two distinct signals
Grade every run on both — they fail independently:
| Signal | Question | A gap here means |
|---|
| Coverage | Did it catch every answer-key issue, using only documented commands? | The skill is missing a check → add it |
| Determinism | Did it have to guess, interpret, or invent anything? | The skill is ambiguous → spell it out |
"Ran clean with no errors" is not success. A command can exit 0, leak a
swallowed traceback, and still produce a plausible-but-wrong bound. Ask for the
friction log: every command that failed, was ambiguous, leaked an error, or forced
a guess — quoted with its exact command and actual output. The friction log,
not the verdict, is what drives the next iteration.
Absence of output is not proof. A clean pass and a silently-broken run look
identical — both print nothing. A check that greps a section and finds no problems,
and a check that ran on zero input (wrong path, empty range, swallowed error),
are indistinguishable from the outside. Make the skill emit a positive count
("N items checked, 0 flagged"), and have the test agent confirm the check ran on
real input, not an empty set, before trusting a clean result.
4. Fix root causes, not symptoms
Each friction item points at a real defect. Fix the cause:
| Real example from a changelog-skill loop | Root-cause fix |
|---|
| Audit loop checked only the lower bound → false "all clear" | Check both bounds of the range |
<V>=7.14.0 built tag therock-7.14.0 (404) → silent origin/develop fallback | Document that tags drop the patch component |
2>/dev/null on gh didn't cover the python3 in the pipe → leaked traceback | Move error handling to cover the whole pipe |
| Agent doubted a correct verdict and nearly overrode it | Add the why so a skeptic trusts it (persuasion, not just rules) |
| A clean placement audit and one that ran on zero commits looked identical | Emit a positive count ("N checked, 0 flagged") so a real pass is distinguishable |
| Skill inspected only the top section; the task spanned the whole file | Match the skill's scope to the task, or make the scope an explicit choice |
| Every rule was prose; the agent had to hand-derive a regex for each check | Give an enforced rule a copy-paste command/regex, not just a description |
| A shortcut grep covered 5 of the 7 layers the doc listed, but read as complete | Make the shortcut cover every item the doc enumerates, or name what it skips |
| "When X, do Y" with X ("changes to the public API") undefined → agent over-applied it | Define the trigger precisely; give the boundary cases (function vs enum vs comment) |
Fixture body was truncated with ..., so whole body-rule classes couldn't be graded | Give the agent the COMPLETE artifact, never a paraphrase or excerpt |
Silent-failure traps (|| fallback, 2>/dev/null) are the most dangerous:
they turn a wrong answer into a confident one. Make failures loud or document them.
5. Stop condition (loop for REAL improvements, optimally)
Stop when a literal run catches every answer-key issue, with zero guessing, no
blocking friction, and no invented steps. Not before — "ran clean" is a trap.
Not after — once it's deterministic and complete, more iterations just gold-plate.
Each iteration must fix a concrete failure a subagent actually hit; if you can't
name the failure, stop.
6. Minimize last, and guard against regressions
Only after GREEN, cut for size — and re-test after every cut, because a cut or
a "cleaner" rewrite can regress:
- Cut redundant tables, checklists, and restated conventions.
- Self-verify cheap mechanical changes locally (run the new regex/command on the
fixture) before spending a subagent.
- A proven heuristic beats theoretical completeness. In one loop a "more
correct" regex added false positives on the real fixture; the terse original had
zero. Reverting was the fix. Validate every change against the fixture; revert
anything that regresses, even if it looks smarter.
Test-Agent Prompt Template
Reuse this skeleton for every iteration (fill the bracketed parts):
Follow this skill LITERALLY and run its commands verbatim. Do NOT use prior
knowledge of [domain]. Do NOT add checks the skill doesn't instruct.
Skill under test (ONLY source of truth): /tmp/<skill>-test/SKILL-vN.md
Ignore any copy at .claude/skills/... — use only the path above.
Setup: [worktree / fixture commands]
Task: [what to audit/produce using only the skill's documented procedure]
Report back:
1. Judgment calls — did you have to guess or interpret ANYTHING? (KEY QUESTION)
2. Every issue found: exact text, responsible commit/line, which skill step
flagged it, the fix.
3. Friction log — every command that failed, leaked an error, produced wrong
output, or was ambiguous. Quote exact command + actual output. Say so
explicitly if it ran clean with zero guessing. (MOST IMPORTANT)
4. Verbatim key outputs.
5. Exact commands run, trimmed. Then clean up: [worktree remove].
Read-only. Do NOT edit the fixture or the skill.
The "KEY QUESTION" and "MOST IMPORTANT" labels matter — without them agents
report a tidy verdict and bury the friction that you actually need.
Common Mistakes
| Mistake | Why it breaks the test |
|---|
| Testing a shell-heavy skill with a no-terminal agent | Only surface issues surface; core logic untested → false pass |
| No fixture / no answer key | Can't distinguish "caught everything" from "ran clean" |
| Pointing the agent at the file you're editing or an installed copy | Stale copy silently overrides; you test the wrong text |
| Accepting the verdict, skipping the friction log | You miss the ambiguities that make the next reader fail |
| Treating "exit 0 / no errors" as success | Swallowed failures produce confident wrong answers |
| Letting the agent invent its own checks and calling it a pass | The check isn't in the skill — the next agent won't have it |
| Minimizing before it's correct, or not re-testing each cut | You shrink a broken skill, or a cut regresses silently |
| Looping forever / gold-plating after GREEN | Diminishing returns; stop when deterministic and complete |
| Using a historical/checked-out-past fixture for a skill with "now"-anchored bounds | Time-relative inputs resolve to the present → clean-but-meaningless run |
| Trusting empty output as a pass | Clean pass and silently-broken run are indistinguishable without a positive count |
| Grading only what the skill inspects, not what the task touches | Scope mismatch hides most of the answer key behind a "clean" verdict |
| Feeding the agent a truncated or paraphrased fixture | Rules that need the full artifact (body wording, trailing lines) can't be graded |
| Assuming a stated rule is enforceable without a command | Prose-only rules force the agent to invent a check; two agents check differently |
| Trusting a shortcut check that lists fewer items than the doc | It reads as complete while silently skipping layers the doc enumerated |
