pressure-testing-skills

# Pressure-Testing Skills — amd-smi Empirically prove a skill (or prompt/rule/checklist) works by making a fresh subagent follow it **literally** on a fixture whose answers you already know, then iterate on the failures until the run is deterministic. Reading a skill tells you it's plausible. Only a literal run by a naive agent tells you it's correct. **Iron Law: A SKILL IS NOT DONE UNTIL A FRESH SUBAGENT, FOLLOWING IT VERBATIM ON A KNOWN-ANSWER FIXTURE, CATCHES EVERY PLANTED ISSUE WITH ZERO GUESSING AND NO BLOCKING FRICTION — WITHOUT INVENTING STEPS THE SKILL DIDN'T GIVE.** If the agent got the right answer by adding its own check, the skill failed — that check belongs in the skill. If it "ran clean" but missed a planted issue, the skill failed. If it had to guess which value to use, the skill failed. **REQUIRED BACKGROUND:** you must understand `writing-skills` (structure, CSO, token budget) and `dispatching-parallel-agents` (how to run subagents) first. ## Why Literal + Naive + Known-Answer The whole method rests on three constraints — drop any one and the test lies: | Constraint | Drop it and… | |-----------|--------------| | **Literal** — agent runs the documented commands verbatim, invents nothing | A clever agent papers over gaps with its own reasoning; you ship a skill only experts can follow | | **Naive** — fresh subagent, no prior context, no domain memory | Your own knowledge leaks in and the skill looks clearer than it is | | **Known-answer** — fixture with a pre-written answer key | You can't tell "caught everything" from "got lucky / ran clean" | ## The Loop ``` 0. Build the fixture + answer key (the hardest and most important step) 1. Snapshot the skill to a stable temp path (vN) 2. Dispatch a FRESH, correctly-tooled subagent — follow vN verbatim, invent nothing 3. Collect: issues-caught-vs-answer-key AND friction log 4. Fix the ROOT CAUSE of each miss / friction → snapshot v(N+1) 5. Repeat 2–4 until GREEN (stop condition below) 6. Minimize: cut redundancy, re-test after EACH cut, revert any regression ``` ### 0. Fixture + answer key You cannot grade a test with no answer key. Before iterating, write down every issue the fixture contains and the exact evidence that should surface it. - **Best fixture:** a real artifact with known defects (e.g. a `CHANGELOG.md` on `develop` you already audited by hand). Real fixtures expose real ambiguity. - **Answer key:** a list like "F1: entry X is misfiled → commit `abc` merged after pin `def`; F2: `### Fixed` is a disallowed heading; F3: deprecation under the wrong section." Grade every run against it. - A skill with no fixture cannot be pressure-tested. If you can't build one, the skill is probably too vague to be useful — fix that first. - **Prefer the LIVE state; historical replays carry time-anchored traps.** If the skill resolves any bound from "now" (current tags, `origin/develop`, HEAD), a checked-out *past* state gives meaningless results — the bound is anchored to the present, not the fixture's era. In one run, auditing a pre-release commit bounded a section by a pin that didn't exist yet, and the untagged-release fallback (`origin/develop`) pointed at *today's* tip, so the "too new" check could never fire and the audit ran clean while proving nothing. Use the live artifact as the fixture, or freeze the time-relative inputs too — otherwise you test a mirage. - **Grade coverage against what the real task touches, not what the skill looks at.** If the fixture's defects span the whole file but the skill only inspects the top section, a literal run reports "clean" and is structurally blind to most of the answer key. A scope mismatch between skill and task is itself a finding. ### 1. Snapshot to a stable path Copy the skill under test to a temp path (`/tmp/<skill>-test/SKILL-vN.md`) and point the subagent at ONLY that path. Never point it at the file you're editing or an installed copy — a stale `.claude/skills/` copy will silently override your work. Tell the agent explicitly: "ignore any installed copy; use only this path." ### 2. Dispatch a correctly-tooled, fresh subagent - **Tools must match what the skill's commands need.** A skill full of `git`/`gh` shell commands tested by a no-terminal agent proves nothing — it will only catch surface issues (headings, wording) and give false confidence on the real logic. Use a terminal-capable general subagent for shell-based skills. - **Fresh context every iteration** so prior runs don't inflate apparent clarity. - Give it the fixture setup, the snapshot path, and the literal-follow rules. ### 3. Collect two distinct signals Grade every run on both — they fail independently: | Signal | Question | A gap here means | |--------|----------|------------------| | **Coverage** | Did it catch every answer-key issue, using only documented commands? | The skill is missing a check → add it | | **Determinism** | Did it have to guess, interpret, or invent anything? | The skill is ambiguous → spell it out | "Ran clean with no errors" is **not** success. A command can exit 0, leak a swallowed traceback, and still produce a plausible-but-wrong bound. Ask for the friction log: every command that failed, was ambiguous, leaked an error, or forced a guess — quoted with its **exact command and actual output**. The friction log, not the verdict, is what drives the next iteration. **Absence of output is not proof.** A clean pass and a silently-broken run look identical — both print nothing. A check that greps a section and finds no problems, and a check that ran on *zero* input (wrong path, empty range, swallowed error), are indistinguishable from the outside. Make the skill emit a positive count ("N items checked, 0 flagged"), and have the test agent confirm the check ran on real input, not an empty set, before trusting a clean result. ### 4. Fix root causes, not symptoms Each friction item points at a real defect. Fix the cause: | Real example from a changelog-skill loop | Root-cause fix | |------------------------------------------|----------------| | Audit loop checked only the lower bound → false "all clear" | Check both bounds of the range | | `<V>`=`7.14.0` built tag `therock-7.14.0` (404) → silent `origin/develop` fallback | Document that tags drop the patch component | | `2>/dev/null` on `gh` didn't cover the `python3` in the pipe → leaked traceback | Move error handling to cover the whole pipe | | Agent doubted a correct verdict and nearly overrode it | Add the *why* so a skeptic trusts it (persuasion, not just rules) | | A clean placement audit and one that ran on zero commits looked identical | Emit a positive count ("N checked, 0 flagged") so a real pass is distinguishable | | Skill inspected only the top section; the task spanned the whole file | Match the skill's scope to the task, or make the scope an explicit choice | | Every rule was prose; the agent had to hand-derive a regex for each check | Give an enforced rule a copy-paste command/regex, not just a description | | A shortcut grep covered 5 of the 7 layers the doc listed, but read as complete | Make the shortcut cover every item the doc enumerates, or name what it skips | | "When X, do Y" with X ("changes to the public API") undefined → agent over-applied it | Define the trigger precisely; give the boundary cases (function vs enum vs comment) | | Fixture body was truncated with `...`, so whole body-rule classes couldn't be graded | Give the agent the COMPLETE artifact, never a paraphrase or excerpt | Silent-failure traps (`|| fallback`, `2>/dev/null`) are the most dangerous: they turn a wrong answer into a confident one. Make failures loud or document them. ### 5. Stop condition (loop for REAL improvements, optimally) Stop when a literal run **catches every answer-key issue, with zero guessing, no blocking friction, and no invented steps.** Not before — "ran clean" is a trap. Not after — once it's deterministic and complete, more iterations just gold-plate. Each iteration must fix a concrete failure a subagent actually hit; if you can't name the failure, stop. ### 6. Minimize last, and guard against regressions Only after GREEN, cut for size — and **re-test after every cut**, because a cut or a "cleaner" rewrite can regress: - Cut redundant tables, checklists, and restated conventions. - Self-verify cheap mechanical changes locally (run the new regex/command on the fixture) before spending a subagent. - **A proven heuristic beats theoretical completeness.** In one loop a "more correct" regex added false positives on the real fixture; the terse original had zero. Reverting was the fix. Validate every change against the fixture; revert anything that regresses, even if it looks smarter. ## Test-Agent Prompt Template Reuse this skeleton for every iteration (fill the bracketed parts): ``` Follow this skill LITERALLY and run its commands verbatim. Do NOT use prior knowledge of [domain]. Do NOT add checks the skill doesn't instruct. Skill under test (ONLY source of truth): /tmp/<skill>-test/SKILL-vN.md Ignore any copy at .claude/skills/... — use only the path above. Setup: [worktree / fixture commands] Task: [what to audit/produce using only the skill's documented procedure] Report back: 1. Judgment calls — did you have to guess or interpret ANYTHING? (KEY QUESTION) 2. Every issue found: exact text, responsible commit/line, which skill step flagged it, the fix. 3. Friction log — every command that failed, leaked an error, produced wrong output, or was ambiguous. Quote exact command + actual output. Say so explicitly if it ran clean with zero guessing. (MOST IMPORTANT) 4. Verbatim key outputs. 5. Exact commands run, trimmed. Then clean up: [worktree remove]. Read-only. Do NOT edit the fixture or the skill. ``` The "KEY QUESTION" and "MOST IMPORTANT" labels matter — without them agents report a tidy verdict and bury the friction that you actually need. ## Common Mistakes | Mistake | Why it breaks the test | |---------|------------------------| | Testing a shell-heavy skill with a no-terminal agent | Only surface issues surface; core logic untested → false pass | | No fixture / no answer key | Can't distinguish "caught everything" from "ran clean" | | Pointing the agent at the file you're editing or an installed copy | Stale copy silently overrides; you test the wrong text | | Accepting the verdict, skipping the friction log | You miss the ambiguities that make the next reader fail | | Treating "exit 0 / no errors" as success | Swallowed failures produce confident wrong answers | | Letting the agent invent its own checks and calling it a pass | The check isn't in the skill — the next agent won't have it | | Minimizing before it's correct, or not re-testing each cut | You shrink a broken skill, or a cut regresses silently | | Looping forever / gold-plating after GREEN | Diminishing returns; stop when deterministic and complete | | Using a historical/checked-out-past fixture for a skill with "now"-anchored bounds | Time-relative inputs resolve to the present → clean-but-meaningless run | | Trusting empty output as a pass | Clean pass and silently-broken run are indistinguishable without a positive count | | Grading only what the skill inspects, not what the task touches | Scope mismatch hides most of the answer key behind a "clean" verdict | | Feeding the agent a truncated or paraphrased fixture | Rules that need the full artifact (body wording, trailing lines) can't be graded | | Assuming a stated rule is enforceable without a command | Prose-only rules force the agent to invent a check; two agents check differently | | Trusting a shortcut check that lists fewer items than the doc | It reads as complete while silently skipping layers the doc enumerated |