| name | skill-creator-trigger-eval-gotchas |
| version | 1.1.0 |
| description | Three chained gotchas when running the skill-creator plugin's trigger-eval
machinery (scripts/run_loop.py, scripts/run_eval.py) against a skill. Use
when: (1) the trigger eval reports 0% recall / "skill never triggers" even
on verbatim trigger phrases like the skill's own example phrasings —
if the skill under test is ALREADY INSTALLED in the project, the harness's
synthetic probe (registered as <name>-skill-<uuid> in .claude/commands/) is
shadowed by the real skill: Claude invokes the real name, the harness only
counts the uuid probe, recall reads 0%. Verify with one ground-truth
`claude -p "<trigger phrase>"` grepping the stream for the real skill name;
(2) `python -m scripts.run_loop` fails with ModuleNotFoundError: anthropic —
run via `uv run --with anthropic python -m scripts.run_loop`; (3) the loop
crashes at the improvement step with "Could not resolve authentication
method" — improve_description.py uses the Anthropic SDK directly and needs
ANTHROPIC_API_KEY, which subscription-authenticated Claude Code sessions
don't export; measurement-only (run_eval.py) still works without it;
(4) scores look wrong and you launched from the plugin's directory —
run_eval resolves the project root by walking UP FROM CWD to the first
.claude/ dir, so launching from ~/.claude/plugins/... roots the test at
$HOME, not your project. Launch with cwd = the project, PYTHONPATH = the
skill-creator directory; (5) 0% recall even though the skill was correctly
moved out of .claude/skills/ — SIBLING-probe shadowing: run_eval's default
--num-workers 10 makes each parallel worker write its own <name>-skill-<uuid>.md
probe into the same .claude/commands/, sessions invoke a sibling's uuid and
every worker's own-name check fails. Always pass --num-workers 1;
(6) harness recall looks low (e.g. ~50-60%) while ground-truth `claude -p`
checks trigger the real skill fine — uuid-probe semantic gap; treat harness
recall as a lower bound.
|
skill-creator-trigger-eval-gotchas
Problem
The skill-creator plugin's description-optimization loop has four failure
modes that compound, and the worst one (probe shadowing) produces a
confidently WRONG result instead of an error.
The gotcha chain, in the order you'll hit them
1. ModuleNotFoundError: anthropic
python -m scripts.run_loop needs the anthropic package. Fix:
cd <skill-creator-dir> && uv run --with anthropic python -m scripts.run_loop ...
2. cwd-sensitive project root
run_eval.find_project_root() walks up from cwd to the first .claude/
directory. Launching from the plugin cache dir (~/.claude/plugins/...)
resolves to $HOME — the probe lands in ~/.claude/commands/ and test
sessions run homed to $HOME, not your project. Fix: cwd = your project,
module path via PYTHONPATH:
cd "<project>" && PYTHONPATH=<skill-creator-dir> uv run --with anthropic \
python -m scripts.run_eval --eval-set ... --skill-path ... --model ...
3. PROBE SHADOWING — the silent one
The harness does NOT test your skill in place. It writes a command file named
<skill>-skill-<uuid8>.md carrying your skill's description, then checks the
stream for THAT name. If the real skill is already installed in the
project, Claude matches the query to the REAL skill name and invokes it; the
uuid probe never fires; the harness records 0% trigger rate on every positive
query — including verbatim phrases from the description itself.
Tell: recall = 0% across ALL positives (even exact trigger phrases) while
all negatives "pass". Total zero is the signature; genuine undertriggering is
partial.
Ground-truth check (one command):
cd "<project>" && env -u CLAUDECODE claude -p "<verbatim trigger phrase>" \
--output-format stream-json --verbose 2>/dev/null | grep -c "<real-skill-name>"
A large count = the real skill triggered; the harness scores are invalid.
To measure an installed skill for real: temporarily move the skill out of
.claude/skills/ (and any plugin copies) so the probe is the only candidate,
run the eval, move it back. Or accept ground-truth spot checks.
4. Improvement step needs an API key
improve_description.py calls the Anthropic SDK directly →
ANTHROPIC_API_KEY required. Subscription-authenticated Claude Code doesn't
export one. Measurement (run_eval.py) is unaffected (it shells out to
claude -p). Without a key, do the improvement loop manually: propose a new
description, re-run run_eval, iterate.
5. SIBLING-probe shadowing — parallel workers poison each other
Same 0%-recall signature as gotcha 3, but it strikes even when the real skill
WAS correctly moved out of .claude/skills/. run_eval's default
--num-workers 10 has each parallel worker write its own
<name>-skill-<uuid8>.md probe into the same shared .claude/commands/.
Spawned sessions see up to 10 identically-described probes, invoke whichever
uuid they pick (usually a sibling's), and every worker's own-name check fails
— uniform 0/N triggers.
Tell it apart from gotcha 3: the real skill is verifiably parked, a
1-probe diagnostic run passes, and a ground-truth claude -p shows the
description matching fine.
Fix: always pass --num-workers 1. Sequential cost is ~13 min for 18
queries in a skill-heavy environment — acceptable, and the only mode that
measures what a real session would see (one candidate probe).
6. Probe-name semantic gap — harness UNDERESTIMATES real recall
Even with shadowing fully eliminated, the harness can read falsely LOW. It
registers the skill description under a uuid name (<skill>-skill-<uuid8>),
discarding the semantic value of the real skill name. Queries that
ground-truth strongly with the real installed skill (e.g. 7 stream mentions
via claude -p) can score 0/N against the uuid probe carrying the
IDENTICAL description — the real name itself was doing matching work the
probe can't replicate.
Observed instance (2026-06-11, Claude Code 2.1.173, claude-fable-5, skill
cc-hooks-main-vs-subagent): a 7-positive eval set scored 57% run-level
recall on the harness while spot-checked queries — including a harness 0/2
failure — triggered the real skill reliably.
Consequence: treat harness recall as a LOWER BOUND. Before "fixing" a
description based on harness failures, ground-truth each failing query
against the real installed skill:
env -u CLAUDECODE claude -p "<query>" --output-format stream-json --verbose \
2>/dev/null | grep -c <real-skill-name>
Only harden the description for queries that ALSO fail ground truth. Harness
rejection of negatives is unaffected (no shadowing/semantics involved) and
remains trustworthy.
Notes
run_eval strips the CLAUDECODE env var to allow nested claude -p; do
the same in manual ground-truth checks (env -u CLAUDECODE).- In skill-heavy environments (hundreds of installed skills), expect slower
claude -p boots; 3 trials × 20 queries is tens of minutes.
run_eval.py supports --runs-per-query N to bound runtime (2 is usually
enough for a first pass).- When running the eval from a subagent: launch via
nohup ... & detached —
a plain Bash run_in_background process is killed at the subagent's turn
boundary; also clean up the dead run's leftover probe file or it
sibling-shadows the relaunch.
- With the autoresearch plugin installed, its
scout-block PreToolUse hook
denies writes to any *.log path — tee the harness output to
run_eval.out, not run_eval.log.
