| name | skill-eval |
| description | Author and tier behavioral probes for a skill, including seeded-defect probes that escape ceiling saturation. Triggers: "measure this skill", "the probe came back INERT", "the control arm aces it", "harden this scenario", "is this skill actually doing anything". |
| practices | ["measurement-over-assertion","ab-testing"] |
| skill_api_version | 1 |
| hexagonal_role | supporting |
| consumes | ["skill-source-package"] |
| produces | ["probe-package","probe-result.v1"] |
| context_rel | [{"kind":"supplier-to","with":"skill-builder"}] |
| user-invocable | true |
| metadata | {"tier":"meta","dependencies":[],"capabilities":["author_seeded_probe","run_probe_tier"],"effects":["write_probe_package","dispatch_probe_producer"],"canonical_status":"canonical","disposition":"keep_specialist","stability":"experimental"} |
/skill-eval
Author one behavioral probe for one skill, at the cheapest tier that can still
separate the arms, and report the verdict honestly. A probe measures
behavior-change — did loading the skill change what the agent did — never
quality-uplift. This skill authors and tiers probes. scripts/probe-skill.sh
runs them.
Insight: when a probe returns INERT because the control arm already aces the
scenario, the measurement failed, not the skill. Weakening the producer is one
escape and it costs realism. The cheaper escape is to plant the defect: build
a scenario containing exactly one flaw the discipline catches and a skim does
not, then grade whether the agent acted on it. Signal you manufacture is signal
you can reproduce.
The failure mode this exists to prevent: a skill catalog whose tier badges are
editorial. A skill nobody measured is a skill nobody can defend, and re-running a
saturated scenario at a lower effort level produces more rows in the ledger
without producing more knowledge.
Modes
| Trigger phrases | Mode | Entry point |
|---|
| "measure this skill", "does this skill do anything" | author tier 1 (quiz probe) | evals/skill-probes/<id>/ |
| "the control arm aces it", "harden this scenario" | author tier 2 (seeded-defect probe) | references/seeding.md |
| "the probe came back INERT" | diagnose headroom | gate skill.probe-headroom |
| "run the probes" | run a tier | scripts/probe-skill.sh |
Inputs
Required: the skill slug, and one sentence naming the action the skill should
cause — a tool call made, an artifact written, a question raised, a sequence
followed. If the sentence names a belief instead of an action ("understands
that…", "considers…"), stop: that is not probeable, and rewriting it as an action
is the actual work.
Optional: an existing probe id to harden.
Non-goals. This skill does not score output quality, rank skills, claim a
skill is good, or gate a release. It does not run claude -p. It does not
generalize from N=2 — small N is directional and every artifact it produces says
so.
The two tiers
