skill-eval

name	skill-eval
description	Author and tier behavioral probes for a skill, including seeded-defect probes that escape ceiling saturation. Triggers: "measure this skill", "the probe came back INERT", "the control arm aces it", "harden this scenario", "is this skill actually doing anything".
practices	["measurement-over-assertion","ab-testing"]
skill_api_version	1
hexagonal_role	supporting
consumes	["skill-source-package"]
produces	["probe-package","probe-result.v1"]
context_rel	[{"kind":"supplier-to","with":"skill-builder"}]
user-invocable	true
metadata	{"tier":"meta","dependencies":[],"capabilities":["author_seeded_probe","run_probe_tier"],"effects":["write_probe_package","dispatch_probe_producer"],"canonical_status":"canonical","disposition":"keep_specialist","stability":"experimental"}

/skill-eval

Author one behavioral probe for one skill, at the cheapest tier that can still separate the arms, and report the verdict honestly. A probe measures behavior-change — did loading the skill change what the agent did — never quality-uplift. This skill authors and tiers probes. scripts/probe-skill.sh runs them.

Insight: when a probe returns INERT because the control arm already aces the scenario, the measurement failed, not the skill. Weakening the producer is one escape and it costs realism. The cheaper escape is to plant the defect: build a scenario containing exactly one flaw the discipline catches and a skim does not, then grade whether the agent acted on it. Signal you manufacture is signal you can reproduce.

The failure mode this exists to prevent: a skill catalog whose tier badges are editorial. A skill nobody measured is a skill nobody can defend, and re-running a saturated scenario at a lower effort level produces more rows in the ledger without producing more knowledge.

Modes

Trigger phrases	Mode	Entry point
"measure this skill", "does this skill do anything"	author tier 1 (quiz probe)	evals/skill-probes/<id>/
"the control arm aces it", "harden this scenario"	author tier 2 (seeded-defect probe)	references/seeding.md
"the probe came back INERT"	diagnose headroom	gate skill.probe-headroom
"run the probes"	run a tier	scripts/probe-skill.sh

Inputs

Required: the skill slug, and one sentence naming the action the skill should cause — a tool call made, an artifact written, a question raised, a sequence followed. If the sentence names a belief instead of an action ("understands that…", "considers…"), stop: that is not probeable, and rewriting it as an action is the actual work.

Optional: an existing probe id to harden.

Non-goals. This skill does not score output quality, rank skills, claim a skill is good, or gate a release. It does not run claude -p. It does not generalize from N=2 — small N is directional and every artifact it produces says so.

The two tiers