evaluate-matrix

# /evaluate:matrix The **executability** gate. Where `/evaluate:legibility` asks "can a fresh agent *comprehend* this skill," this skill asks "can a *weak model* actually *do* it" — run the skill's evals on haiku (and opus/sonnet) with real tool execution, grade the produced artifact, and surface the per-skill verdict `executable_on_haiku`. A skill that opus passes and haiku fails leans on reasoning the cheap model lacks. This builds the orchestration the cross-model design ([`docs/cross-model-evaluation.md`](../../docs/cross-model-evaluation.md)) calls a follow-up. It reuses, without duplicating: `prepare_run.sh`, `grade_deterministic.py` (zero-token first pass), the `eval-grader` agent (deferred fuzzy checks only), `model-matrix.json`, and `render_matrix_report.py`. ## When to Use This Skill | Use this skill when... | Use alternative when... | |------------------------|------------------------| | Checking whether a weak model can *execute* a skill | Checking whether the SKILL.md *reads* clearly -> `/evaluate:legibility` | | Running the Tier-2 cross-model sweep on a golden-set skill | Single-skill, single-model effectiveness -> `/evaluate:skill` | | Diagnosing a skill that opus passes but haiku fails | Structural/lint validation -> `scripts/plugin-compliance-check.sh` | | Re-checking canaries after a new model ships | Improving a skill from results -> `/evaluate:improve` | ## Context - Available plugins: !`find . -maxdepth 2 -type d -name '*-plugin' -not -name '.claude-plugin'` ## Parameters | Parameter | Default | Description | |-----------|---------|-------------| | `<plugin/skill-name>` | required | Target skill as `plugin-name/skill-name` | | `--models <list>` | `opus,haiku` | Comma-separated pinned aliases to run | | `--with-skill-only` | false | Skip the cached baseline side (with-skill runs only) | | `--runs N` | 1 | Runs per (model × eval × config) | Aliases float across model generations, so `--models` accepts either an alias or a full id; pass the **full id** when the run must be reproducible (the `model` field on the dispatch accepts one) and record the id that actually ran — never the alias — in `model-matrix.json metadata.models[].model_id` (`.claude/rules/skill-evaluation.md`). ## Execution Execute this cross-model matrix: ### Step 1: Resolve skill and evals Read `<plugin-name>/skills/<skill-name>/evals.json`. If absent, report that the matrix needs eval cases (point at `/evaluate:skill --create-evals`) and stop. Validate it against the evals.json schema ([`references/schemas.md`](../../references/schemas.md)). ### Step 2: Run the matrix (serialized) Loop over (model ∈ `--models`) × eval × config ∈ {`with_skill`, `cached_baseline`} — skip `cached_baseline` if `--with-skill-only`, and reuse a baseline cached for the same model-version rather than re-running it. For each combination: 1. Scaffold the run dir: ``` bash ${CLAUDE_PLUGIN_ROOT}/scripts/prepare_run.sh \ --skill-dir <plugin-name>/skills/<skill-name> \ --eval-id <eval-id> --run <N> ``` If the eval carries a `fixture` block, apply it for an honest execution context — without one a context-needing skill fails on haiku purely for lack of fixtures, a false negative that poisons this gate: ``` bash ${CLAUDE_PLUGIN_ROOT}/scripts/apply_fixture.sh \ --fixture '<eval.fixture JSON>' --repo-root "$(pwd)" ``` Parse `WORKDIR=`; the subagent operates there. Tear it down after the transcript is copied out (`--teardown "$WORKDIR"`). 2. Dispatch **one `Task` subagent with the `model` field set to the loop model** (full `Bash`/`Edit` — it does real tool execution, not just reading): ``` Task subagent_type: general-purpose model: <loop model alias> prompt: <eval prompt; with_skill runs also receive the SKILL.md content> ``` **Serialize** the dispatches — one at a time, never a parallel batch. `[1m]` models hit cascading rate limits with concurrent subagents (`.claude/rules/skill-fork-context.md`). Spawns run in the background by default (Claude Code ≥ 2.1.232), so dispatch order alone no longer serializes them: pass `run_in_background: false` on each dispatch, or wait for that agent's result to arrive before issuing the next one — serialization is the wait, not the dispatch order. 3. Write the subagent's produced artifact to `$RUN_DIR/transcript.md`. ### Step 3: Grade — deterministic first, judge only on deferral For each run, grade the produced output: 1. Run the zero-token deterministic grader first: ``` python3 ${CLAUDE_PLUGIN_ROOT}/scripts/grade_deterministic.py \ --evals <evals.json> --eval-id <eval-id> --output $RUN_DIR/transcript.md --json ``` 2. Only if it reports `JUDGE_PENDING > 0`, dispatch the `eval-grader` agent for the deferred fuzzy expectations: ``` Task subagent_type: evaluate-plugin:eval-grader Prompt: Grade ONLY the deferred (judge) expectations for <eval-id> ... ``` Most expectations grade deterministically — the judge fires on a fraction. ### Step 4: Aggregate to model-matrix.json Combine per-run pass rates into `<skill-dir>/eval-results/model-matrix.json` following the schema. Compute, per model alias, the mean `with_skill` and `baseline`, the `delta`, and `prev_delta` from any stored prior run. ### Step 5: Render the report ``` python3 ${CLAUDE_PLUGIN_ROOT}/scripts/render_matrix_report.py \ <skill-dir>/eval-results/model-matrix.json ``` The renderer emits the delta table, per-model verdicts, the **portability flag** (opus−haiku spread ≥20 pts), and the **executability flag** (`executable_on_haiku=false` when haiku's absolute with-skill rate is below the 0.5 floor while opus clears it). Print the report and call out whether the executability flag fired. ## Minimal Provable Increment Run the matrix for **`git-plugin/git-commit` only** (it already has typed-check evals), `--models opus,haiku --with-skill-only`: grade deterministically, render, and confirm the executability callout lights up or stays dark correctly. This exercises every reused piece end-to-end before scaling to the golden set. ## Agentic Optimizations | Context | Command | |---------|---------| | Inspect eval setup | `bash evaluate-plugin/scripts/inspect_eval.sh --plugin-dir <plugin>/skills/<skill>` | | Prepare a run dir | `bash evaluate-plugin/scripts/prepare_run.sh --skill-dir <dir> --eval-id <id> --run <N>` | | Deterministic grade | `python3 evaluate-plugin/scripts/grade_deterministic.py --evals <f> --eval-id <id> --output <out> --json` | | Render the matrix | `python3 evaluate-plugin/scripts/render_matrix_report.py <dir>/eval-results/model-matrix.json` | ## Quick Reference | Flag | Meaning | |------|---------| | `--models opus,haiku` | Which pinned aliases to run (default opus,haiku) | | `--with-skill-only` | Skip the cached baseline side | | `--runs N` | Runs per (model × eval × config) | ## Related - `/evaluate:legibility` — the comprehension gate (cold-read, no execution) - `/evaluate:skill` — single-model effectiveness with a baseline - [`docs/cross-model-evaluation.md`](../../docs/cross-model-evaluation.md) — the design this implements - `.claude/rules/skill-evaluation.md` — tiered methodology, golden set, cadence - `.claude/rules/skill-fork-context.md` — why subagent dispatch is serialized