build-evals

# build-evals — the eval builder !`cat .forge/state.json 2>/dev/null || echo NO_STATE` Generate the complete `evals/` tree for a target plugin. In `new` mode the plugin does not exist yet: tasks derive from `design/CONTRACT.md` (the PDR), not from code. The suite this skill emits becomes the definition of "done" for the entire build — treat every task as a contract clause, not a test. This skill drafts and lints the suite. It never arms, freezes, or scores it — those are `/plugin-forge:arm-evals` and `bin/forge-eval`. It is model-invocable by design (decision D2): the forge conductor chains it via the Skill tool; safety comes from the state guard below, not from invocation control. ## State guard (new mode only) Apply this table to the injected state line above before doing anything else in `new` mode (also the default when no argument is given). `retrofit` and `mine` skip this guard entirely — they run standalone against any plugin, with or without a forge pipeline (see their sections). | Injected state | Action | |---|---| | `NO_STATE` | REFUSE. Say: "No forge pipeline in this project. Run `/plugin-forge:forge` to start one, or use `/plugin-forge:build-evals retrofit <plugin-path>` for an existing plugin." Stop. | | `phase` is `interview` or `contract` | REFUSE. Say: "The contract is not approved yet — evals derive from it. Run `/plugin-forge:forge resume`." Stop. | | `phase` is `evals` | Proceed with new mode. | | `phase` is `armed`, `building`, `smoke`, `verify`, or `ship` | REFUSE edits. The suite is frozen (sha256 in `.forge/freeze.json`); guard hooks will block writes anyway. Say: "The suite is frozen. Append the desired change to `.forge/eval-change-requests.md`, then run `/plugin-forge:arm-evals` to apply it, re-gate, and re-freeze as a new version." Stop. | ## Mode: new (pipeline phase) Work through the steps in order. Load the referenced file before executing its step — each reference is binding, not advisory. ### 1. Map the surface from the contract Read `design/CONTRACT.md`. Extract, per capability row: - the chosen composition (skills, hooks, agents, MCP servers, scripts) and its archetype - the **C5 Testability** column: grader type, observable signal, and planned task IDs — this column IS the task backlog; build-evals exists to make those IDs real - the **C1 Trigger** column: every auto-invoked component (model-invoked skill, hook, agent with proactive description) needs positive AND negative trigger tasks - external dependencies (MCP servers, APIs, corpora) and whether the contract chose live or mock for each Also read `design/INTENT.md` for the runtime-environment answers (headless/cloud/local) — they constrain what graders can observe. ### 2. Interview — propose clusters, recommend, WAIT Never one-shot the suite (interviewing beats one-shot generation — LangChain's central empirical finding). Use AskUserQuestion to: 1. Propose 3–5 testable ability clusters derived from the PDR rows, each with a one-line description, its archetype, and a rough task count. Recommend one ordering and say why. WAIT for the user's choice before generating anything. 2. For every external dependency, ask live-vs-mock. Default recommendation: mock fixture stubs for write-heavy, paid, or rate-limited tools; live only for cheap read-only calls the user explicitly accepts. Record the decision per dependency — the harness stores which mode each result came from. For scientific or business domains where fixture corpora need domain-real content, offer to run the `/plugin-forge:skill-forge` research pipeline to ground fixtures in real sources instead of inventing plausible-looking data (decision D10). ### 3. Scaffold Copy the scaffold from `${CLAUDE_PLUGIN_ROOT}/templates/evals/` into the target plugin as `evals/`, and create the sibling `evals-holdout/` next to it (NOT inside `evals/`). The scaffold provides `registry.yaml`, `targets/{default,no-plugin,crowded}.yaml`, `suites/{capability,regression,triggers}.yaml`, `judges/{fact,rubric,battle}.yaml`, `labels/labels.jsonl`, and `tasks/_example/`. Delete `tasks/_example/` after real tasks exist. Then vendor the harness runner (stdlib-only, so the suite outlives plugin-forge): ``` mkdir -p <plugin_dir>/evals/bin && cp "${CLAUDE_PLUGIN_ROOT}/scripts/harness/run.py" <plugin_dir>/evals/bin/run.py && chmod +x <plugin_dir>/evals/bin/run.py ``` ### 4. Author tasks Follow `references/task-authoring.md` for everything about tasks: the exact `task.yaml` schema, the 20–50 task budget, the two-experts bar, the REQUIRED reference solution, the goal-condition grammar for `instruction.md`, fixture rules, the holdout split, and `elicitation_answers`. Every task ID planned in the contract's C5 column must exist in `evals/tasks/` or `evals-holdout/tasks/` when this step ends. ### 5. Choose graders Follow `references/grader-menu.md`: deterministic-first hierarchy (workspace → transcript → state_check → judge → trigger, plus the simulate_user driver), archetype routing, the mandatory numeric `tolerance`, the never-assert-tool-ORDER rule, and the crowded-listing requirement for trigger tasks. ### 6. Author judges Only where the grader menu genuinely runs out of deterministic options. Follow `references/judge-authoring.md`: modelgraded-spec YAML shape, one rubric dimension per isolated judge call, Unknown always legal, pinned cheap judge model different from the target model, anti-echo phrasing. Seed `evals/labels/labels.jsonl` with the unlabeled sample skeleton — calibration itself is arm-evals' gate, not this skill's. ### 7. Generate the elicitation auto-answer hook (when applicable) If the target plugin bundles or drives an MCP server that uses elicitation, headless eval runs will hang or die — no dialog exists under `claude -p`. For every such server: - set `elicitation_answers` in each task.yaml that exercises it (see task-authoring.md) - emit `evals/hooks/elicitation-auto-answer.sh` (bash, `set -euo pipefail`, executable) and `evals/hooks/elicitation-settings.json` registering it as an exec-form Elicitation hook: `"command": "${CLAUDE_PLUGIN_ROOT}/evals/hooks/elicitation-auto-answer.sh"` — harness runs merge this settings file; production users never load it. The script reads the elicitation request JSON from stdin, looks up the requested fields in `.forge-eval/elicitation-answers.json` (staged into the workspace by the harness from the task's `elicitation_answers`), and prints the matching answers object to stdout with exit 0. No match → one-line stderr message and exit 2, so the trial fails loudly instead of hanging. ### 8. Lint the suite Run every mechanical lint in `references/antipatterns.md` (L01–L16) and fix all findings before proceeding. L01 (numeric tolerance) and L02 (task-text/grader consistency) are the builder meta-eval contract: plugin-forge's own dogfood suite feeds this skill adversarial specs engineered to reproduce the CORE-Bench and METR grader bugs, and passes only if the emitted suite provably cannot contain them. Do not hand a suite to arm-evals with any lint red. ### 9. Smoke-run drafts and hand off An eval that has never run is not delivered: - Run `forge-eval reference --task <ID>` for at least 3 representative tasks (one per archetype used) and show the user the graded output. Use `forge-eval run --dry-run` for a cost estimate of the full suite; state it plainly — trials × tasks × judges is real money, and `--max-cost-usd` exists for a reason. - Present the suite summary: task count per suite (capability/regression/triggers), holdout count, judge dimensions, live-vs-mock map, estimated full-run cost. - Tell the user the exact next command: `/plugin-forge:arm-evals` — the four arming gates (reference-green, red baseline, judge calibration, freeze) all run there. Do not modify `.forge/state.json`. The phase advances to `armed` only when arm-evals freezes the suite. ## Mode: retrofit <plugin-path> Standalone — no forge pipeline, no state guard, works on any installed or in-repo plugin. 1. Map the plugin surface from files (there is no PDR): every `skills/*/SKILL.md` and command frontmatter (descriptions = trigger contracts), `agents/*.md`, `hooks/hooks.json` (matchers = firing contracts), `.claude-plugin/plugin.json`, `.mcp.json`. Derive one ability cluster per user-facing capability and one trigger axis per auto-invoked component. MCP tool checks must use plugin-scoped names: `mcp__plugin_<plugin>_<server>__<tool>`. 2. Then run steps 2–9 above unchanged, writing `evals/` and `evals-holdout/` into the plugin's own directory. Where step 1 of new mode says CONTRACT.md, use the surface map; flag any capability whose observable signal is unclear as "unverifiable — needs a design decision" instead of inventing a grader for it. 3. Recommend arming: retrofit suites get the same four gates via `/plugin-forge:arm-evals <plugin-path>` before their scores mean anything. ## Mode: mine Standalone — converts captured production failures into tasks (the flywheel). 1. Read traces from the generated plugin's trace-capture hook output: `${CLAUDE_PLUGIN_DATA}/traces/*.jsonl` (tool args, results, errors per session). If no traces exist, say so and stop — mine has no other input; suggest confirming the trace-capture hook is installed. 2. Cluster failures (repeated errors, user corrections, wrong outputs) and propose each cluster as a candidate task via AskUserQuestion — same interview discipline: propose, recommend, WAIT. 3. For each accepted failure, author a task per `references/task-authoring.md` with the real failing input as `instruction.md` seed, a fixture reconstructing the state, and a reference solution showing the correct behavior. New tasks go to `evals/tasks/` in the regression suite (they protect against a now-known failure). 4. A frozen suite that gains tasks is a NEW suite: finish by instructing the user to run `/plugin-forge:arm-evals` to re-gate and re-freeze (registry version bumps, scores stay comparable). If guard hooks block the writes, that is the freeze working — route the change through `.forge/eval-change-requests.md` as the guard message says. ## Standing rules (all modes) - Deterministic-first, always. A judge where a file check would do is a bug. - Every task ships a reference solution. No exceptions — 0% pass usually means a broken task, and only the reference proves otherwise. - Positive AND negative trigger cases for every auto-invoked component; one-sided suites train over-triggering. - Golden answers, key-fact lists, and reference solutions live under `reference/`, which the harness stages OUTSIDE the agent-visible worktree. Never copy any of it into `fixtures/` or `instruction.md`. - Holdout tasks (`evals-holdout/`) are never named, quoted, or summarized in any session the builder loop can read. - Track cost honestly: `tracked_metrics` are recorded, never graded; suite cost estimates go to the user before, not after, an expensive run. - Never emit deprecated shapes: no legacy decision/approve hook JSON, no legacy mount-path skill references, and MCP tool names always in the full plugin-scoped form `mcp__plugin_<plugin>_<server>__<tool>` — never the short server-only form. ## References - `references/task-authoring.md` — task budget, schema, instruction grammar, fixtures, holdout, elicitation fixtures - `references/grader-menu.md` — grader hierarchy, archetype routing, trigger + crowded target, simulate_user - `references/judge-authoring.md` — modelgraded spec shape, judge rules, calibration handoff - `references/antipatterns.md` — reward-hacking catalog with the mechanical lints L01–L16