eval-case

name	eval-case
description	Scaffold a new golden case for @butler/evals — validate and append the case to golden.jsonl, re-capture the baseline, and confirm the eval gate still passes. Use when a real agent failure or escaped bug should become a regression benchmark.
disable-model-invocation	true

Eval Case

Add a new golden case to @butler/evals in one command: validate it, append it to packages/evals/golden.jsonl, re-capture packages/evals/baseline.json, and run the eval gate to confirm the case is picked up and still passes. Growing the regression benchmark should be repeatable, not hand-assembly.

A golden case is one real, observed failure — never a synthetic or hypothetical one. If a bug escaped, an agent looped, or a guard silently did nothing, capture it here so the harness benchmarks against it forever.

Three intake sources

A failure reaches the golden set through one of three front doors. All converge on the same sink — the GoldenStore appends to golden.jsonl — so there is one dataset and one format regardless of entry point.

Source	When	How
Failing session trace	A captured trace (.eval-traces/<traceId>.json, written by the session-trace-sink hook) recorded an error.	eval:intake-trace — derives a candidate via ingest(), queues it, and (with labels on stdin) completes it. See From a session trace below.
Phase-7 review finding	Code review surfaced a real defect the agent produced.	Hand-author the case JSON (source = the PR/review) and pipe it to eval:new-case.
Manual incident	You witnessed the failure directly and can describe it precisely.	Hand-author the case JSON (source = the session/observation) and pipe it to eval:new-case.

The review-finding and manual-incident paths are the same mechanism — a hand-authored JSON case — differing only in the source provenance. The trace path is automated through the Candidate Queue (packages/evals/src/candidate-queue.ts), which stages auto-derived candidates, dedupes them against the live golden set, and appends the completed case through the GoldenStore.

Gather the case

A case is a flat record validated by GoldenCaseSchema (packages/evals/src/schemas/golden-case.ts). Fields:

Field	Notes
id	Stable, kebab-case, dated — e.g. 2026-06-01-pretooluse-exit1-nonblocking.
failure_mode	One of the taxonomy enums (see docs/eval-failure-taxonomy.md): infinite_tool_loop, hallucinated_data, silent_error_swallowing, tool_output_misparse, config_schema_drift, timezone_locale_mishandling, unsafe_destructive_action, spec_deviation. Pick the closest real category; add a new one only if a real incident fits none.
title	One line (≤200 chars).
source	Where it really happened — session, issue/PR, commit, or observation id.
input	The task/prompt/context the agent was given.
expected	The correct behavior.
observed	What actually happened — the failure.
labels	Free-form tags (array).
created_at	ISO 8601 timestamp of the incident.
metadata	Optional object (remediation, related ids, counts).

Scaffold it

Write the case as a single JSON object and pipe it in:

pnpm --filter @butler/evals eval:new-case < case.json

The scaffolder (packages/evals/scripts/new-eval-case.mjs) then:

1. Validates + appends the case through the GoldenStore — the sole owner of the golden.jsonl line format. A schema violation aborts with a line-level error; a duplicate id is a no-op (idempotent).
2. Re-captures baseline.json from the updated golden set under the same deterministic config the gate uses (the exact-match, json-schema, and ci-signal scorers, grading each case's captured output artifact via the stored-output provider). This keeps the committed baseline in lockstep with the dataset — baseline-snapshot.test.ts asserts exactly this, so skipping it would break the build.
3. Runs the eval gate and aborts if it does not pass.

On success it prints the case id, the new case count, and eval gate PASS. Always commit golden.jsonl and baseline.json together — if you commit only golden.jsonl, the baseline is stale and baseline-snapshot.test.ts will fail on the next CI run. The eval gate output will surface a WARNING — baseline staleness line listing the missing case IDs when this drift is detected, so you can recapture before an unrelated PR is blocked.

From a session trace

When the failure is already captured as a persisted trace, let the Candidate Queue do the auto-derivable work. First review the candidate (no stdin) — this queues it and prints the auto-derived source/observed plus a labels skeleton:

pnpm --filter @butler/evals eval:intake-trace .eval-traces/<traceId>.json

Then capture it by piping the labels a trace cannot derive — failure_mode, title, input, expected, and the captured output — on stdin:

pnpm --filter @butler/evals eval:intake-trace .eval-traces/<traceId>.json < labels.json

This ingest()s the trace into a candidate, enqueues it, completes it into a valid golden case (carrying id/source/observed/created_at from the trace), appends it through the GoldenStore, then re-captures the baseline and runs the eval gate — exactly like eval:new-case. A trace with no error spans is rejected (nothing to capture); a failure already in the golden set is a no-op (deduped by id). The queue file (packages/evals/candidates.jsonl) is transient and git-ignored.

How a case is scored (scorer wiring)

Cases are not bound to a scorer per-row. The deterministic eval gate (run-eval-gate.ts, the CI Eval Gate job) builds a scorer registry, registers the deterministic scorers (exact-match, json-schema, ci-signal), and scores every case in golden.jsonl — grading each case's captured output artifact via the stored-output provider. A case with no artifact is an explicit miss (scored 0 and surfaced — never a silent expected-against-expected pass), so today's un-backfilled set yields an honest all-miss baseline; the gate then fails on any drop versus baseline.json. The llm-judge scorer (packages/evals/src/scorers/) plugs into the same registry without changing the runner but is not in the per-PR gate (it is non-deterministic and paid) — see scorer.ts for the 60/30/10 deterministic/judge/human direction and docs/eval-loop.md for the full capture-and-grade loop. Because the scaffolder re-captures the baseline from the live golden set, a new case is automatically part of the suite the gate runs.

Manual vs. automated capture

The two mechanisms are complementary, not duplicates:

• Trace ingestion (trace-ingest.ts + the Candidate Queue) mines real captured traces into candidate cases at scale, then has a human label the fields a trace cannot derive.
• Hand-authoring (eval:new-case) writes one case for an incident you already understand precisely.

Both converge on the same sink — they append through the GoldenStore to golden.jsonl — so there is one dataset and one format, regardless of entry point. Use eval:new-case when you witnessed the failure and can describe it; use eval:intake-trace when harvesting from captured traces.