agent-eval

# Agent Eval Role: orchestrator. This skill dispatches fixtures to agents and grades results — it does not review code itself. You have been invoked with the `/agent-eval` skill. Run review agents against eval fixtures and grade the results. **Monorepo-relative by design (#1637).** Every `scripts/eval_*.py`/`citation_lint.py` call in this skill is intentionally bare, not a dangling-path defect: none of these scripts ship under `plugins/dev-team/scripts/` (verified mechanically — see `tests/repo/test_agent_implemented_by_resolves.py`'s `INTENTIONAL_BARE_INVOCATION` set), so no `${CLAUDE_PLUGIN_ROOT}` path could ever resolve one. They operate on *this repo's own* `evals/` corpus, explicitly not shipped — see repo CLAUDE.md's Repository Structure — and the repo-root eval runners (`scripts/run-full-eval.sh`, `scripts/eval-changed.sh`, `scripts/run-ablation.sh`) plus `.github/workflows/agent-eval.yml` all materialize `evals/fixtures`/`evals/expected` into `.claude/evals/` via a `$PWD`-relative symlink before invoking this skill — meaningful only from this checkout's own root. The `allowed-tools` frontmatter above additionally hardcodes these bare strings as Bash permission patterns, which would stop matching if qualified. ## Orchestrator constraints 1. **Do not review or design yourself.** Delegate reviews to `/review-agent` and advisory analysis to the skill under eval (e.g. `test-design-advisor`). Your job is dispatching and grading. 2. **Grade deterministically.** Compare agent JSON output against expected JSON using exact criteria (status match, count ranges, keyword checks). Do not apply judgment. 3. **Minimize context per agent.** Pass only the fixture file to the agent — not the expected results, not other fixtures, not prior transcripts. One narrow exception: a `test-review` dispatch also receives its fixture's own `test_review_mechanics.py` mechanical pre-phase result (Step 3 below, #2169 follow-up) — computed from the fixture file itself, not from grading data, so it does not weaken this constraint's intent. 4. **Track results.** Save transcripts for saturation detection. Do not modify fixtures or expected files. 5. **Be concise.** Output the report table and failure details. No narration of each fixture run — just the grades. ## Parse Arguments Arguments: $ARGUMENTS - `--agent <name>`: Run only the named review agent (e.g., `js-fp-review`) - `--skill <name>`: Run only the named advisory skill (e.g., `test-design-advisor`) against its skill fixtures - `--fixture <name>`: Run only the named fixture (e.g., `fp-array-mutations.ts`) - `--trials <n>`: Run each fixture N times (default: 1). Enables pass@k scoring. - `--in-session`: Dispatch agents in this session instead of a fresh subprocess (see *Dispatch mode* below). Faster, but evaluates the agent definitions as already loaded — use only for cheap re-grades when no agent/skill file has been edited since they were loaded. - `--integration`: Dispatch the **integration tier** instead of the unit tier (see *Confidence pyramid* below). Routes fixtures that declare an `integration` block in their expected JSON through `scripts/run_integration_eval.py` (orchestrator dispatched against a spec in an ephemeral worktree, then the fixture's `testCommands` are graded by the `integration` grader). Cannot be combined with `--agent` or `--skill` — integration fixtures target the orchestrator, not a single reviewer. - `--ablation <agent>`: Causal drop-candidate evidence (#868). Implies the integration tier and runs it **twice** — a baseline arm (full review-agent roster) and an ablated arm (roster minus `<agent>`) — K=3 trials per arm, then reports the outcome delta (issues caught, `testCommands` results, token cost). See *Ablation mode* below for the full procedure. Rejected in combination with `--agent`/`--skill` (integration fixtures target the orchestrator) and with more than one target agent — v1 supports exactly one ablated agent per run: "error: --ablation supports exactly one agent per run (v1); combining with --agent/--skill or naming multiple agents is not supported." - `--no-cache`: Bypass the fingerprint replay cache (see *Cache* below). Default is cache-on, so a pair whose transitive inputs are unchanged replays from `evals/.eval-cache.json` at zero token cost. Use `--no-cache` when you suspect cache corruption or want a clean cost baseline. - `--verbose`: Show full agent output for each fixture - No arguments: run all agents against all applicable fixtures ## Confidence pyramid `/agent-eval` dispatches at one of three tiers, in increasing confidence and cost. See [ADR 0007](../../../../docs/adr/0007-eval-confidence-pyramid-tier-vocabulary.md) for the full vocabulary; this skill is responsible for selecting the tier and routing to the right runner. - **Unit** (default) — one reviewer or one advisory skill against one fixture, graded by the `verdict` or `skill_gate` grader. Cheap, deterministic, used for every reviewer change. This is what runs when neither `--integration` nor any tier-specific flag is set. - **Integration** (`--integration`) — the orchestrator builds against a golden-repo spec in an ephemeral worktree; the fixture's `testCommands` must all exit 0 (graded by the `integration` grader). Validates that a plan produces code that compiles and passes tests, not just that a reviewer flagged the right thing. Opt-in, costs orchestrator + builder tokens. - **Acceptance** — production-style runs over real PRs / merged code, scheduled outside this skill (the live regression gate in CI). `/agent-eval` does not dispatch acceptance directly; mention it for context only. When the operator does not specify a tier, run **unit** and report in the final table how many fixtures would have routed to integration (the count of expected JSON files whose `integration` block is non-empty). That nudge keeps integration coverage visible without forcing every run to pay for it. ## Ablation mode (`--ablation <agent>`, #868) Gives `/harness-audit` **causal** drop-candidate evidence instead of the correlational usage data it derives from `.claude/metrics/review-value.jsonl` alone: does removing this agent from the roster actually change integration-tier pipeline outcomes? **OPT-IN / LABEL-GATED — same policy as the integration tier (#134).** Before doing anything else, confirm the opt-in gate is present (a `run-integration` / live-eval label on the invoking issue or PR, or the operator explicitly confirming a live, cost-incurring run in this session). If the gate is absent: ```text refused: --ablation requires the label-gated live-eval opt-in (#134) — the same policy that gates the integration tier. Add the `run-integration` label or have the operator explicitly confirm a live run, then retry. Nothing was dispatched; nothing was written to .claude/metrics/. ``` Stop there — dispatch nothing, write nothing to `.claude/metrics/`. **Argument validation.** Reject before any dispatch if `--ablation` is combined with `--agent`/`--skill`, or if more than one agent is named (v1 is single-agent only — see *Parse Arguments* above for the exact error text). **Fixtures.** Default to every expected JSON with a non-empty `integration` block (narrow with `--fixture <name>`). **Cost estimate + explicit confirmation — before any dispatch.** Two arms (baseline, ablated) × K=3 trials each × the number of selected fixtures is 6× the integration tier's normal per-fixture cost. Print an estimate before dispatching the first trial of either arm: ```text Ablation cost estimate: <agent> × <N> fixture(s) × 2 arms × 3 trials = <6·N> integration dispatch(es). Prior integration runs on this fixture set averaged ~<tokens> tokens/dispatch (or: "no prior run recorded — using the integration tier's stated per-dispatch heuristic"), so expect roughly <estimate> tokens total. Proceed? [y/N] ``` Declining aborts immediately — zero tokens spent, nothing written. **Dispatch — cache bypassed.** Both arms must be live to be comparable, so skip the fingerprint-replay cache pre-flight entirely for ablation arms (same stance as the plain integration tier). For each selected fixture, run 3 trials of each arm, writing each trial's actuals to its own file under a per-arm trials directory: ```bash # baseline arm (full roster) — repeat 3x into baseline-trials/trial-<n>.json python3 scripts/run_integration_eval.py --fixtures-dir evals/fixtures \ --expected-dir evals/expected --only <fixture> --model <model> \ --out baseline-trials/trial-1.json # ...trial-2.json, trial-3.json # ablated arm (roster minus <agent>) — repeat 3x into ablated-trials/ python3 scripts/run_integration_eval.py --fixtures-dir evals/fixtures \ --expected-dir evals/expected --only <fixture> --model <model> \ --ablate-agent <agent> --out ablated-trials/trial-1.json # ...trial-2.json, trial-3.json ``` `--ablate-agent <agent>` scopes `DEV_TEAM_ABLATE_AGENT=<agent>` to that one subprocess's environment — no file mutation, no cleanup risk; the roster reverts the instant the subprocess exits. Never edit an agent/skill file to achieve the exclusion. **Delta computation and persistence.** Hand both trial directories to `eval_ablation.py --mode agent` — the deterministic, model-free half that grades each arm (pass@k over the 3 trials, per the same machinery `eval_variance.py` uses for `--trials`) and computes the delta. `eval_ablation.py` ships under `plugins/dev-team/scripts/` (#1653 — its `--find-latest` mode has no repo-shape assumption), so it is `${CLAUDE_PLUGIN_ROOT}`-qualified below even though this specific invocation (`--mode agent --expected-dir evals/`) still only succeeds from this repo's own dev checkout, same as its still-bare siblings (`eval_cache.py`, `eval_variance.py`, `run_integration_eval.py`, `citation_lint.py`) — this is per-script, not per-skill: the FILE moved, the eval-corpus dependency the `--mode agent` path carries did not: ```bash python3 "$CLAUDE_PLUGIN_ROOT/scripts/eval_ablation.py" --mode agent \ --expected-dir evals/expected \ --baseline-trials-dir baseline-trials --ablated-trials-dir ablated-trials \ --ablated-agent <agent> --model <model> \ --append metrics/eval-ablation.jsonl ``` This appends one `schema: "eval-ablation/v1"` JSONL record (`recorded_at`, `ablated_agent`, `fixtures`, `model`, `baseline`/`ablated`/`delta` — each covering `issues_caught`, `test_commands`(_passed for delta), `tokens` — and `verdict`). **A failing baseline arm blocks any drop/retain claim**: the script itself forces `verdict: "baseline failed — inconclusive"` when the baseline didn't pass every trial — report that plainly, do not paraphrase it into a drop/retain recommendation. **Console report.** Show a small table: baseline vs ablated vs delta for the three dimensions, the verdict, and the path (`metrics/eval-ablation.jsonl` — deliberately not relocated under `.claude/`) the record was appended to. ## Cache (fingerprint replay) `scripts/eval_cache.py` SHAs each `fixture::target` over the target's definition + the **transitive closure** of files it reaches (knowledge/, skills/, and any bare `scripts/<n>.py` it names — e.g. `test-review.md`'s pointer to `test_review_mechanics.py`, #2169) + the fixture + expected JSON + grader version. An unchanged SHA with a stored PASS replays at zero token cost; any changed input busts the cache — including an edit to `test_review_mechanics.py` itself, so a Phase 0 detection-logic change correctly busts cached `test-review` pairs instead of silently replaying a result computed under the old script. Default behaviour (cache-on): 1. Before dispatch, run `python3 scripts/eval_cache.py --plan --replay-out replay.json` over the resolved fixture/target pair list. This produces the **hit/miss** split. 2. Dispatch ONLY the misses through Step 3. 3. After grading, run `python3 scripts/eval_cache.py --store --actuals actuals.json` to memoize the new passes. 4. Report `cache: <hits>/<total> replayed at $0` in the summary. `--no-cache` skips steps 1-3 and dispatches everything. Use it only when you suspect the cache is wrong; the cache busts automatically on any input change, so there is normally no operator decision to make. ## Cites: enforcement (drift prevention for the evals themselves) This skill grades reviewers; if a reviewer carries thresholds with no `Cites:` declaration, `/agent-eval` is grading drift it cannot detect. Before dispatching a `--agent <name>` run: 1. Probe `python3 scripts/citation_lint.py --plugin-root plugins/dev-team plugins/dev-team/agents/<name>.md`. 2. If it emits an `advisory — states N normative token(s) ... declares no Cites:` warning, surface that **once** above the run report: ```text ⚠ agent <name> states N numeric threshold(s) on RFC-2119 lines but has no Cites: declaration. Eval results will not catch drift in those thresholds. Add a Cites: list naming the canonical sources. ``` 3. Continue with dispatch — the warning is advisory, never blocking (matches the Phase-1 lint contract). This makes the eval skill itself a regression guard against threshold drift, not just a grader for it. ## Dispatch mode The agent and skill definitions under eval are **plain files on disk**. An eval is only honest if it grades what is *currently on disk*, not a copy that was loaded into this session before you started editing. - **Default — fresh subprocess (disk is authoritative).** Each fixture/agent pair is dispatched via a fresh `claude -p` subprocess (Step 3). The subprocess loads the agent/skill definitions from disk on every run, so a definition you edited a moment ago is reflected immediately. This is the correct mode after any edit and the default for that reason. - **`--in-session` — fast path (no fresh load).** Dispatches via the in-session `/review-agent` and skill invocations. This reuses whatever was already loaded, so it is cheaper and quicker, but it can evaluate a **stale** definition if the file changed mid-session. Use it only for repeated grading when you have not touched the agent/skill/knowledge files. **No silent fallback.** Default mode requires the `claude` CLI on `PATH`. If `command -v claude` finds nothing, **stop with a clear error** — do not quietly run in-session, because that would grade a stale definition without the user knowing: ```text error: /agent-eval default (fresh-subprocess) mode needs the `claude` CLI on PATH, which was not found. Install it, or re-run with --in-session to dispatch in this session (warning: --in-session can grade a stale, already-loaded definition if you have edited agent/skill files this session). ``` ## Steps ### 1. Resolve eval corpus Verify `.claude/evals/fixtures/` exists. If not, error: "Cannot find eval fixtures. Expected at `.claude/evals/fixtures/`." ### 2. Load fixtures and expected results Read all files from `.claude/evals/fixtures/` and corresponding JSON from `.claude/evals/expected/`. For each fixture: - Match the fixture stem (filename without extension) to its expected JSON - For directory fixtures (cs-*), the directory name is the stem - Parse `applicableAgents` (review-agent fixtures) and/or `applicableSkills` (advisory-skill fixtures, e.g. the `tlg-*` test-layer-gates corpus) to know what to dispatch. A fixture is an **agent fixture**, a **skill fixture**, or both, depending on which keys its expected JSON declares. If `--agent` is specified, filter to fixtures where that agent is in `applicableAgents`. If `--skill` is specified, filter to fixtures where that skill is in `applicableSkills`. If `--fixture` is specified, filter to that fixture only. ### 3. Run agents against fixtures If `--ablation <agent>` is set, skip the rest of this section and follow *Ablation mode* above in full (opt-in gate check, argument validation, cost estimate + confirmation, two-arm × 3-trial dispatch, delta computation and persistence, console report). Do not fall through to plain `--integration` handling below. If `--integration` is set, skip the rest of this section and dispatch through `python3 scripts/run_integration_eval.py` for every fixture whose expected JSON declares an `integration` block. The runner builds the ephemeral worktree from the fixture's `goldenRepo` tarball, dispatches the orchestrator against the spec, runs the `testCommands`, records exit codes, and tears the worktree down unconditionally. Pass the actuals it writes directly to Step 4 — the `integration` grader will pick them up from the expected JSON's `integration` block. Skip the cache pre-flight in this branch for now (cache wiring for the integration tier is tracked separately to keep the rollout small). For unit-tier runs, first resolve the dispatch mode (see *Dispatch mode* above) and consult the cache (see *Cache* above): - **Default (fresh subprocess).** Run `command -v claude`. If it is missing, STOP with the error in *Dispatch mode* — do not fall back to in-session. Otherwise dispatch each pair through a fresh subprocess so the definition is re-read from disk every run. - **`--in-session`.** Skip the `claude` check and use the in-session invocations described under each pair below. For each fixture/agent pair (agent fixtures): 1. **Test-review Phase 0 pre-phase (#2169 follow-up).** When `<agent-name>` is `test-review`, first compute the mechanical pre-phase result for the fixture file — the same invocation `/code-review` step 2b uses: