evaluate-agent

name	evaluate-agent
description	Run a coding agent blind against a benchmark task, review its work, score it on every metric, and write a report. Also runs stateful, resumable multi-task campaigns ("fully eval <model>") that tick a durable checklist case-by-case in serial order. Use when the user says "evaluate <model> on <task>", "fully eval <model>", "run the benchmark", "score an agent", "test <model> against <project>", "resume the eval". Spins up a throwaway git worktree in /tmp/work, dispatches the agent (pi-as-mcp by default), waits serially, then produces reports/<timestamp>-<model>/ with report.md, metrics.json, frozen request+solution, and diff.zip.
version	1.1.0

evaluate-agent

Run blind evaluations and produce scored reports — one task at a time, or a whole campaign (every task, every project) tracked by a durable, resumable checklist.

Usage:

• Single cell: /evaluate-agent <project> <task> [model] [--runner pi-as-mcp] (task = the task-dir name, e.g. hard-coder or, for meta-projects, polygon-cli-easy-scout).
• Campaign: /evaluate-agent <model> --all · fully eval <model> · resume eval <model> · or a bare <model> with no project/task. Optional --scope <project> narrows to one project's cells; --rerun re-runs cells that already have a report for this model.

model defaults to whatever the user names; list options with the runner's model list (pi-as-mcp: models()).

Mode detection. First argument is a known project slug and a task is given → single-cell mode (run §"The loop (per task)" once). Otherwise (model only, --all, fully/resume eval, or first arg isn't a project) → campaign mode (§"Campaign mode"). When genuinely ambiguous, ask.

The agent under test must believe it's working in an ordinary repo on an ordinary request. Never pass it anything from this bench repo other than task.md.

Campaign mode

A campaign is a durable, resumable checklist of every (project × task) cell to evaluate for one model. It lets a long "fully eval " run survive interruptions: progress is on disk, so you (or a fresh session) can pick up at the first unchecked box. State is owned by lib/campaign.py:

• campaigns/<safe-model>/state.json — source of truth (one entry per cell).
• campaigns/<safe-model>/checklist.md — rendered, tickable view (regenerated on every update; don't hand-edit).

C1. Create or resume the campaign

lib/campaign.py init "<model>" [--scope <project>|all] [--runner pi-as-mcp] [--rerun]

Idempotent: it enumerates cells from manifest.json in serial order (manifest project order, then each project's tasks[]), and — unless --rerun — marks a cell done when a report already exists for that (model, project, task) (resume). Existing done/failed statuses are preserved across re-inits. Default scope is all (~every authored cell across all projects). Show the user the printed done/total and confirm scope before a large run.

C2. Walk the checklist, one cell at a time (serial)

Loop until next prints nothing:

lib/campaign.py next "<model>"        # -> "<project>\t<task>\t<task_id>"  (empty = done)

For each returned cell:

1. lib/campaign.py mark "<model>" <project> <task> running
2. Run §"The loop (per task)" below verbatim for that <project> <task> (worktree → report → dispatch → collect → diff → review → score → report.md → cleanup). This is the unchanged single-cell pipeline.
3. On success: lib/campaign.py mark "<model>" <project> <task> done --report <report_dir>. If the run failed/hung and you scored what exists: … failed --note "<why>" (the cell stays visible; re-running the campaign will retry it — change to done once a report is written, or leave failed to skip).
4. Re-read next; continue. Never run two cells in parallel — review each fully before dispatching the next.

After the loop, lib/campaign.py show "<model>" and report the final tally + where the per-cell reports live.

Resuming

Re-invoke with the same model (fully eval <model> / resume eval <model>): init reconciles the checklist with reports on disk, then C2 continues from the first pending/running cell. A cell left running by a crashed session is retried. Nothing already done is repeated unless --rerun.

The loop (per task)

1. Prepare a blind worktree

lib/new_worktree.sh <project>        # prints: <worktree_path> then <run_id>

This creates /tmp/work/<run_id> as a clean git worktree of the project at its pinned commit. Record base_commit = git -C projects/<project>/repo rev-parse HEAD.

2. Scaffold the report (freezes the prompt + solution now)

lib/new_report.sh <model> <project> <task>   # prints the report dir

This copies task.md → request.md and solution.md → solution.md into the report dir so the report keeps the ground truth as it was at run time, and drops in blank report.md + metrics.json from the templates.

3. Dispatch the agent (runner = pi-as-mcp by default)

Read projects/<project>/tasks/<task>/task.md and pass its contents verbatim as the prompt. Map the task kind to capability:

• scout → tool_mode: "read-only"
• coder → tool_mode: "full"

ALWAYS give coder agents full, never write. In pi-as-mcp the "write" mode does not include bash, so the agent cannot build, compile, or run tests — it reaches for a run/shell tool that doesn't exist, can't self-verify, and either ships broken/uncompilable code or loops until it stalls (observed: 200+ turns repeating "let me run the tests"). full gives it bash so it can actually verify its work. (read-only already includes bash, so scouts are fine as-is.)

Call the pi-as-mcp delegate tool. The tool name depends on which orchestrator you are: in Claude Code it's mcp__pi-as-mcp__delegate; in Codex it's the pi-as-mcp MCP server's delegate tool (same arguments). pi-as-mcp must be registered as an MCP server for the current orchestrator (it already is in ~/.codex/config.toml).

delegate(
  prompt   = <contents of task.md>,
  cwd      = "/tmp/work/<run_id>",
  model    = "<model>",
  tool_mode= "read-only" (scout) | "full" (coder)
)

Note the start time (date -u +%Y-%m-%dT%H:%M:%SZ) and the returned agent_id. Run the returned monitor command in a background shell and wait for it to finish. Do not agent_reply mid-run — that contaminates the eval. (Other runners: see docs/runners.md; record the runner in metrics.json:run.framework.)

4. Collect the run

• Record end time → efficiency.wall_clock_seconds.
• agent_peek(agent_id, verbosity="debug") (pi-as-mcp; mcp__pi-as-mcp__agent_peek in Claude Code) to capture the agent's final answer + transcript. From it, fill efficiency.tool_calls_total, tool_calls_by_type, turns, and tokens if exposed (else null).
• For scout tasks, the final answer is the deliverable — save it into the report dir (e.g. agent_answer.md).

5. Capture the diff

lib/collect_diff.sh /tmp/work/<run_id> <report_dir>

Writes diff.zip and prints {files_touched, lines_added, lines_removed} → copy into efficiency.*. (Scout runs should produce an empty/near-empty diff.)

6. Review & score — the careful part

Read the diff (and the scout answer) against solution.md's must-have checklist. Where the task has an objective verify command (task.json), run it inside the worktree and record outcome.verifiable_pass / tests_passed.

Fill all of metrics.json per docs/metrics.md:

• outcome.* — completed, completion_level, correctness, tests.
• efficiency.* — from steps 4–5; judge efficiency_score.
• code_quality.* (coder) or investigation.* (scout) — leave the other block null.
• communication.*, proactivity.* (check nice_to_haves from task.json and sum bonus_points), safety.*.
• Use null, never a fake 0, for axes that don't apply.

Then compute the composite:

lib/score.py <report_dir>/metrics.json --write

This blends the per-kind weights, adds bonus points, and writes composite.weighted_total + grade + category_breakdown.

7. Write report.md

Fill the template with a real, evidence-backed evaluation: verdict, what the agent did (reconstructed from diff + transcript), correctness vs. expected, quality, efficiency, proactivity/nice-to-haves, safety, and a score justification that ties each judged number to concrete evidence. Cite file:line.

8. Clean up

lib/cleanup_worktree.sh <project> /tmp/work/<run_id>

The diff is already archived in the report; the worktree is disposable.

9. Finalize run metadata

Ensure metrics.json:run has model, framework, project, task, difficulty, kind, timestamp, worktree_id, base_commit, evaluator. run.task must be the task-dir name (e.g. hard-coder, polygon-cli-easy-scout) — campaign resume matches (model, project, task) against it, so a mismatch makes a finished cell look unrun. difficulty/kind are the last two hyphen-separated tokens of the task name.

Also fill run.model_config — the full typed model/serving config (quant, quant provider, sampling, context window, capabilities like vision/mtp/reasoning, hardware, pricing; see docs/metrics.md §0). The user typically pastes the serving config once per model — fill the fields you can confirm from it (and from the runner), leave the rest null. It's the same for every cell of a campaign, so reuse it across the whole run. Report the path to the new report and a one-line headline.

Principles

• Blind. Only task.md reaches the agent; cwd is the worktree; nothing meta.
• Serial. One run at a time, fully reviewed before the next. Careful review beats throughput.
• Stateful & resumable. A campaign's progress lives on disk (campaigns/<model>/), not in session memory. Tick each box as its report lands; an interrupted run resumes from the first unchecked cell. The checklist is the source of truth for "what's left", not your recollection.
• Honest scoring. Every judged number must be defensible from report.md. When unsure, say so in the report and score conservatively.
• Absolute scoring. Judge against the task's bar — solution.md's must-haves and the verify oracle — never relative to what you'd expect from a model of this size, quant, or price. A small/cheap/quantized model that half-solves the task scores exactly what any model scores for that same result. You know the model identity (run.model_config); do not let it move any axis. And "impressive for its size" goes nowhere — not in a judged number, not in report.md, not in composite.headline. Describe what it did, not how surprising it is for the weight class. If size/quant context matters, it lives only in run.model_config (the facts), never in the evaluation language.
• Reproducible. Each report is self-contained (frozen prompt + solution + diff + metrics), so changing tasks later never invalidates old reports.
• Timeouts. If an agent hangs, agent_stop(agent_id), note it in the report, and score what exists (likely outcome.completed=false).