agent-eval

name	agent-eval
description	Head-to-head comparison of coding agents (Claude Code, Aider, Codex, etc.) on custom YAML-defined tasks with pass rate, cost, time, and consistency metrics. Use when choosing between coding agents for a codebase, benchmarking an agent after a model or tooling update, or producing data-backed agent-selection decisions for a team. Do NOT use for grading a single fix or claim (that is adversarial-verification) or for eval-driven development of Claude Code sessions and prompts themselves (that is eval-harness).
origin	ECC
tools	Read, Write, Edit, Bash, Grep, Glob

Agent Eval Skill

A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it.

When to use

• Comparing coding agents (Claude Code, Aider, Codex, etc.) on your own codebase
• Measuring agent performance before adopting a new tool or model
• Running regression checks when an agent updates its model or tooling
• Producing data-backed agent selection decisions for a team

When NOT to use:

• Grading whether a single fix, claim, or result actually holds — that is the adversarial-verification skill.
• Eval-driven development of Claude Code sessions, prompts, or skills (pass@k on your own workflows rather than head-to-head agent comparison) — that is the sibling eval-harness skill at .claude/skills/eval-harness/SKILL.md.
• Product imagery or render QC — that is the QC-judge pipeline, not a coding-agent benchmark.

Inputs

All of these must exist before starting. Any one absent → stop and report the missing input — never substitute a hand-rolled comparison loop and label it an agent-eval result.

1. The agent-eval CLI on PATH. Observed absent on this machine 2026-07-29 (command -v agent-eval → exit 1) [repro]. Install from the repository in Links only after reviewing its source.
2. A target git repository with a pinned commit SHA, so every worktree starts from identical state.
3. ≥1 task YAML (template below) containing ≥1 deterministic judge (pytest, build command). A task whose only judge is an LLM is not a valid task definition.
4. Every agent under test installed and authenticated — its own CLI on PATH, API keys loaded from env, never hardcoded.

Core Concepts

YAML Task Definitions

Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success:

name: add-retry-logic
description: Add exponential backoff retry to the HTTP client
repo: ./my-project
files:
  - src/http_client.py
prompt: |
  Add retry logic with exponential backoff to all HTTP requests.
  Max 3 retries. Initial delay 1s, max delay 30s.
judge:
  - type: pytest
    command: pytest tests/test_http_client.py -v
  - type: grep
    pattern: "exponential_backoff|retry"
    files: src/http_client.py
commit: "abc1234"  # pin to specific commit for reproducibility

Git Worktree Isolation

Each agent run gets its own git worktree — no Docker required. This provides reproducibility isolation so agents cannot interfere with each other or corrupt the base repo.

Metrics Collected

Metric	What It Measures
Pass rate	Did the agent produce code that passes the judge?
Cost	API spend per task (when available)
Time	Wall-clock seconds to completion
Consistency	Pass rate across repeated runs (e.g., 3/3 = 100%)

Workflow

1. Define Tasks

Create a tasks/ directory with YAML files, one per task:

mkdir tasks
# Write task definitions (see template above)

2. Run Agents

Execute agents against your tasks:

agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3

Each run:

1. Creates a fresh git worktree from the specified commit
2. Hands the prompt to the agent
3. Runs the judge criteria
4. Records pass/fail, cost, and time

3. Compare Results

Generate a comparison report:

agent-eval report --format table

Task: add-retry-logic (3 runs each)
┌──────────────┬───────────┬────────┬────────┬─────────────┐
│ Agent        │ Pass Rate │ Cost   │ Time   │ Consistency │
├──────────────┼───────────┼────────┼────────┼─────────────┤
│ claude-code  │ 3/3       │ $0.12  │ 45s    │ 100%        │
│ aider        │ 2/3       │ $0.08  │ 38s    │  67%        │
└──────────────┴───────────┴────────┴────────┴─────────────┘

Judge Types

Code-Based (deterministic)

judge:
  - type: pytest
    command: pytest tests/ -v
  - type: command
    command: npm run build

Pattern-Based

judge:
  - type: grep
    pattern: "class.*Retry"
    files: src/**/*.py

Model-Based (LLM-as-judge)

judge:
  - type: llm
    prompt: |
      Does this implementation correctly handle exponential backoff?
      Check for: max retries, increasing delays, jitter.

Verification

The report table is the tool's self-assessment, not evidence. After a comparison completes, run each of these — every one can return "no":

command -v agent-eval
git -C "$TARGET_REPO" worktree list
git -C "$TARGET_REPO" status --porcelain
pytest tests/test_http_client.py -v

1. command -v agent-eval — PASS: prints a path and exits 0. [repro] If it exits 1, no tool ran, so any "results" you are holding came from somewhere else — discard them (fail-open pattern, bug-230).
2. git -C "$TARGET_REPO" worktree list — PASS: one worktree per agent-run, each rooted at the pinned commit. [repro]
3. git -C "$TARGET_REPO" status --porcelain on the base repo — PASS: empty output; agents wrote only inside their own worktrees. [repro]
4. Re-run the winning row's deterministic judge yourself inside that agent's worktree, substituting the task's own judge command (for the template task: pytest tests/test_http_client.py -v) — PASS: your independently observed verdict matches the report row. [test] At least one re-derivation per report; a table you never spot-checked is [inferred].

Judge falsifiability (rule 3 of docs/skill-authoring-standard.md): before trusting any task, run its judge in a fresh worktree at the pinned commit before any agent touches it. PASS: the judge is RED there — the task is genuinely unsolved at base. A judge already green on the unmodified repo can never fail, so every agent "solves" it.

Worked example

A real invocation on this machine (2026-07-29), exercising the Inputs gate:

$ command -v agent-eval; echo "exit=$?"
exit=1

Observed output: exit=1 — the CLI is not installed here [repro]. The correct response was to stop at Inputs and report the blocker, which is what this section records: no comparison was run and none is claimed. Once installed, a run against the template task is invoked as:

agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3
agent-eval report --format table

Command shape is from the upstream README in Links [docs] — re-verify flags against agent-eval --help after installing, before citing any of them.

Failure modes

• Fail-open preflight (bug-230, ×6 in this repo). The tool is missing or the run crashed, and an empty/zero-row report gets read as "no differences between agents". An errored gate's output is an artifact — re-run and demand exit 0 before reading any table.
• Vacuous judge. A grep judge matches the prompt text an agent echoed into a comment, or the judge is already green at the pinned commit. Guard: prove RED-at-base before the eval (Verification, last step).
• Single-run comparison. Agents are non-deterministic; one run per agent makes the consistency column meaningless. Minimum 3 runs.
• Unpinned commit. The base repo drifts between runs, so results across days are not comparable. Pin commit: in every task YAML.
• Worktree pollution (bug-231 pattern: shared-state contamination). An agent writes outside its worktree, corrupting the base repo or a sibling run's results. Guard: Verification step 3; also never use a shared stash stack across eval worktrees.
• Cost apples-to-oranges. Some agent CLIs report no spend; comparing a measured cost against "n/a" is [inferred], not measured. Annotate missing columns — never fill them in.
• LLM-judge drift. Model-based judges change verdicts across model versions. Keep at least one deterministic judge per task and treat LLM judgments as advisory.

Best Practices

• Start with 3-5 tasks that represent your real workload, not toy examples
• Run at least 3 trials per agent to capture variance — agents are non-deterministic
• Pin the commit in your task YAML so results are reproducible across days/weeks
• Include at least one deterministic judge (tests, build) per task — LLM judges add noise
• Track cost alongside pass rate — a 95% agent at 10x the cost may not be the right choice
• Version your task definitions — they are test fixtures, treat them as code

Links

• Repository: github.com/joaquinhuigomez/agent-eval