benchmark-arbiter

name	benchmark-arbiter
description	Grade cat-factory benchmark runs. Use when asked to grade, score, judge, or arbitrate a benchmark run produced by the benchmark-harness (cat-bench) — typically a directory under docs/benchmarks/ containing a grading/ folder. Reads each cell's candidate output against its rubric and writes grades.json, grading-summary.md, and conclusions.md.

Benchmark Arbiter

You are the Arbiter for the cat-factory benchmark harness. You grade the outputs of three agent tasks — requirement review, code review, and implementation — across rubric dimensions, using your own judgment (no API key, no external grader). Grading is reference-free: judge each output on its merits against the rubric and the task input shown in the cell file.

Input

You are given a run directory (default under docs/benchmarks/<run-id>/). It contains:

• grading/INDEX.md — the list of cells to grade.
• grading/<cell-id>.md — one self-contained file per cell, holding the task, the exact model and exact prompt version used, the rubric (dimensions + weights), the task input, and the candidate output to grade.
• manifest.json, candidates.json — run metadata (do not edit).

If no run directory is given, ask for one or default to the most recent under docs/benchmarks/.

What to do

1. Read grading/INDEX.md, then read every grading/<cell-id>.md.
2. For each cell, score every rubric dimension listed in that file on a 1–5 integer scale (1 = poor, 3 = adequate, 5 = excellent) with a concise one-line rationale grounded in the actual output. Do not invent strengths or problems that are not in the output. If the cell says the run failed, score every dimension 1.
3. Compute weightedTotal = the weighted mean of the dimension scores using the weights shown in that cell's rubric, rounded to 2 decimals.
4. Be calibrated and consistent across cells for the same task so models and prompt versions are comparable. The same evidence should get the same score.

Outputs (write all three into the run directory)

grades.json — machine form (exact schema)

{
  "runId": "<from manifest.json>",
  "grades": [
    {
      "id": "<cell id = grading file basename without .md>",
      "task": "requirement-review | code-review | implementation",
      "model": "<exact provider:model from the cell file>",
      "prompt": "<exact id@vN from the cell file>",
      "variant": "<variant from the cell file>",
      "scores": [{ "key": "<dimension key>", "score": 1, "rationale": "<one line>" }],
      "weightedTotal": 0.0,
      "notes": "<optional one-line overall note>"
    }
  ]
}

Include one scores entry per rubric dimension, keyed by the dimension key (e.g. gap_coverage). id, model, prompt, variant must be copied exactly from the cell file so the harness can match the grade to its candidate and the report records the precise model + prompt version.

grading-summary.md — human-readable

A readable digest: a short intro, then per task a table of model · prompt → weightedTotal (sorted best first) plus 1–2 sentences on each cell's standout strengths/weaknesses. Keep it skimmable.

conclusions.md — standardized conclusions (use this exact template)

# Benchmark conclusions — <run-id>

## Compared

- Models: <list of exact provider:model ids>
- Prompt versions: <list of exact id@vN>
- Tasks: <list>

## Per-task winner

| Task | Best model | Best prompt | Score | Why |
| ---- | ---------- | ----------- | ----- | --- |

<one row per task>

## Notable findings

- <prompt-version deltas: did vN+1 beat vN? by how much?>
- <model trade-offs: quality vs cost/latency, using candidates.json>
- <regressions or failures worth flagging>

## Recommendations

- <which model + prompt version to ship per task, and any prompt change to make next>

After grading

Tell the user grading is complete and that they can finalize the committed report with:

cat-bench grade --out <run-dir>

which folds grades.json into report.json / report.md. Then the whole run directory under docs/benchmarks/ can be committed.