benchmark

name

benchmark

description

Personal model benchmark. Replays the user's real recurring tasks (mined from their own Claude Code session history) against different models and effort levels in isolated headless sessions, grades the outputs blind against the user's own rubric, and produces a local HTML report plus a plain-verdict answer. Use whenever the user types /benchmark, asks to compare models ("opus vs fable", "opus 5 vs fable 5 on email triage, 3 trials"), asks if a new model is better or worth switching to, wants to compare effort levels (low vs high), or asks whether they can run a cheaper model/effort for the same result. Also handles "/benchmark mine" to build or refresh the test pack from history.

Personal Benchmark

Public benchmarks test someone else's tasks. This skill replays the user's own recurring work against candidate models so the verdict is about their life. The user's history is the eval set; the user's rubric is the judge's law.

Everything lives in this skill folder:

• rubric.md — the user-defined scoring rubric. Never invent quality criteria; read this file.
• testpack.json — mined task patterns. Created by the Mine phase.
• scripts/run_trial.ps1 — runs one isolated headless trial and captures all metrics.
• Results go to ~/.claude/benchmarks/<yyyyMMdd-HHmmss>/ (one folder per run).

Phase 0 — Parse the request

From the user's command extract:

• Models: resolve informal names to CLI values. fable/fable 5 → fable, opus/opus 5 → opus, sonnet → sonnet, haiku → haiku. A versioned older model ("opus 4.8") becomes a full ID like claude-opus-4-8; if unsure the ID is still valid, verify with one cheap ping (claude -p "say ok" --model <id> --output-format json) before planning the matrix, and tell the user if it's gone.
• Efforts: low|medium|high|xhigh|max, passed via --effort. If the user doesn't mention effort, omit the flag (session default) — don't invent pairs.
• Trials: default 1. Only more if the user says so ("3 trials").
• Tasks: if the user names a task ("just the executive email triage task"), match it against testpack.json names. Otherwise take the top 3 tasks by workload_pct. Never default to the whole pack — the user is token-frugal.
• /benchmark mine → skip to the Mine phase only.

Phase 1 — Test pack (Mine)

If testpack.json is missing, or the user asked to mine/refresh:

1. Spawn an Explore agent over ~\.claude\projects\*\*.jsonl (and ~\.codex\sessions\* if present). Sample the first few user messages of each session — that's where the task statement lives. Don't read whole transcripts; this is pattern mining, not archaeology.
2. Cluster into recurring task patterns (email drafts, repo audits, summaries, copywriting, data extraction, ...). For each, estimate what share of total sessions it represents.
3. For each pattern write a self-contained representative prompt: inline any needed input material (the email to triage, the code to review) directly into the prompt text. Trials run in empty temp folders, so a prompt that references files the trial can't see is a broken test.
4. Present the draft pack to the user for approval/edit before saving. The pack is their eval set; they know if a task is stale or missing.
5. Save as testpack.json:

{
  "mined_at": "2026-07-26",
  "tasks": [
    {
      "id": "exec-email-triage",
      "name": "Executive email triage",
      "workload_pct": 18,
      "prompt": "Full self-contained prompt text including inlined inputs...",
      "notes": "what a good output looks like, any task-specific pass criteria"
    }
  ]
}

Put anything task-specific a judge should know in notes — it gets appended to the rubric at grading time.

Phase 2 — Confirm before spending

Show the trial matrix (tasks × model/effort combos × trials) and the total session count, then get an explicit go via AskUserQuestion (Run as-is / Change scale / Cancel). Each cell is a full model run of a real task — real usage burn. If the matrix exceeds 12 sessions, say so prominently and suggest a smaller cut first. Never start runs without this confirmation.

Phase 3 — Run trials

Create the run folder ~/.claude/benchmarks/<timestamp>/ and write each task's prompt to prompts\<task-id>.txt. Then for each cell run:

& "$env:USERPROFILE\.claude\skills\benchmark\scripts\run_trial.ps1" `
  -Model <model> -Effort <effort-or-empty> `
  -PromptFile <run>\prompts\<task-id>.txt `
  -OutFile <run>\results\<task-id>__<model>__<effort>__t<n>.json

Rules that make the comparison fair (from the source: "each cell is an isolated real session — same prompt, same files, fresh context, no memory carried over"):

• One script call per cell; the script uses a fresh empty temp workdir and --no-session-persistence, so nothing leaks between trials.
• Launch cells with run_in_background so they run concurrently, but no more than ~4 at once — parallel sessions share the user's rate limits.
• A trial that errors or returns empty gets one retry, then is recorded as failed (shown in the report, excluded from averages).

Each result JSON (from claude --output-format json) contains result (the output text), usage (token counts), num_turns, duration_ms, and total_cost_usd. The script also writes a .meta.json sidecar with wall-clock time, model, and effort. Read fields defensively — treat missing ones as null, not zero.

Phase 4 — Blind grading

Self-grading is biased, and knowing which model wrote what is the bias. So:

1. For each task, collect the output texts and relabel them randomly as Output A, B, C... Keep the label→cell mapping yourself; the judge never sees model names, efforts, or metadata.
2. One judge call per task — a fresh headless session, same judge model for every task in the run (default fable, low effort is fine for grading):

Get-Content judge_prompt.txt -Raw | claude -p --model fable --effort low `
  --output-format json --no-session-persistence `
  --json-schema '{"type":"object","properties":{"scores":{"type":"array","items":{"type":"object","properties":{"label":{"type":"string"},"fidelity":{"type":"number"},"correctness":{"type":"number"},"brevity":{"type":"number"},"clarity":{"type":"number"},"voice":{"type":"number"},"missed_brief":{"type":"boolean"},"note":{"type":"string"}},"required":["label","fidelity","correctness","brevity","clarity","voice","missed_brief","note"]}}},"required":["scores"]}'

3. The judge prompt = full text of rubric.md + the task prompt + the task's notes + the anonymized outputs. Each sub-dimension scored 0–10.
4. Compute weighted quality per rubric.md. If missed_brief is true, cap quality at 4/10 regardless of sub-scores — a beautiful answer to the wrong question is a failure.

Phase 5 — Report and verdict

Compute per cell: quality /10, tokens (input+output from usage), speed (duration_ms, fall back to wall-clock), turns (num_turns), and efficiency = quality points per 1K tokens. Average across trials per cell.

Write report.html in the run folder — self-contained, no external assets — and open it with Start-Process. Structure:

1. Verdict banner — one plain-English sentence per comparison ("Fable low quietly wins", "Dead even — not worth switching").
2. Main table — rows = tasks, column group per model/effort: quality /10, tokens, speed, turns, efficiency. Best cell per row highlighted.
3. Per-task detail (collapsed) — the prompt tested, each output's judge note, and the anonymization mapping revealed.
4. Failed trials, if any, listed honestly.

Then print in the terminal: a ≤3-line verdict — winner, the tradeoff, and the recommendation (switch / stay / run cheaper effort). Plain language; the numbers live in the report.

"Dead even" is a useful result: it means don't migrate, and say so.

Cost guardrails

• Defaults: 3 tasks × 1 trial. The confirm step is never skipped.
• Remind the user of rough scale in the confirm step ("12 sessions ≈ 12 real task runs on your usage").
• If the user's plan is huge (>30 sessions), offer the frugal cut first and the full run as a follow-up on the shortlisted winner.