benchmark

name	benchmark
description	Multi-provider AI coding benchmark. Dispatches a task to Codex, Gemini, Claude, and Cursor-Agent in parallel, scores responses with a Claude meta-judge, and produces a ranked leaderboard with cost, time, quality, and self-eval bias metrics. Supports corpus mode (fixed OrderService + useSearchProducts tasks) for apples-to-apples comparison across runs.

zuvo:benchmark — Multi-Provider Coding Benchmark

Measures how well different AI coding agents handle a task. Dispatches to all available providers in parallel, collects responses, and uses an opposite-model Claude meta-judge to score on four dimensions (completeness, accuracy, actionability, no_hallucinations). Outputs a ranked leaderboard with cost and time breakdown.

Use corpus mode (--mode corpus) to compare runs across time. The corpus uses fixed OrderService + useSearchProducts tasks — the same prompt every time — so runs from different days are directly comparable.

Argument Parsing

Parse $ARGUMENTS for these flags:

Flag	Effect
--diff [ref]	Benchmark against a git diff (default: HEAD~1)
--files <path>	Benchmark on a file or newline-separated list of files as task input
--prompt <text>	Use a literal text prompt as the task (alias: --task)
--mode corpus	Use fixed corpus tasks (OrderService + useSearchProducts)
--mode default	Use user-provided task (default)
--with-tests	Run Round 2: have providers write tests for their own Round 1 code
--with-adversarial	Run adversarial cross-review on Round 1 code
--with-test-adversarial	Run adversarial cross-review on Round 3 tests (requires --with-tests)
--with-static-checks	Run tsc + jest on generated code (best-effort, null if tools missing)
--provider <name>	Restrict to one or more providers, comma-separated (alias: --providers)
--show-costs	Print provider cost table ($/M tokens) and exit
--compare [id1] [id2]	Compare two prior runs from zuvo/reports/; default: last two (skill-only, not passed to runner)
--replay-last	Re-run benchmark with the same task as the most recent run (skill-only, not passed to runner)
--json	Output raw JSON to stdout instead of formatted leaderboard (handled by both runner and skill)
--no-snapshot	Suppress task_snapshot storage (use if task contains secrets or PII)
--dry-run	Print prompt + providers without dispatching
(remaining text)	Treated as --prompt <text> if no other input flag given

Mandatory File Loading

Before starting work, read each file below. Print the checklist with status.

CORE FILES LOADED:
  1. ../../shared/includes/benchmark-output-schema.md  -- READ/MISSING
  2. ../../shared/includes/run-logger.md               -- READ/MISSING
  3. ../../shared/includes/retrospective.md               -- READ/MISSING
  3. ../../shared/includes/env-compat.md               -- READ/MISSING

If any core file is missing, proceed in degraded mode and note it in the BENCHMARK COMPLETE block.

---

Phase 0: Parse and Validate Arguments

1. Parse all flags from $ARGUMENTS per the table above.

2. --show-costs: run scripts/benchmark.sh --show-costs, print the table, stop. No benchmark run.

3. --compare: load the two specified run JSONs from zuvo/reports/ (or the last two if no IDs given). Print a delta table: quality, time_s, cost_usd per provider. Warn if task_hash values differ (different tasks, comparison may be misleading). Stop — no benchmark run.

4. --replay-last: find the most recent .json file in zuvo/reports/. Read task_snapshot as the task input. Set INPUT_MODE=prompt. Error if no history found. Error if --mode corpus (corpus uses fixed prompts, replay makes no sense).

5. Validate flag combinations:

   • --with-tests and --with-adversarial require --mode corpus (they assume a two-round structure). If set without --mode corpus, print a warning and auto-set --mode corpus.
   • --provider with a name not in the detected provider list → warn, but don't fail (provider may appear after dispatch attempt).

6. Determine task source:

   • --mode corpus → task_source: "corpus", load from ../../shared/includes/benchmark-corpus/task-code.md
   • --diff → task_source: "diff", input via git diff
   • --files → task_source: "files", input via file concatenation
   • --prompt or remaining text → task_source: "user", input is the literal text
   • None of the above → task_source: "diff", default to HEAD~1 (matches runner behavior)

7. Print run header:

   BENCHMARK RUN
   Mode: [default|corpus] | Task: [first 60 chars or "corpus"] | Providers: [list]
   Options: tests=[bool] adversarial=[bool] static=[bool]
   

---

Phase 1: Dispatch Providers

1. Find and run benchmark.sh. Search in order:

   • scripts/benchmark.sh (if in zuvo-plugin repo)
   • ~/.codex/scripts/benchmark.sh (Codex install)
   • ~/.cursor/scripts/benchmark.sh (Cursor install)
   • ~/.claude/plugins/cache/zuvo-marketplace/zuvo/*/scripts/benchmark.sh (Claude Code plugin cache)

   Capture stdout to $TMPDIR/benchmark-raw.json.

   scripts/benchmark.sh \
     --mode <mode> \
     [--prompt <text>|--files <path>|--diff <ref>] \
     [--with-tests] [--with-adversarial] [--with-test-adversarial] [--with-static-checks] \
     [--provider <list>] [--json] \
     --run-id <run_id> \
     --round-dir "$TMPDIR/rounds" \
     --output "$TMPDIR/benchmark-raw.json"
   

   Runner-only flags: --prompt, --files, --diff, --mode, --with-*, --provider, --output, --run-id, --round-dir, --json, --show-costs, --dry-run, --no-snapshot. Skill-only flags (NOT passed to runner): --compare, --replay-last. These are handled in Phase 0 before the runner is invoked.

2. Check exit code:

   • 0: success, proceed to Phase 2
   • 1: bad arguments → show error and stop
   • 2: no providers available → show install instructions and stop
   • 3: all providers failed → show error and stop

3. Print progress as providers complete (read from stderr in real time or wait for script exit):

   [1/4] claude — done (42s)
   [2/4] gemini — done (31s)
   [3/4] codex-fast — done (28s)
   [4/4] cursor-agent — timeout
   

---

Phase 2: Meta-Judge Scoring

Never skip this phase. Even if only one provider succeeded, still score it.

Model Selection (opposite-model rule)

The judge must be a different model than the one running this skill to reduce self-serving bias:

• If $CLAUDE_MODEL contains "opus" → use claude-sonnet-5
• Otherwise (sonnet, haiku, or unset) → use claude-opus-4-8

This is the opposite model from the one executing this skill.

Input Preparation (CQ6 — bounded input)

For each provider response in providers_raw:

1. Extract response_excerpt or the full response text
2. CQ6 truncation: limit each response to floor(80000 / provider_count) characters before including in the judge prompt. Set judge_input_truncated: true in output if any response was truncated.

Shuffle the presentation order randomly (store in judge_presentation_order) to reduce positional bias.

Judge Prompt

Send a single Claude call to the judge model with this prompt structure:

You are an objective code quality judge. Score each provider's response on four dimensions (0–5 each):

- completeness: All required methods, fields, and behaviors implemented
- accuracy: Logic is correct, state machines valid, edge cases handled
- actionability: Production-ready; no stubs, TODOs, or placeholder logic
- no_hallucinations: No invented APIs, non-existent methods, or fabricated behavior

TASK:
<task_snapshot or task prompt>

PROVIDER RESPONSES (in random order):
[provider ID redacted — labeled A, B, C, D]

<response A>
---
<response B>
...

Respond with JSON only, no prose:
{
  "A": { "completeness": N, "accuracy": N, "actionability": N, "no_hallucinations": N },
  "B": { ... },
  ...
}

Parse Judge Response

1. Extract JSON from the response. If jq . fails → mark run UNSCORED, skip to Phase 3 with null scores. Log the raw judge output for debugging.
2. Map labeled responses (A, B, C...) back to provider names using judge_presentation_order.
3. Compute per-provider:
   • code_composite = completeness + accuracy + actionability + no_hallucinations (0–20)
   • quality = round(code_composite * 5) (0–100) — code-only mode
   • self_eval_bias = self_eval_raw - code_composite — null if self_eval_raw is null

---

Phase 3: Assemble Leaderboard

Rank Providers

Sort by:

1. quality DESC (higher is better)
2. time_s ASC (faster wins ties)
3. cost_usd ASC (cheaper wins cost ties)
4. provider alphabetical (deterministic final tiebreaker)

Build Leaderboard Array

For each provider in ranked order:

{
  "rank": 1,
  "provider": "claude",
  "quality": 87,
  "code_score": 18,
  "test_score": null,
  "time_s": 42.1,
  "cost_usd": 0.031,
  "compile_ok": null,
  "tests_pass": null,
  "self_eval_bias": 1.2,
  "adversarial_delta": null,
  "status": "scored"
}

Build Scorecards Object

{
  "claude": {
    "code_completeness": 5,
    "code_accuracy": 4,
    "code_actionability": 5,
    "code_no_hallucinations": 4,
    "code_composite": 18,
    "test_completeness": null,
    "test_accuracy": null,
    "test_actionability": null,
    "test_no_hallucinations": null,
    "test_composite": null,
    "adversarial_delta": null,
    "test_adversarial_delta": null,
    "self_eval_bias": 1.2,
    "response_excerpt": "<first 500 chars of response>"
  }
}

Leaderboard Display

Print the leaderboard as a markdown table:

## Benchmark Results — [run_id]
Task: [first 60 chars] | Mode: [default|corpus]

| Rank | Provider     | Quality | Code | Tests | Time  | Cost    | Tokens     | Adv.Δ | Status |
|------|-------------|---------|------|-------|-------|---------|------------|-------|--------|
|  1   | codex-fast   |    85   |  17  |  17   | 345s  | $0.025  | ~12K/~8K   |  -2   | scored |
|  2   | gemini       |    75   |  15  |  15   | 291s  | $0.000  | ~11K/~9K   |  -4   | scored |
|  3   | claude       |    70   |  14  |  14   | 300s  | $0.062  | ~12K/~7K   |  -3   | scored |
|  4   | cursor-agent |    65   |  13  |  13   | 357s  |  —      | ~11K/~6K   |  -3   | scored |

Tokens column: input/output estimates (~ = estimated via word count × 1.3).
Self-eval bias (positive = overconfident): codex-fast +1, claude +4, cursor-agent +5, gemini +5
Adversarial delta (negative = score dropped after adversarial critique): codex-fast -2, claude -3, cursor-agent -3, gemini -4

---

Phase 4: Persist and Log

Output Files

Auto-increment the run number (NNN) by reading existing files in zuvo/reports/:

• zuvo/reports/benchmark-NNN-[run_id].md — human-readable report
• zuvo/reports/benchmark-NNN-[run_id].json — machine-readable JSON per benchmark-output-schema.md

The JSON file must conform to the schema version "2.0" defined in ../../shared/includes/benchmark-output-schema.md.

JSON Output Structure

{
  "version": "2.0",
  "skill": "benchmark",
  "run_id": "<run_id>",
  "timestamp": "<ISO-8601>",
  "project": "<project path>",
  "mode": "default",
  "task_source": "user",
  "task_hash": "<64-char SHA-256>",
  "task_snapshot": "<first 30000 chars>",
  "options": { "with_tests": false, "with_adversarial": false, "with_static_checks": false },
  "providers_attempted": ["claude", "gemini", "codex-fast"],
  "providers_succeeded": ["claude", "gemini", "codex-fast"],
  "scored": ["claude", "gemini", "codex-fast"],
  "leaderboard": [...],
  "scorecards": {...},
  "meta_judge_model": "claude-opus-4-8",
  "judge_presentation_order": ["gemini", "claude", "codex-fast"],
  "judge_input_truncated": false
}

Run Log

Run: <ISO-8601-Z>	benchmark	<project>	-	-	<VERDICT>	<providers_count>-providers	<mode>	<notes>	<BRANCH>	<SHA7>	<INCLUDES>	<TIER>

Retrospective (REQUIRED)

Follow the retrospective protocol from retrospective.md. Gate check → structured questions → TSV emit → markdown append. If gate check skips: print "RETRO: skipped (trivial session)" and proceed.

After printing this block, append the Run: line value (without the Run: prefix) to the log file path resolved per ../../shared/includes/run-logger.md.

VERDICT: PASS (all providers scored), PARTIAL (some failed), FAIL (all failed), UNSCORED (judge parse failed).

BENCHMARK COMPLETE Block

BENCHMARK COMPLETE
Run ID:    [run_id]
Mode:      [default|corpus]
Task:      [hash prefix] — [first 60 chars or "corpus tasks"]
Providers: [N attempted / M succeeded / K scored]
Judge:     [model name] (opposite-model rule)

Results saved:
  zuvo/reports/benchmark-NNN-[run_id].md
  zuvo/reports/benchmark-NNN-[run_id].json

Top provider: [name] (quality: [score], time: [Xs], cost: $[N])
Self-eval bias range: [min] to [max] (positive = overconfident)

---

Corpus Mode — Additional Phases (phases 5–8)

Corpus mode activates when --mode corpus is set. Phases 5–8 extend the default pipeline.

Architecture note: Phases 5–8 are orchestrator-managed — they are executed by the LLM running this skill, not by scripts/benchmark.sh. The bash runner handles Round 1 dispatch and raw result collection only. The skill orchestrator manages multi-round flow: reading round files from --round-dir, interpolating prompts, re-dispatching providers, and scoring.

See Phase 5 (adversarial round), Phase 6 (Round 2 test writing), Phase 7 (test scoring), and Phase 8 (corpus leaderboard) defined in the corpus mode extension below.

Corpus mode quality formula (both rounds): quality = round((code_composite + test_composite) * 2.5) → range 0–100

Default mode formula (code only): quality = round(code_composite * 5) → range 0–100

---

Corpus Mode Extension

This section only runs when --mode corpus is set.

Phase 5: Adversarial Cross-Review (if --with-adversarial)

Each provider's Round 1 output is stored as tmp/round1_<provider>.txt.

For each provider that succeeded in Phase 1:

1. Run scripts/adversarial-review.sh --files tmp/round1_<provider>.txt --json
2. Have the adversarial reviewers (other providers) challenge the code
3. Record the adversarial findings; provider may rewrite → tmp/round1_fixed_<provider>.txt
4. Compute adversarial_delta: re-score the post-adversarial output. Delta = fixed_score − original_score (typically negative: critiques lower scores).
5. CQ8: if adversarial fails for a provider, continue with unfixed round1_<provider>.txt output

Phase 6: Round 3 — Write Tests (if --with-tests)

Round 1 produces code. Round 3 produces tests for that code. (Round 2 is adversarial review — optional, can run between rounds.)

For each provider that succeeded Round 1:

1. Load ../../shared/includes/benchmark-corpus/task-tests.md
2. Interpolate {{ROUND_1_CODE}} with the provider's round1_<provider>.txt output
3. Dispatch the interpolated prompt back to the same provider
4. Capture Round 3 response (test code) → tmp/round3_<provider>.txt + TEST_EVAL_SUMMARY block

Phase 7: Round 4 — Adversarial on Tests (if --with-test-adversarial)

Mirrors Phase 5 but targets test files from Round 3.

Each provider's Round 3 output is stored as tmp/round3_<provider>.txt.

For each provider that succeeded Round 3:

1. Run scripts/adversarial-review.sh --files tmp/round3_<provider>.txt --json
2. Other providers critique: missing branches, weak assertions, echo tests, tautological oracles
3. Author may rewrite tests → tmp/round3_fixed_<provider>.txt
4. CQ8: if adversarial fails for a provider, continue with unfixed round3_<provider>.txt

test_adversarial_delta = meta-judge score of fixed tests − original tests (typically negative).

Phase 7b: Score Round 3 Responses

Use the same meta-judge model to score test quality on four dimensions (0–5 each):

• test_completeness: coverage of happy paths, error paths, edge cases
• test_accuracy: assertions are meaningful; mocks are correctly set up
• test_actionability: tests are runnable and follow project conventions
• test_no_hallucinations: no invented test utilities or non-existent matchers

test_composite = test_completeness + test_accuracy + test_actionability + test_no_hallucinations

Phase 8: Corpus Leaderboard

Assemble final leaderboard using the both-rounds quality formula: quality = round((code_composite + test_composite) * 2.5)

Include test_score, adversarial_delta, tests_pass in the leaderboard and scorecards.

Print extended leaderboard with all columns populated.