| name | bench-routing |
| description | Rank published benchmark entries (Artificial Analysis Intelligence Index), recommend a model+effort per orchestration role given what each harness can actually dispatch, and check that recommendation against this repo's OWN measured routing outcomes. Use when the user asks whether a new/higher model should replace what a role currently runs on, wants a benchmark-informed routing recommendation, or asks "should we upgrade X to Y for this role" — the answer must be checked against the ledger, not just the benchmark. |
| allowed-tools | Bash, Read |
Bench routing
Run the engine that ships with this plugin. Use the ${CLAUDE_PLUGIN_ROOT} env var Claude Code
sets for plugin-executed content; if it is unset, fall back to resolving
../../bin/bench_routing.py relative to this SKILL.md to an absolute path.
python3 "${CLAUDE_PLUGIN_ROOT}/bin/bench_routing.py" rank --top 10
python3 "${CLAUDE_PLUGIN_ROOT}/bin/bench_routing.py" roles --harness claude
python3 "${CLAUDE_PLUGIN_ROOT}/bin/bench_routing.py" compare
python3 "${CLAUDE_PLUGIN_ROOT}/bin/bench_routing.py" demo
The dataset it reads, ${CLAUDE_PLUGIN_ROOT}/data/benchmarks.aa.json, is a screenshot-transcribed
snapshot of the Artificial Analysis Intelligence Index — a general-capability composite, not this
repo's pricing and not a bill. Never edit that file from this skill.
Subcommands
rank — ranked tables from the benchmark file: by intelligence index, by VALUE (index /
usd_per_task — a ranking ratio, never a price), and a "capability frontier" (cheapest entry
clearing each index floor). --provider P, --model M, --top N.roles — per-harness role recommendation (--harness claude|codex|copilot|all).
Availability is derived at run time from data/pricing.json / pricing.codex.json /
pricing.copilot.json — a benchmark entry that matches no available model is reported
UNAVAILABLE, never silently dropped. Role floors are this repo's own editorial judgement
(ROLE_POLICY in the script), overridable with a repeatable --floor "role=N".compare — THE ONE TO REACH FOR when the question is "should we route role X to a
different/higher model." Joins each roles recommendation (Claude harness only — the only
harness whose models map onto this repo's own haiku/sonnet/opus/frontier tiers) against this
repo's own measured first-try rate per tier (reusing bin/routing_scorecard.py's ledger, never
re-parsed) — but only for the role that ledger actually evidences. This repo's ledger
records per-TASK outcomes, which is implementer work; it carries no per-role result data for
architect/planner, reviewer, orchestrator, or verifier, so those roles are reported
no_role_evidence (benchmark pick stands unchallenged, no rate is quoted) rather than judged
on a number that measures a different job function. Only implementer gets a real verdict:
states plainly whether the ledger CONTRADICTS the benchmark's implied upgrade, and reports the
MARGINAL GAIN — not just both numbers. Measured outcomes win. A nominally-positive gain
that is small (e.g. "opus already clears ~100% vs sonnet ~97%") is reported as not_supported,
because a few points don't justify the routing change. Where a tier has too few finished
tasks, it says "insufficient sample" rather than guessing. mechanical sweep gets
cheapest_tier when its pick lands on the cheapest repo tier — a tier-ladder fact, not a
borrowed outcome claim.demo — synthetic smoke, no real files touched. Use to sanity-check the tool itself.
Presenting the results
- Lead with
compare's verdict, not the raw benchmark ranking — the benchmark is a prior;
the ledger is what this repo has actually observed. If compare says not_supported, say so
plainly and name the marginal gain; do not soften it into "might be worth considering."
- Never present
usd_per_task as a bill. It is a benchmark-workload cost for relative
ranking only. This repo's actual dollars come from routing_scorecard / cost_report, a
different measurement — never add the two together.
- Surface coverage gaps. If the model
compare recommends is measured at fewer effort
points than the tier-mate it would displace (the tool computes and reports this — e.g. an
"Opus at medium beats Sonnet" claim when Sonnet has no medium-effort benchmark entry), repeat
that caveat: the comparison covers only the effort points actually measured.
- State the index's scope. The Intelligence Index is a general-capability composite; Coding
Index and Agentic Index are separate boards not included here. A strong entry is not
automatically strong at agentic tool use — say this before recommending a routing change for
an agentic role (implementer, verifier, orchestrator, mechanical sweep).
- Name the provenance. The benchmark file is screenshot-transcribed (see its
cached_date /
transcribed_from), not an API export — flag it as re-verify-worthy if it looks stale.
- Never borrow implementer's rate for another role. If
compare says no_role_evidence for
architect/planner, reviewer, orchestrator, or verifier, do not paraphrase it as "the ledger
supports/contradicts this" — say plainly that this repo has no per-role evidence for that role
(the ledger only measures implementer task outcomes) and the benchmark recommendation stands
unchallenged either way.
