| name | marshal-benchmark |
| description | Compare AI coding strategies on a real task with Marshal: run the same goal through several configured clients (e.g. a premium model vs a cheaper one) and get a measured cost/latency/outcome comparison. Use when you want evidence for which provider/model to route a kind of work to, or to prove what a cheaper strategy actually saved. Honest by construction - it measures, it never guesses.
|
Benchmarking routing strategies
Marshal's differentiator is proving what a routing choice costs, with measured numbers - not a
fixed % claim. A strategy is a configured client (a backend + model + permission combo). This
playbook runs one goal through N strategies and compares them.
Run it
list_clients - pick the strategies (client names) to compare, e.g. a premium worker vs a cheap
one, both capable of the task.benchmark(goal, clients, task_id?, max_concurrency?) - runs the same goal through each client
in parallel (each in its own worktree), then returns a comparison:
strategies[] - per client: status, cost_usd, source, duration_ms, tokens, run_id.cheapest / fastest - the winning client, but only among strategies that succeeded and have
a known cost. A strategy whose cost is unavailable never wins "cheapest."
report(task_id) re-derives the same comparison later from the ledger (read-only) - reproducible
and auditable.
Multi-workspace: benchmark and report are scoped to one workspace, and task_ids are
per-workspace. If you benchmark with workspace=<name>, pass the same workspace to
report(task_id, workspace=<name>) - otherwise it reads the wrong repo's ledger and finds nothing.
Read it honestly
- Check
source on each row. native = the backend reported the cost; admin-api = the real
charge read back from a provider usage API (e.g. EastRouter) - also ground truth; estimated =
computed from tokens via the price table (only as good as the table - keep it current);
unavailable = unknown (e.g. Cursor without the Admin API, or an unpriced model). Never read an
unavailable $0 as free.
- The benchmark measures cost, latency, and outcome status - not correctness. A cheapest strategy
can still produce worse code. Review the diffs (
collect_run) before drawing conclusions.
empty / failed / verify_failed strategies still cost tokens; they appear in per-run cost but
are excluded from cheapest / fastest.
What to do with the result
- Route future work of this kind to the strategy with the best cost/quality trade-off (the cheapest
one that also produces correct diffs).
- To keep a strategy's output,
integrate its run_id (see marshal-orchestrate).
- The historical record of what each strategy cost - derived on read, never a stored guess - is the
defensible asset. Build it up over real tasks rather than promising a universal savings number.
