一键在 Manus 中运行任何 Skill

marshal-benchmark

星标0

分支0

更新时间2026年7月17日 17:30

Compare AI coding strategies on a real task with Marshal: run the same goal through several configured clients (e.g. a premium model vs a cheaper one) and get a measured cost/latency/outcome comparison. Use when you want evidence for which provider/model to route a kind of work to, or to prove what a cheaper strategy actually saved. Honest by construction - it measures, it never guesses.

安装

用 Codex 或 Claude 帮你安装复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。

在 Manus 中运行

来源

chiruu12

chiruu12/marshal

打开 GitHub 仓库查看创作者相关仓库

下载

在 Manus 中运行

Benchmarking routing strategies

Marshal's differentiator is proving what a routing choice costs, with measured numbers - not a fixed % claim. A strategy is a configured client (a backend + model + permission combo). This playbook runs one goal through N strategies and compares them.

Run it

list_clients - pick the strategies (client names) to compare, e.g. a premium worker vs a cheap one, both capable of the task.
benchmark(goal, clients, task_id?, max_concurrency?) - runs the same goal through each client in parallel (each in its own worktree), then returns a comparison:
- strategies[] - per client: status, cost_usd, source, duration_ms, tokens, run_id.
- cheapest / fastest - the winning client, but only among strategies that succeeded and have a known cost. A strategy whose cost is unavailable never wins "cheapest."
report(task_id) re-derives the same comparison later from the ledger (read-only) - reproducible and auditable.

Multi-workspace: benchmark and report are scoped to one workspace, and task_ids are per-workspace. If you benchmark with workspace=<name>, pass the same workspace to report(task_id, workspace=<name>) - otherwise it reads the wrong repo's ledger and finds nothing.

Read it honestly

Check source on each row. native = the backend reported the cost; admin-api = the real charge read back from a provider usage API (e.g. EastRouter) - also ground truth; estimated = computed from tokens via the price table (only as good as the table - keep it current); unavailable = unknown (e.g. Cursor without the Admin API, or an unpriced model). Never read an unavailable $0 as free.
The benchmark measures cost, latency, and outcome status - not correctness. A cheapest strategy can still produce worse code. Review the diffs (collect_run) before drawing conclusions.
empty / failed / verify_failed strategies still cost tokens; they appear in per-run cost but are excluded from cheapest / fastest.

What to do with the result

Route future work of this kind to the strategy with the best cost/quality trade-off (the cheapest one that also produces correct diffs).
To keep a strategy's output, integrate its run_id (see marshal-orchestrate).
The historical record of what each strategy cost - derived on read, never a stored guess - is the defensible asset. Build it up over real tasks rather than promising a universal savings number.

同仓库更多 Skills

同仓库

marshal-orchestrate

chiruu12/marshal

Drive a fleet of headless coding agents through Marshal's MCP server: decompose a goal into independent tasks, run them in parallel in isolated git worktrees, review each diff, and integrate the good ones. Use when you have a multi-part coding goal to delegate to worker agents (Cursor, OpenCode, Codex, Antigravity, Claude Code) instead of doing it all yourself. The engine is mechanism; this playbook is the judgment - decomposition, prompt-writing, and merge decisions live here.

2026-07-210

marshal-workflow

chiruu12/marshal

Author and run declarative YAML workflows on the Marshal fleet - reusable orchestration recipes (fan a goal out across clients, collect the diffs, then merge the good ones) that you run by name instead of re-planning each time. Use when the same multi-agent shape recurs (review, compare, fix-across-clients). For a one-off decomposition, use marshal-orchestrate instead. The engine runs the recipe by sequencing safe primitives; the judgment - which recipe, and which diff to keep - stays with you.

2026-07-170

marshal-review-gate

chiruu12/marshal

Gate a candidate diff behind an independent, multi-reviewer consensus before integrating it. Spawn several biased reviewers (architect, quality, tests) that each judge the SAME diff in isolation, then apply a fixed truth table - integrate only when no reviewer rejects and at least one approves. Use when a single-pass review (marshal-orchestrate step 4) is not enough: a high-stakes merge, or when you want a quorum instead of one opinion. The engine runs the agents; this playbook is the judgment.

2026-07-170

marshal-plan-consensus

chiruu12/marshal

Reach a consensus on the APPROACH before committing a fleet to build it. Spawn several biased, independent solvers that each propose a plan for the same question, then an independent judge that converges them into one concrete plan (or narrows the question and re-runs). Use when the approach is contested - multiple valid designs, ambiguous scope, or a costly wrong turn - not for obvious work. The engine runs the agents; this playbook is the judgment. Hand the converged plan to marshal-orchestrate to build it.

2026-06-230

name	marshal-benchmark
description	Compare AI coding strategies on a real task with Marshal: run the same goal through several configured clients (e.g. a premium model vs a cheaper one) and get a measured cost/latency/outcome comparison. Use when you want evidence for which provider/model to route a kind of work to, or to prove what a cheaper strategy actually saved. Honest by construction - it measures, it never guesses.

Benchmarking routing strategies

Run it

list_clients - pick the strategies (client names) to compare, e.g. a premium worker vs a cheap one, both capable of the task.
benchmark(goal, clients, task_id?, max_concurrency?) - runs the same goal through each client in parallel (each in its own worktree), then returns a comparison:
- strategies[] - per client: status, cost_usd, source, duration_ms, tokens, run_id.
- cheapest / fastest - the winning client, but only among strategies that succeeded and have a known cost. A strategy whose cost is unavailable never wins "cheapest."
report(task_id) re-derives the same comparison later from the ledger (read-only) - reproducible and auditable.

Multi-workspace: benchmark and report are scoped to one workspace, and task_ids are per-workspace. If you benchmark with workspace=<name>, pass the same workspace to report(task_id, workspace=<name>) - otherwise it reads the wrong repo's ledger and finds nothing.

Read it honestly

Check source on each row. native = the backend reported the cost; admin-api = the real charge read back from a provider usage API (e.g. EastRouter) - also ground truth; estimated = computed from tokens via the price table (only as good as the table - keep it current); unavailable = unknown (e.g. Cursor without the Admin API, or an unpriced model). Never read an unavailable $0 as free.
The benchmark measures cost, latency, and outcome status - not correctness. A cheapest strategy can still produce worse code. Review the diffs (collect_run) before drawing conclusions.
empty / failed / verify_failed strategies still cost tokens; they appear in per-run cost but are excluded from cheapest / fastest.

What to do with the result

Route future work of this kind to the strategy with the best cost/quality trade-off (the cheapest one that also produces correct diffs).
To keep a strategy's output, integrate its run_id (see marshal-orchestrate).
The historical record of what each strategy cost - derived on read, never a stored guess - is the defensible asset. Build it up over real tasks rather than promising a universal savings number.