| name | shellbench-research-runbook |
| description | Plan, smoke-test, execute, checkpoint, publish, audit, and reproduce full ShellBench native benchmark campaigns across OpenClaw, Hermes, Codex, and Claude Code, including model and reasoning identity, pinned harness versions, n=3 qualification through n=6 research runs, S3 trace retention, and task-turn-tool-token-cost exports. |
ShellBench Research Runbook
Use this skill for a real benchmark campaign, not a one-off local score.
Read references/runbook.md before provisioning machines.
It is the normative campaign contract and contains the commands, gates, artifact
schema, and recovery rules.
Non-negotiable gates
- Use remote Crabbox AWS beasts for benchmark execution. Never run scored
trials on the operator laptop.
- Pin one public-task commit, runner commit or patch hash, provider model ID,
harness version, reasoning level, and judge route for the whole campaign.
- Run one
r0 qualification for every distinct harness and model-family
route, using exactly ten pinned representative tasks. Do not start full-suite
jobs until model identity, real traces, tools, usage, judge routing, and
artifact export pass.
- Retain and audit every
r0, but force it out of leaderboard scoring. Qualify
with independent full-suite repetitions r1 through r3. After a clean
audit, add r4 through r6; the research result is six total repetitions.
- Run every provider-supported non-maximum reasoning level. Never label a
reasoning level as tested unless the route applies it and the trace or proxy
evidence proves it. Record unsupported levels instead of fabricating them.
- Use
gpt-5.6-sol at high as the default judge. Keep the judge alias,
credentials, logs, and identity audit separate from the agent route.
- Start checkpointing after the first completed trial and continue at least
every ten minutes or ten new results. Verify each local archive before it
counts.
- Upload every verified checkpoint and final archive to the private S3 prefix
from
SHELLBENCH_TRACE_S3_URI. Never put bucket names or credentials in git,
PR text, public logs, or generated reports.
- A run is not research-clean when traces are missing, observed model identity
differs from the request, reasoning is unproven, coverage is incomplete, or
infrastructure failures dominate.
Required commands
Generate reasoning-specific plans with unique labels:
python -m scripts.native_eval.plan \
--tasks-root "$TASKS_ROOT" \
--output "$CAMPAIGN/manifests/run-index-high.json" \
--public-tasks-commit "$PUBLIC_TASKS_COMMIT" \
--run-date "$RUN_DATE" \
--reasoning-effort high \
--judge-model-id gpt-5.6-sol \
--judge-reasoning-effort high \
--repetitions 3
Use repeatable --harness and --model filters for r0 or phased plans.
Set --repetitions 6 only after the first three repetitions pass qualification.
Generate each family/harness r0 separately:
python -m scripts.native_eval.plan \
--tasks-root "$TASKS_ROOT" \
--output "$CAMPAIGN/manifests/r0-openclaw-gpt56.json" \
--public-tasks-commit "$PUBLIC_TASKS_COMMIT" \
--run-date "$RUN_DATE" \
--phase r0 \
--qualification-family gpt-5.6 \
--harness openclaw \
--model gpt56-sol \
--reasoning-effort high \
--judge-model-id gpt-5.6-sol \
--judge-reasoning-effort high \
--task "<task-01>" \
--task "<task-02>" \
--task "<task-03>" \
--task "<task-04>" \
--task "<task-05>" \
--task "<task-06>" \
--task "<task-07>" \
--task "<task-08>" \
--task "<task-09>" \
--task "<task-10>"
The planner marks r0 leaderboard_eligible=false with exclusion reason
r0_non_scoring_qualification.
After extraction, produce the research tables and strict identity report:
python -m scripts.native_eval.research_audit \
--run-index "$CAMPAIGN/manifests/run-index.json" \
--extracted-root "$CAMPAIGN/extracted" \
--output-dir "$CAMPAIGN/summaries/research"
Treat any false row in model_identity_audit.csv as a blocker. The exporter
does not invent prices: missing exact spend remains explicitly unavailable.
Stop conditions
- Stop a route after any r0 contains the wrong or multiple model IDs.
- Stop when the judge's observed provider model cannot be proven.
- Stop when a harness version differs from its campaign pin.
- Preserve and exclude infra-dominated runs, then rerun the same repetition
with a suffix and lower concurrency.
- Never delete, overwrite, or silently replace an artifact or repetition.
