| name | geobench-run |
| description | Run the GeoBench 30-task coding-agent suite for one model×harness combination. Use when the user asks to run GeoBench, run the benchmark, evaluate a model, grade a suite, or invokes /geobench-run. Dispatch to geobench-run-codex, geobench-run-claude, geobench-run-grok, geobench-run-cursor, geobench-run-pi, geobench-run-opencode, or geobench-run-aiterator for harness-specific flags.
|
GeoBench suite run
One leaderboard row is one model × harness combination. Reset first. Never reuse a workspace across combos.
Human reference: MANUAL_RUN.md. Command-tab scripts at http://127.0.0.1:8765 stop on the first non-zero exit. A full suite must continue after a failed or timed-out task.
Use Git Bash. Scripts start as bash geobench-….sh from Git Bash, not cmd.exe. Agents run inside the geobench-agent Docker container; Docker Desktop must be running (docker info succeeds) and the image built (docker build -t geobench-agent docker). Host paths going into docker -v want Windows form (cygpath -w); scripts must export MSYS_NO_PATHCONV=1 so Git Bash does not rewrite in-container paths like /work.
Dispatch
Read the matching skill, then use this file for reset, isolation, grade, and import.
| User / CLI | Skill | --harness |
|---|
Codex / codex | geobench-run-codex | codex |
Claude Code / claude | geobench-run-claude | claude-code |
Grok CLI / grok | geobench-run-grok | grok |
Cursor Agent / cursor-agent | geobench-run-cursor | cursor-agent |
Pi / pi | geobench-run-pi | pi |
OpenCode / opencode | geobench-run-opencode | opencode |
aiterator / aiter exec | geobench-run-aiterator | aiterator |
If the binary is missing or the model cannot authenticate, skip that combo and say so. Do not invent a substitute model.
Reset
From the repo root, before a new combo:
python scripts/generate_tasks.py
Do not run this after the agent has started. Do not regenerate between tasks of the same combo.
Isolation
The evaluated agent may see only tasks/<id>/instruction.md and tasks/<id>/workspace/.
Never give it hidden/, gold/, verify.py, grader output, or another combo's workspace. Do not copy agent output into hidden/ or gold/.
Default: run the agent inside the geobench-agent container (build, login, and env details in docker/README.md; per-harness container commands in MANUAL_RUN.md section 0). Mount only the task workspace and instruction:
docker run --rm --security-opt seccomp=unconfined \
-v geobench-agent-home:/home/agent \
-v "$(cygpath -w "$root/tasks/<id>/workspace"):/work" \
-v "$(cygpath -w "$root/tasks/<id>/instruction.md"):/instruction.md:ro" \
-w /work geobench-agent <agent command>
Inside the container the workspace is /work, the instruction is /instruction.md, and the harness flags are the same as the host forms in the per-harness skills. Pi needs -e OPENROUTER_API_KEY (and -e LLAMA_BASE_URL=http://host.docker.internal:8080 for llama.cpp on this host). For Pi llama.cpp combos, the shared runner runs its model preflight before timing starts so headless Pi has a persisted loaded-model catalog. The shared runner can hydrate the OpenRouter key from the existing Pi auth record in geobench-agent-home when the shell variable is unset; it never prints the secret. The workspace is edited in place through the mount, so grading is unchanged. Host-native invocation is the fallback when Docker is unavailable — say so in the run notes. aiterator has no container install; run it host-native.
Continue remaining tasks if one attempt fails or times out (default 3600s).
Windows Docker Desktop operational notes
On this Windows host, keep MSYS_NO_PATHCONV=1 for Docker invocations and use
Windows-form paths for absolute arguments passed to the Windows Python
executable. The suite template converts timing paths with cygpath -w and
strips CRLF from task names discovered through Python.
Codex CLI 0.147.0 bundles a bwrap sandbox. Docker's default seccomp profile
blocks its user-namespace setup here, so the shared runner includes
--security-opt seccomp=unconfined on every Docker task container. Keep the
container unprivileged, retain Codex --approve-for-me, and do not add a Codex
--sandbox flag.
Use the persistent geobench-agent-home volume for CLI authentication and run
one disposable --rm task container at a time. Never remove that volume. If an
outer runner is interrupted, verify docker ps and terminate stale runner
processes before removing only their named task containers. The runner's
bounded retry loop handles transient provider/network errors; a persistent
rate limit or token/session stop leaves a partial timing/log checkpoint and
must be resumed as the same model×harness item after a fresh reset.
Host-side Go grading on this machine can fail in the default cache with
Access is denied. Set GOCACHE to a writable repository-local .gocache before
the timed verify step.
Logs: results/agent-logs/<sanitized-model>--<harness>[--effort]--<yyyyMMdd-HHmmss>/. Provenance only; the grade is the edited workspace/. Treat a combo as complete only when its timing is finished and its matching results/verify-*.json identifies the same model, harness, and knob. Do not advance a queue item from a partial workspace.
Wall time
The leaderboard records per-task wall time, and the grader reads it from the run's timing file. Start the suite runner from scripts/run_suite.template.sh: copy it to results/agent-logs/run-<combo>.sh, fill the CONFIGURE block (harness, model, notes, effort/variant) — the per-harness docker commands are already in the template — and everything else (the timing.json boundary writes via harness/timing.py, timeout and stall handling, container cleanup, and the timed grade) is already correct. Run it with bash run-<combo>.sh from Git Bash.
If writing a loop from scratch instead, call python harness/timing.py at every boundary: start before the first task, task-start / task-finish --status ok|failed|timeout|stall|rate_limited --exit-code N --attempts N around each task, and finish at the end. A provider rate-limit checkpoint finishes the current task as rate_limited, finishes the suite as aborted --reason rate_limited, and must not be graded as a complete result. A task that never finished stays open; the grader skips it instead of guessing. Legacy status.json files from older PowerShell runners are still accepted by --timing-file.
Tasks
Every tasks/<id>/ directory that contains task.yaml, sorted by its explicit order field. The directory name is the stable task id (device_manifest, not 05_device_manifest).
Grade
python -m harness.run verify --all --harness <harness> --model "<model-id>" --notes "<knob>" --timing-file "<log-dir>/timing.json"
--notes records an extra knob (effort high, thinking high, variant high). Omit when there is none.
--timing-file embeds per-task wall time in the verify artifact; it rides through Publish onto the leaderboard. Pass the run's timing.json (a legacy status.json from older runners is also accepted). Never omit it when the run has one; a bad file degrades to a warning, not a failed grade. The shared suite runner grades only timing tasks with status: ok; failed, timed-out, stalled, or rate-limited tasks stay in provenance and are excluded from result coverage.
For a deliberate task-level resume, set GEOBENCH_FROM_TASK to the first
unfinished task after resetting workspaces. After the continuation finishes,
use python scripts/merge_resume_verify.py with the prior verify artifact and
the continuation artifact. It includes low rewards from prior status: ok
tasks, excludes every failed/stopped task, and refuses to write a result unless
the successful pieces cover the 30-task suite exactly once.
For an explicitly requested grader-refresh rerun, set
GEOBENCH_TASKS=trace_doctor,tilemap_decoder (exact comma-separated task ids)
after resetting workspaces. The shared runner grades only that selected set.
Do not publish the resulting two-task artifact as a leaderboard row. Instead,
merge it into the matching complete artifact with
scripts/merge_task_rerun.py, then apply that complete artifact to the
explicit saved row id with scripts/apply_task_rerun_to_leaderboard.py.
Those tools refuse incomplete bases, mismatched model/harness/notes, failed
timing, and accidental updates to a different row.
Writes results/verify-YYYYMMDD-HHMMSS.json.
Import
http://127.0.0.1:8765 → Publish. The verify artifact is waiting in the queue: open its row, check the model and harness, and Save to Runs. The row is Model (Agent Harness).
Do not save a verify that has no model and harness of its own unless the user says what they were. Do not add a second row for the same combo unless the user asked for a rerun.
Comparability
Same timeout, same flags, one attempt per task. Runs from older suite revisions (8- or 20-task) are not comparable to the 30-task suite.
