geobench-run

name	geobench-run
description	Run the GeoBench 30-task coding-agent suite for one model×harness combination. Use when the user asks to run GeoBench, run the benchmark, evaluate a model, grade a suite, or invokes /geobench-run. Dispatch to geobench-run-codex, geobench-run-claude, geobench-run-grok, geobench-run-cursor, geobench-run-pi, geobench-run-opencode, or geobench-run-aiterator for harness-specific flags.

GeoBench suite run

One leaderboard row is one model × harness combination. Reset first. Never reuse a workspace across combos.

Human reference: MANUAL_RUN.md. Command-tab scripts at http://127.0.0.1:8765 stop on the first non-zero exit. A full suite must continue after a failed or timed-out task.

Use Git Bash. Scripts start as bash geobench-….sh from Git Bash, not cmd.exe. Agents run inside the geobench-agent Docker container; Docker Desktop must be running (docker info succeeds) and the image built (docker build -t geobench-agent docker). Host paths going into docker -v want Windows form (cygpath -w); scripts must export MSYS_NO_PATHCONV=1 so Git Bash does not rewrite in-container paths like /work.

Dispatch

Read the matching skill, then use this file for reset, isolation, grade, and import.

User / CLI	Skill	--harness
Codex / codex	geobench-run-codex	codex
Claude Code / claude	geobench-run-claude	claude-code
Grok CLI / grok	geobench-run-grok	grok
Cursor Agent / cursor-agent	geobench-run-cursor	cursor-agent
Pi / pi	geobench-run-pi	pi
OpenCode / opencode	geobench-run-opencode	opencode
aiterator / aiter exec	geobench-run-aiterator	aiterator

If the binary is missing or the model cannot authenticate, skip that combo and say so. Do not invent a substitute model.

Reset

From the repo root, before a new combo:

python scripts/generate_tasks.py

Do not run this after the agent has started. Do not regenerate between tasks of the same combo.

Isolation

The evaluated agent may see only tasks/<id>/instruction.md and tasks/<id>/workspace/.

Never give it hidden/, gold/, verify.py, grader output, or another combo's workspace. Do not copy agent output into hidden/ or gold/.

Default: run the agent inside the geobench-agent container (build, login, and env details in docker/README.md; per-harness container commands in MANUAL_RUN.md section 0). Mount only the task workspace and instruction:

docker run --rm --security-opt seccomp=unconfined \
  -v geobench-agent-home:/home/agent \
  -v "$(cygpath -w "$root/tasks/<id>/workspace"):/work" \
  -v "$(cygpath -w "$root/tasks/<id>/instruction.md"):/instruction.md:ro" \
  -w /work geobench-agent <agent command>

Inside the container the workspace is /work, the instruction is /instruction.md, and the harness flags are the same as the host forms in the per-harness skills. Pi needs -e OPENROUTER_API_KEY (and -e LLAMA_BASE_URL=http://host.docker.internal:8080 for llama.cpp on this host). For Pi llama.cpp combos, the shared runner runs its model preflight before timing starts so headless Pi has a persisted loaded-model catalog. The shared runner can hydrate the OpenRouter key from the existing Pi auth record in geobench-agent-home when the shell variable is unset; it never prints the secret. The workspace is edited in place through the mount, so grading is unchanged. Host-native invocation is the fallback when Docker is unavailable — say so in the run notes. aiterator has no container install; run it host-native.

Continue remaining tasks if one attempt fails or times out (default 3600s).

Windows Docker Desktop operational notes

On this Windows host, keep MSYS_NO_PATHCONV=1 for Docker invocations and use Windows-form paths for absolute arguments passed to the Windows Python executable. The suite template converts timing paths with cygpath -w and strips CRLF from task names discovered through Python.

Codex CLI 0.147.0 bundles a bwrap sandbox. Docker's default seccomp profile blocks its user-namespace setup here, so the shared runner includes --security-opt seccomp=unconfined on every Docker task container. Keep the container unprivileged, retain Codex --approve-for-me, and do not add a Codex --sandbox flag.

Use the persistent geobench-agent-home volume for CLI authentication and run one disposable --rm task container at a time. Never remove that volume. If an outer runner is interrupted, verify docker ps and terminate stale runner processes before removing only their named task containers. The runner's bounded retry loop handles transient provider/network errors; a persistent rate limit or token/session stop leaves a partial timing/log checkpoint and must be resumed as the same model×harness item after a fresh reset.

Host-side Go grading on this machine can fail in the default cache with Access is denied. Set GOCACHE to a writable repository-local .gocache before the timed verify step. Logs: results/agent-logs/<sanitized-model>--<harness>[--effort]--<yyyyMMdd-HHmmss>/. Provenance only; the grade is the edited workspace/. Treat a combo as complete only when its timing is finished and its matching results/verify-*.json identifies the same model, harness, and knob. Do not advance a queue item from a partial workspace.

Wall time

The leaderboard records per-task wall time, and the grader reads it from the run's timing file. Start the suite runner from scripts/run_suite.template.sh: copy it to results/agent-logs/run-<combo>.sh, fill the CONFIGURE block (harness, model, notes, effort/variant) — the per-harness docker commands are already in the template — and everything else (the timing.json boundary writes via harness/timing.py, timeout and stall handling, container cleanup, and the timed grade) is already correct. Run it with bash run-<combo>.sh from Git Bash.

If writing a loop from scratch instead, call python harness/timing.py at every boundary: start before the first task, task-start / task-finish --status ok|failed|timeout|stall|rate_limited --exit-code N --attempts N around each task, and finish at the end. A provider rate-limit checkpoint finishes the current task as rate_limited, finishes the suite as aborted --reason rate_limited, and must not be graded as a complete result. A task that never finished stays open; the grader skips it instead of guessing. Legacy status.json files from older PowerShell runners are still accepted by --timing-file.

Tasks

Every tasks/<id>/ directory that contains task.yaml, sorted by its explicit order field. The directory name is the stable task id (device_manifest, not 05_device_manifest).

Grade

python -m harness.run verify --all --harness <harness> --model "<model-id>" --notes "<knob>" --timing-file "<log-dir>/timing.json"

--notes records an extra knob (effort high, thinking high, variant high). Omit when there is none.

--timing-file embeds per-task wall time in the verify artifact; it rides through Publish onto the leaderboard. Pass the run's timing.json (a legacy status.json from older runners is also accepted). Never omit it when the run has one; a bad file degrades to a warning, not a failed grade. The shared suite runner grades only timing tasks with status: ok; failed, timed-out, stalled, or rate-limited tasks stay in provenance and are excluded from result coverage.

For a deliberate task-level resume, set GEOBENCH_FROM_TASK to the first unfinished task after resetting workspaces. After the continuation finishes, use python scripts/merge_resume_verify.py with the prior verify artifact and the continuation artifact. It includes low rewards from prior status: ok tasks, excludes every failed/stopped task, and refuses to write a result unless the successful pieces cover the 30-task suite exactly once.

For an explicitly requested grader-refresh rerun, set GEOBENCH_TASKS=trace_doctor,tilemap_decoder (exact comma-separated task ids) after resetting workspaces. The shared runner grades only that selected set. Do not publish the resulting two-task artifact as a leaderboard row. Instead, merge it into the matching complete artifact with scripts/merge_task_rerun.py, then apply that complete artifact to the explicit saved row id with scripts/apply_task_rerun_to_leaderboard.py. Those tools refuse incomplete bases, mismatched model/harness/notes, failed timing, and accidental updates to a different row.

Writes results/verify-YYYYMMDD-HHMMSS.json.

Import

http://127.0.0.1:8765 → Publish. The verify artifact is waiting in the queue: open its row, check the model and harness, and Save to Runs. The row is Model (Agent Harness).

Do not save a verify that has no model and harness of its own unless the user says what they were. Do not add a second row for the same combo unless the user asked for a rerun.

Comparability

Same timeout, same flags, one attempt per task. Runs from older suite revisions (8- or 20-task) are not comparable to the 30-task suite.