| name | geobench-run-claude |
| description | Run GeoBench with Claude Code. Use when the user asks to run Claude, claude-code, opus, sonnet, or /geobench-run-claude. Shared reset, isolation, grade, and import live in the geobench-run skill.
|
GeoBench via Claude Code
Read geobench-run first. --harness is claude-code.
Invoke
Runs in the geobench-agent container; -w /work is the working directory. Put the prompt after -- so instruction text such as -File is not parsed as a Claude flag.
export MSYS_NO_PATHCONV=1
prompt="$(<"$root/tasks/$task/instruction.md")"
docker run --rm --security-opt seccomp=unconfined \
-v geobench-agent-home:/home/agent \
-v "$(cygpath -w "$root/tasks/$task/workspace"):/work" \
-v "$(cygpath -w "$root/tasks/$task/instruction.md"):/instruction.md:ro" \
-w /work geobench-agent \
claude -p --model "<claude-model-id>" --permission-mode acceptEdits \
--allowedTools "Bash,Read,Edit,Write,Glob,Grep" \
--no-session-persistence --verbose --output-format stream-json -- "$prompt"
Host-native fallback (Docker unavailable — say so in the run notes): same flags, cd into the task workspace first.
On Windows Docker Desktop, retain the shared runner's --security-opt seccomp=unconfined, use exactly one disposable --rm task container at a time,
and never remove geobench-agent-home. If a rate limit, token budget, or outer
session stop interrupts a combo, preserve its run directory, clean only its
named stale container, regenerate tasks, and resume the same bare model id
later. Do not advance from a partial workspace; require the matching finished
verify artifact.
Models
Installed/accepted ids seen on this host include claude-opus-4-8,
claude-sonnet-5, and claude-opus-5. Use the bare ids claude-opus-5 and
claude-sonnet-5 for the requested high queue entries; record effort high in
the verify notes when that queue knob is selected.
A settings default such as us.anthropic.claude-opus-5 can be invalid. Use a bare id the installed CLI accepts.
Pitfalls
- Omitting
-- before "$prompt" → Claude treats -File (and similar) as options.
- With
--output-format stream-json in print mode, Claude CLI 2.1.233 also requires --verbose; otherwise every task exits before making an agent request.
- A Claude JSONL
rate_limit_event with out_of_credits / HTTP 429 is a provider checkpoint, not a task failure. The shared runner stops the suite, preserves the log, and resumes the same model after the reset window.
- Container login missing → first run
claude interactively in the login container (docker/README.md).
Bedrock
When Claude Code is authenticated through AWS Bedrock, configure the AWS
profile in the retained geobench-agent-home volume, verify it with
aws sts get-caller-identity --profile <profile>, then export these selectors
before running the shared suite:
export CLAUDE_CODE_USE_BEDROCK=1
export AWS_PROFILE=Claude_on_Bedrock-986192223246
export AWS_REGION=us-east-1
The shared runner forwards only these mode/profile/region selectors into each
task container. It does not forward raw AWS credentials.
