geobench-run-pi

name	geobench-run-pi
description	Run GeoBench with Pi. Use when the user asks to run Pi, llama.cpp, OpenRouter, deepseek-v4-flash, Qwen via Pi, or /geobench-run-pi. Shared reset, isolation, grade, and import live in the geobench-run skill.

GeoBench via Pi

Read geobench-run first. --harness is pi.

Invoke

Runs in the geobench-agent container. Pi auth is passed as env-based credentials with -e; on this host the runner can safely hydrate OPENROUTER_API_KEY from the existing .pi/agent/auth.json inside the retained login volume when the host shell variable is absent. The prompt is @ plus the in-container instruction path; do not paste instruction.md onto the command line. Split --provider and --model.

export MSYS_NO_PATHCONV=1
pi_env=()
[[ -n "${OPENROUTER_API_KEY:-}" ]] && pi_env+=(-e "OPENROUTER_API_KEY=$OPENROUTER_API_KEY")
[[ -n "${LLAMA_BASE_URL:-}" ]] && pi_env+=(-e "LLAMA_BASE_URL=$LLAMA_BASE_URL")
docker run --rm --security-opt seccomp=unconfined "${pi_env[@]}" \
  -v geobench-agent-home:/home/agent \
  -v "$(cygpath -w "$root/tasks/$task/workspace"):/work" \
  -v "$(cygpath -w "$root/tasks/$task/instruction.md"):/instruction.md:ro" \
  -w /work geobench-agent \
  pi -p --no-session --approve --mode json \
    --provider "<provider>" --model "<model-id>" --thinking low "@/instruction.md"

For llama.cpp served on this host, LLAMA_BASE_URL must be http://host.docker.internal:8080 — inside the container 127.0.0.1 is the container itself. A llama.cpp box elsewhere is just its IP or hostname.

For a llama.cpp/... combo, the shared suite runs scripts/prepare_pi_llama.py in a short-lived container before timing starts. That preflight checks /models, loads the requested model when needed, waits for it to become ready, and persists the loaded catalog into Pi's retained models-store.json. Pi's headless mode does not fetch extension catalogs on startup; without this step it can report Unknown provider "llama.cpp" even when the interactive /llama picker can see the server.

The preflight auto-loads the requested model by default. Set GEOBENCH_LLAMA_AUTO_LOAD=0 to require the model to be loaded manually, or adjust GEOBENCH_LLAMA_LOAD_TIMEOUT_SEC and GEOBENCH_LLAMA_POLL_SECONDS for slower servers. It never reads or changes credentials.

Auth checks also need --provider. Combined --model provider/id can work, but GGUF ids contain : and Pi also uses :<thinking> as a suffix — always split.

--thinking is low, high, or xhigh on the configured Pi/OpenRouter client. Record it in --notes (thinking high or thinking xhigh). Default low if the user did not name an effort.

Models

Requested queue forms and their required thinking settings:

--provider	--model	--thinking
llama.cpp	unsloth/Qwen3.8-27B-GGUF:Q4_K_M	low
openrouter	deepseek/deepseek-v4-flash-latest	low or high
openrouter	meta/muse-glimmer-30b	high
openrouter	deepseek/deepseek-v4-pro-0813	high
openrouter	qwen/qwen3.8-2.4t-a95b	xhigh
openrouter	google/gemini-3.7-flash	high
openrouter	qwen/qwen3.8-max	high

Grade --model as provider/model-id (e.g. openrouter/deepseek/deepseek-v4-flash-latest).

Pitfalls

• Pasting instruction text as argv → quoting breakage. Use @/instruction.md.
• Combined GGUF id with a colon → Pi may parse :<thinking>. Split provider/model.
• LLAMA_BASE_URL=http://127.0.0.1:8080 inside the container → dead end; use host.docker.internal.
• An unloaded router model is not available to Pi's provider catalog; let the preflight load it and persist the catalog before starting tasks.

Windows Docker and graceful resume

On the Windows Docker Desktop host, keep the shared runner's --security-opt seccomp=unconfined for the bundled container tools, retain geobench-agent-home, and use one disposable --rm task container at a time. Pi itself remains inside the container; only its OpenRouter key is passed with -e OPENROUTER_API_KEY=....

For llama.cpp, the host URL and model catalog preflight are passed separately; the server model is loaded once before the first task and is reused by all task containers.

Treat each model/thinking pair as an independent queue item. Preserve its results/agent-logs/<run-id>/timing.json and task logs, and only consider the item complete once a matching results/verify-*.json exists. If the outer session stops mid-item, clean up only the exact stale task container/process, run python scripts/generate_tasks.py, and rerun that item before advancing; never reuse a partial workspace for another model. Completed verify artifacts are durable checkpoints and must not be rerun unless explicitly requested. A provider quota/session response is recorded as rate_limited and aborts the current suite; resume the same provider/model after its reset window.