| name | run-deep-swe |
| description | Run reproducible DeepSWE coding-agent benchmark evaluations through OpenRouter and mini-swe-agent. |
| category | agent-evaluation |
| risk | critical |
| source | community |
| source_repo | davidondrej/skills |
| source_type | community |
| date_added | 2026-07-07 |
| author | davidondrej |
| tags | ["benchmark","deepswe","openrouter","evaluation"] |
| tools | ["claude","codex"] |
| license | MIT |
| license_source | https://github.com/davidondrej/skills/blob/main/LICENSE |
| disable-model-invocation | true |
Run DeepSWE via OpenRouter
When to Use
- Use when the user wants to benchmark a model on DeepSWE or mini-swe-agent tasks.
- Use when you need a reproducible coding-agent evaluation plan and output artifacts.
DeepSWE (deepswe.datacurve.ai) is a 113-task Harbor-compatible coding-agent benchmark. It runs via Pier (Harbor fork) driving mini-swe-agent (model-agnostic). Any model reachable through OpenRouter can be scored.
Prerequisites — state-check first
which uv git docker || echo "MISSING: install uv, git, docker"
docker info >/dev/null 2>&1 || echo "MISSING: Docker daemon not running (Pier's default sandbox)"
echo "OPENROUTER_API_KEY set? ${OPENROUTER_API_KEY:+YES}"
Docker must be running — Pier sandboxes each task in Docker by default (--env modal for cloud instead).
OPENROUTER_API_KEY must already be present in the environment. If it is unset,
ask the user to configure their preferred secret-management path; do not read
shell startup files, print secrets, or invent a key.
Setup
git clone https://github.com/datacurve-ai/deep-swe && cd deep-swe
uv tool install datacurve-pier
Run all pier commands from inside deep-swe/, using relative -p tasks/....
OpenRouter wiring (the part the docs don't spell out)
mini-swe-agent has a native OpenRouter model class. Both routes below use OPENROUTER_API_KEY and the OpenRouter slug (vendor/model, e.g. minimax/minimax-m3):
Route A — native OpenRouter class (preferred, hits openrouter.ai/api/v1 directly):
pier run -p deep-swe/tasks --agent mini-swe-agent \
--model minimax/minimax-m3 --model-class openrouter
