run-deep-swe

name	run-deep-swe
description	Run reproducible DeepSWE coding-agent benchmark evaluations through OpenRouter and mini-swe-agent.
category	agent-evaluation
risk	critical
source	community
source_repo	davidondrej/skills
source_type	community
date_added	2026-07-07
author	davidondrej
tags	["benchmark","deepswe","openrouter","evaluation"]
tools	["claude","codex"]
license	MIT
license_source	https://github.com/davidondrej/skills/blob/main/LICENSE
disable-model-invocation	true

Run DeepSWE via OpenRouter

When to Use

• Use when the user wants to benchmark a model on DeepSWE or mini-swe-agent tasks.
• Use when you need a reproducible coding-agent evaluation plan and output artifacts.

DeepSWE (deepswe.datacurve.ai) is a 113-task Harbor-compatible coding-agent benchmark. It runs via Pier (Harbor fork) driving mini-swe-agent (model-agnostic). Any model reachable through OpenRouter can be scored.

Prerequisites — state-check first

which uv git docker || echo "MISSING: install uv, git, docker"
docker info >/dev/null 2>&1 || echo "MISSING: Docker daemon not running (Pier's default sandbox)"
echo "OPENROUTER_API_KEY set? ${OPENROUTER_API_KEY:+YES}"

Docker must be running — Pier sandboxes each task in Docker by default (--env modal for cloud instead).

OPENROUTER_API_KEY must already be present in the environment. If it is unset, ask the user to configure their preferred secret-management path; do not read shell startup files, print secrets, or invent a key.

Setup

git clone https://github.com/datacurve-ai/deep-swe && cd deep-swe
uv tool install datacurve-pier            # PyPI (preferred)
# or: uv tool install git+https://github.com/datacurve-ai/pier
# pier bundles mini-swe-agent as the --agent driver

Run all pier commands from inside deep-swe/, using relative -p tasks/....

OpenRouter wiring (the part the docs don't spell out)

mini-swe-agent has a native OpenRouter model class. Both routes below use OPENROUTER_API_KEY and the OpenRouter slug (vendor/model, e.g. minimax/minimax-m3):

Route A — native OpenRouter class (preferred, hits openrouter.ai/api/v1 directly):

pier run -p deep-swe/tasks --agent mini-swe-agent \
  --model minimax/minimax-m3 --model-class openrouter