ak-deep-swe

name	ak:deep-swe
description	Benchmark a coding model on DeepSWE through Pier and OpenRouter. Use when users ask to run DeepSWE, score a model, or verify coding-agent benchmark results.
user-invocable	true
when_to_use	Invoke for a costed external coding-agent evaluation, not repository-local optimization.
category	dev-tools
keywords	["benchmark","deepswe","pier","openrouter","evaluation"]
license	MIT
argument-hint	<OpenRouter model slug>
metadata	{"author":"agentkit","version":"1.0.0","upstream":"Pinned MIT source archive: run-deep-swe@ce70edaa26247b84c2b9491a0cdb4964f65cf3a5"}

DeepSWE Benchmark

Run an independent coding-agent evaluation on the DeepSWE benchmark through Pier. This skill handles benchmark setup and reporting. It does not evaluate a repository metric or authorize model spend without the user's confirmation.

Guardrails

• First confirm that uv, git, Docker, and the Docker daemon are available.
• Confirm OPENROUTER_API_KEY is set without printing its value. If it is absent, stop and ask the user to configure it.
• Never echo, persist, commit, or place the key in a command, report, prompt, or benchmark artifact.
• Run one task before a subset. Get explicit user confirmation before a full 113-task corpus run because it consumes time and model tokens.

Setup

git clone https://github.com/datacurve-ai/deep-swe
uv tool install datacurve-pier
pier --help
pier run --help

Run the commands below from the directory that contains the cloned deep-swe folder. If you change into deep-swe instead, use tasks or tasks/ as the -p value.

DeepSWE currently documents 113 tasks and Pier as the runner. Confirm current flags and task paths with help before running because this toolchain changes: https://github.com/datacurve-ai/deep-swe https://pypi.org/project/datacurve-pier/ https://deepswe.datacurve.ai/

Run Safely

1. Verify the exact OpenRouter model slug at openrouter.ai/models.

2. Run a single task first. The current native model form is:

   pier run -p deep-swe/tasks/<task-id> --agent mini-swe-agent \
     --model openrouter/<vendor/model>
   

3. For a deterministic subset, use the installed help to confirm the task-count and sample-seed flags, then run a small fixed sample:

   pier run -p deep-swe/tasks --agent mini-swe-agent \
     --model openrouter/<vendor/model> --n-tasks 10 --sample-seed 0
   

4. Use a separate model-class route only when the installed Pier help explicitly documents it; do not guess a version-specific flag.

5. Before a full corpus run, present the exact command, expected cost exposure, and stop condition. Continue only after explicit confirmation.

Inspect and Report

Inspect the generated job directory with the current Pier commands. When the installed version provides them, use pier view, pier analyze, and pier critique to inspect the result. Report the exact command, Pier version, task count, model slug, score or reward, cost when available, and blockers. Do not submit results or contact a leaderboard without the user's request.