菜单
Import and pin a codebase into the benchmark. Use when the user says "import repo", "add a project to the bench", "pin <repo>", or gives a git URL/path to benchmark against. Clones the repo at a frozen commit, writes its SUMMARY.md orientation doc and project.json, and registers it in manifest.json.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Author a benchmark task (prompt + expected solution) for a pinned project. Use when the user says "create a task", "make a hard task on <project>", "add a scout/coder challenge", or asks to design an easy/medium/hard evaluation. Studies the repo, designs a difficulty-calibrated task, and writes task.md, solution.md, and task.json.
Run a coding agent blind against a benchmark task, review its work, score it on every metric, and write a report. Also runs stateful, resumable multi-task campaigns ("fully eval <model>") that tick a durable checklist case-by-case in serial order. Use when the user says "evaluate <model> on <task>", "fully eval <model>", "run the benchmark", "score an agent", "test <model> against <project>", "resume the eval". Spins up a throwaway git worktree in /tmp/work, dispatches the agent (pi-as-mcp by default), waits serially, then produces reports/<timestamp>-<model>/ with report.md, metrics.json, frozen request+solution, and diff.zip.
Show the state of the benchmark — pinned projects, which of the six tasks each has, and how many evaluation reports exist per task. Use when the user says "list the bench", "what projects do we have", "show benchmark coverage", "what's been evaluated".
| name | import-repo |
| description | Import and pin a codebase into the benchmark. Use when the user says "import repo", "add a project to the bench", "pin <repo>", or gives a git URL/path to benchmark against. Clones the repo at a frozen commit, writes its SUMMARY.md orientation doc and project.json, and registers it in manifest.json. |
| version | 1.0.0 |
Pin a new project into the benchmark so tasks can be authored against a frozen, non-drifting snapshot.
Usage: /import-repo <git-url-or-path> [ref]
(e.g. /import-repo https://github.com/foo/bar main)
projects/<name>/repo (gitignored — not committed).projects/<name>/SUMMARY.md — the "summary pole" that orients task
authors and evaluators (the agent under test never sees this).projects/<name>/project.json — machine-readable pin + build commands.manifest.json.Pick a slug. Derive a short kebab-case <name> from the repo (e.g.
github.com/foo/Bar-Baz → bar-baz). Confirm it isn't already in
manifest.json.
Clone & pin using the helper (resolves and prints the full SHA):
lib/clone_pin.sh <source> <name> [ref]
If no ref is given it pins the default branch's current HEAD. The repo ends
on a detached commit so nothing tracks a moving branch.
Understand the repo. Explore projects/<name>/repo enough to write a real
summary: read the README, top-level layout, build/test config, main modules.
For anything non-trivial, prefer launching an Explore agent to map it.
Identify the build/install/test/run commands.
Write SUMMARY.md from templates/SUMMARY.md. Fill every section
honestly: what it is, architecture/layout, how to build & test, key concepts,
good task surface (where easy→hard Scout & Coder tasks could live), and
gotchas. This is the single most valuable artifact for later task authoring —
make it genuinely useful.
Write project.json from templates/project.json: name, source, ref,
the full commit SHA, imported_at (use date -u +%Y-%m-%dT%H:%M:%SZ),
languages, and the build/test/run commands you found.
Register in manifest.json. Append an entry to projects[] with name,
source, ref, commit, imported_at, language, one-line summary, and an empty
tasks: []. Keep the JSON valid (the schema is docs/manifest.schema.json).
Report back: the slug, the pinned commit, languages, build/test commands,
and a one-line pitch of where good tasks could live. Suggest running
/create-task <name> easy scout next.
projects/*/repo/ — it's gitignored by design and rebuildable
from source + commit.projects/<name>/repo first, then
re-run, and warn that existing tasks/base_commits may need re-checking.