mi-primer-benchmarkito

Author a benchmark task (prompt + expected solution) for a pinned project. Use when the user says "create a task", "make a hard task on <project>", "add a scout/coder challenge", or asks to design an easy/medium/hard evaluation. Studies the repo, designs a difficulty-calibrated task, and writes task.md, solution.md, and task.json.

Run a coding agent blind against a benchmark task, review its work, score it on every metric, and write a report. Also runs stateful, resumable multi-task campaigns ("fully eval <model>") that tick a durable checklist case-by-case in serial order. Use when the user says "evaluate <model> on <task>", "fully eval <model>", "run the benchmark", "score an agent", "test <model> against <project>", "resume the eval". Spins up a throwaway git worktree in /tmp/work, dispatches the agent (pi-as-mcp by default), waits serially, then produces reports/<timestamp>-<model>/ with report.md, metrics.json, frozen request+solution, and diff.zip.

Import and pin a codebase into the benchmark. Use when the user says "import repo", "add a project to the bench", "pin <repo>", or gives a git URL/path to benchmark against. Clones the repo at a frozen commit, writes its SUMMARY.md orientation doc and project.json, and registers it in manifest.json.

Show the state of the benchmark — pinned projects, which of the six tasks each has, and how many evaluation reports exist per task. Use when the user says "list the bench", "what projects do we have", "show benchmark coverage", "what's been evaluated".