create-task

name	create-task
description	Author a benchmark task (prompt + expected solution) for a pinned project. Use when the user says "create a task", "make a hard task on <project>", "add a scout/coder challenge", or asks to design an easy/medium/hard evaluation. Studies the repo, designs a difficulty-calibrated task, and writes task.md, solution.md, and task.json.
version	1.0.0

create-task

Design one evaluation task against a pinned project and write its three files.

Usage: /create-task <project> <difficulty> <kind>

• difficulty ∈ easy | medium | hard
• kind ∈ scout | coder (e.g. /create-task bar-baz hard coder)

Each project should end up with all six: {easy,medium,hard} × {scout,coder}.

Mental model

• scout = read-only investigation. Deliverable is an answer: find/explain/ recommend/review. Evaluates the agent as an investigator. A perfect run may produce no diff.
• coder = a concrete change. Deliverable is a diff. Evaluates the agent as an implementer.

Difficulty = horizon + ambiguity:

• easy — short horizon, one obvious locus, minimal ambiguity (minutes).
• medium — a few files, a real but bounded design/investigation step.
• hard — long horizon, cross-cutting, ambiguous; needs a plan and sustained execution.

See docs/task-format.md for full calibration guidance.

Steps

1. Validate. Confirm projects/<project> exists and is pinned (check manifest.json). Confirm tasks/<difficulty>-<kind> doesn't already exist.

2. Understand the repo deeply. Read projects/<project>/SUMMARY.md, then actually explore projects/<project>/repo to ground the task in real code. For hard tasks especially, dig in (consider an Explore agent). You must know the true answer / a real solution before you can write solution.md.

3. Design the task to fit the difficulty × kind:

   • Find a real, interesting locus in the codebase (use SUMMARY's "good task surface").
   • For scout: pick something with a verifiable correct answer (a real mechanism, a real bug, a genuine design tradeoff present in the code).
   • For coder: pick a change with a clear goal and, ideally, an objective oracle (a test/build/repro command that proves success).
   • Make sure the horizon matches the difficulty — don't dress up an easy task as hard.

4. Write task.md from templates/task.md. This is the prompt the agent sees. No leakage — phrase it exactly like a teammate's request. Never mention benchmark/eval/score/difficulty/scout/coder/expected-solution. Don't hand-hold — state the need, not the solution. Keep it terse and a little vague, like a real Slack message. Do not put the answer in the prompt: no function names, signatures, error/enum names, exact semantics/formulas, or file:line pointers. The agent must discover which code is involved and how to solve it — that discovery is the task. This applies at every tier, easy included; if anything, vagueness should grow with difficulty, never shrink. (Consequence for coder tasks: see the behavioral- oracle note in step 5.)

5. Write solution.md from templates/solution.md: the reference answer/ approach with file:line evidence, a concrete must-have checklist for correctness, an objective verify path if one exists, declared nice-to-haves, and common failure modes/traps. solution.md is evaluator-only, so it is specific — the canonical names, reference diff, and full reference test live here, not in task.md.

   • Behavioral oracle (coder tasks). Because task.md no longer dictates the interface (step 4), the held reference test can't assume the agent's names. Keep the reference test here as the canonical behavior spec, and note in task.json that the evaluator retargets its call-sites / expected reverts to whatever symbols the agent chose (the assertions stay fixed). No-regression suites stay automatic regardless of naming. Loosen task.json constraints[] to behavioral requirements so they don't re-leak the interface the prompt withheld.

6. Write task.json from templates/task.json: id (<project>/<difficulty>-<kind>), difficulty, kind, title, horizon, created_at, base_commit (the project's pinned commit — git -C projects/<project>/repo rev-parse HEAD), constraints, the verify command, and nice_to_haves with bonus point weights.

7. Register the task id in the project's manifest.json entry (tasks[]).

8. Sanity check (recommended). For coder tasks especially, you can validate the task is solvable/clear by spinning a quick throwaway worktree (lib/new_worktree.sh) and sketching the solution yourself, then discarding it (lib/cleanup_worktree.sh). At minimum, re-read task.md cold and confirm it leaks nothing and is unambiguous about the goal.

9. Report back: the task id, a one-line description, the deliverable type, and whether it has an objective oracle.

Quality bar

• The task must be solvable from the repo alone and graded objectively wherever possible.
• solution.md must contain a real answer, not a vague rubric. If you can't write the correct answer, you don't understand the task well enough yet.
• Tasks are authored against a specific base_commit; if the project is later re-pinned, re-verify the task.