create-task

Guide the user through creating a new Harbor task end-to-end. Don't just dump commands — walk them through each decision, especially around the verifier (which is usually the hardest part). ## Step 1: Scaffold the task ```bash harbor task init "<org>/<task-name>" ``` Useful flags: - `--description "..."` - `--author "Jane Doe <jane@example.com>"` (repeat for multiple authors) - `--no-pytest` — skip the pytest test template (use if planning Reward Kit or custom verifier) - `--no-solution` — skip solution/ directory - `--metadata-template path.toml` — pre-populate task.toml Produces: ``` <task-name>/ ├── instruction.md # Task prompt for the agent ├── task.toml # Config and metadata ├── environment/Dockerfile # Container definition ├── solution/solve.sh # Reference solution (optional) └── tests/test.sh # Verifier script ``` ## Step 2: Write instruction.md This is the prompt the agent receives. Help the user write it clearly: - **State the goal concretely** — what file to create, what behavior to produce - **Specify expected outputs** — paths, formats, content - **Include constraints** — language, tools, approach - **Don't leak the tests** — describe what "done" looks like, not how you'll check it Example (from the ssh-key-pair tutorial): ```markdown # SSH Key Pair Generation Generate an SSH key pair in the files `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`. Don't make them password protected. ``` ## Step 3: Build the environment Edit `environment/Dockerfile` to install dependencies the task needs. The agent works inside this container. ```dockerfile FROM ubuntu:24.04 WORKDIR /app # Install what the task requires — NOT the solution RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/lists/* ``` For multi-container setups, use `environment/docker-compose.yaml` instead (note: most cloud sandbox providers only support Dockerfile). **Test the environment interactively** before writing the solution or tests: ```bash harbor task start-env -p "<task-path>" -e docker -a -i ``` This is usually where task authors realize something is missing from the Dockerfile. ## Step 4: Decide how to verify **This is the most important decision.** Ask the user: *"How do you want to grade this task?"* Then help them pick: ### Option A: Reward Kit (recommended for most cases) Use when the verifier has multiple criteria, needs partial credit, uses an LLM/agent judge, or would benefit from composable reusable checks. See the `rewardkit` skill. Good fit signals: - Multiple things to check (file exists + content correct + command works) - Subjective quality dimensions (readability, correctness of prose) - Want partial credit rather than pass/fail - Want to compose built-ins like `file_contains`, `command_succeeds`, `json_key_equals` `tests/test.sh`: ```bash #!/bin/bash uvx --from harbor-rewardkit==0.1.* rewardkit /tests ``` Note: the package is named `harbor-rewardkit` but the executable is `rewardkit`, hence `--from`. Running `uvx harbor-rewardkit` directly will fail. Then add `tests/checks.py` and/or `tests/judge.toml`. Invoke the `rewardkit` skill to design the criteria. ### Option B: pytest (good for deterministic unit-style checks) Use when the verification is straightforward assertion-style Python. Default template if `--no-pytest` wasn't passed. `tests/test.sh`: ```bash #!/bin/bash apt-get update && apt-get install -y curl curl -LsSf https://astral.sh/uv/0.9.7/install.sh | sh source $HOME/.local/bin/env uvx --with pytest==8.4.1 pytest /tests/test_outputs.py if [ $? -eq 0 ]; then echo 1 > /logs/verifier/reward.txt else echo 0 > /logs/verifier/reward.txt fi ``` Example `tests/test_outputs.py`: ```python from pathlib import Path def test_file_exists(): assert (Path.home() / ".ssh" / "id_rsa").exists() ``` ### Option C: Custom shell For simple single-command checks (e.g. a binary pass/fail from one command): ```bash #!/bin/bash if diff -q /app/output.txt /tests/expected.txt; then echo 1 > /logs/verifier/reward.txt else echo 0 > /logs/verifier/reward.txt fi ``` ### Reward file format (all options) - `/logs/verifier/reward.txt` — single number (usually `0` or `1`) - `/logs/verifier/reward.json` — `{"accuracy": 0.95, "runtime_sec": 1.2}` for multiple metrics **Always use absolute paths in `test.sh`.** ## Step 5: Write the solution Write `solution/solve.sh` — a script that actually solves the task. The Oracle agent runs this to sanity-check that the task is solvable and the tests pass on a correct solution. ```bash #!/bin/bash ssh-keygen -t rsa -f ~/.ssh/id_rsa -N "" ``` Make it executable: `chmod +x solution/solve.sh`. ## Step 6: Configure task.toml Walk through the important fields: ```toml [task] name = "<org>/<task-name>" description = "One-line description" keywords = ["jax", "mnist", "rewardkit"] # always populate — used for search/filtering [metadata] difficulty = "easy" | "medium" | "hard" category = "programming" | "machine-learning" | "gpu" | ... tags = ["..."] [environment] cpus = 1 # CPU cores memory_mb = 2048 # RAM in MB storage_mb = 10240 # Disk in MB allow_internet = true # Network access [agent] timeout_sec = 120.0 # How long the agent has [verifier] timeout_sec = 600.0 # How long tests have ``` **Always populate `keywords`.** Pick 3–8 lowercase tokens covering the domain (language/framework/benchmark family), the verifier style (`rewardkit`, `judge-grading`, `pytest`), and any notable hardware (`gpu`). They're surfaced in `harbor datasets list` and registry search. For Reward Kit judges needing API keys: ```toml [verifier.env] ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}" ``` ## Step 7: Verify with the Oracle agent ```bash harbor run -p "<task-path>" -a oracle ``` Oracle runs `solution/solve.sh` and then the verifier. Reward should be `1.0`. If it's not, debug in this order: 1. Does `solve.sh` actually solve it? (`start-env -a -i` and run it manually) 2. Does the verifier correctly detect success? (check `/logs/verifier/` output) 3. Are paths correct? (absolute vs relative) 4. Are dependencies installed in the Dockerfile? ## Step 8: Test with a real agent (optional) ```bash harbor run -p "<task-path>" -a terminus-2 -m anthropic/claude-sonnet-4-6 ``` If the task is too easy (every model 1.0) or impossible (every model 0.0), consider adjusting difficulty. ## Step 9: Update README.md (always the final step) `harbor task init` leaves `README.md` as a stub. Before wrapping up, populate it so future humans (and agents) can understand the task without reading every file. Include: - **What the agent does** — one paragraph, link to `instruction.md`. - **Environment** — base image, key installed packages, cached data, hardware (GPU/CPU/RAM), agent timeout. - **Verifier** — for Reward Kit tasks, a table of reward dimensions with type (programmatic / LLM judge / agent judge) and what each measures; how they're aggregated. - **Layout** — a tree of the task directory with one-line annotations. - **Running** — the concrete `harbor run` commands (Oracle + real agent), with the right provider flag if the task needs a GPU. Treat this as docs, not marketing — the reader wants to know *what they'd need to change* to modify the task. ## Special features (mention if relevant) - **MCP servers**: Add `[[environment.mcp_servers]]` in task.toml for agent tooling - **Healthcheck**: Add `[environment.healthcheck]` for services that need to be ready - **GPU**: Set `environment.gpus` and optionally `environment.gpu_types` - **Pre-built image**: Set `environment.docker_image` instead of building from Dockerfile - **Non-root user**: Set `agent.user` / `verifier.user` for isolation ## Common pitfalls - Forgetting to write the reward file → task "passes" silently with reward 0 - Using relative paths in `test.sh` → breaks when Harbor runs it from a different cwd - Installing the solution into the Dockerfile → agent already gets the answer - Test script leaks into `instruction.md` → agent sees the rubric and gaming becomes trivial - Forgetting `chmod +x solution/solve.sh` → Oracle agent fails - Leaving `keywords = []` in task.toml → task is invisible to registry search - Leaving `README.md` as a stub → teammates have no way to understand the task at a glance