create-task

name	create-task
description	Create a new Harbor task for evaluating agents. Use when the user wants to scaffold, build, or design a new task, benchmark problem, or eval. Guides through instruction writing, environment setup, verifier design (pytest vs Reward Kit vs custom), and solution scripting.
argument-hint	["org/task-name"]

Guide the user through creating a new Harbor task end-to-end. Don't just dump commands — walk them through each decision, especially around the verifier (which is usually the hardest part).

Step 1: Scaffold the task

harbor task init "<org>/<task-name>"

Useful flags:

• --description "..."
• --author "Jane Doe <[e-mail]>" (repeat for multiple authors)
• --no-pytest — skip the pytest test template (use if planning Reward Kit or custom verifier)
• --no-solution — skip solution/ directory
• --metadata-template path.toml — pre-populate task.toml

Produces:

<task-name>/
├── instruction.md         # Task prompt for the agent
├── task.toml              # Config and metadata
├── environment/Dockerfile # Container definition
├── solution/solve.sh      # Reference solution (optional)
└── tests/test.sh          # Verifier script

Step 2: Write instruction.md

This is the prompt the agent receives. Help the user write it clearly:

• State the goal concretely — what file to create, what behavior to produce
• Specify expected outputs — paths, formats, content
• Include constraints — language, tools, approach
• Don't leak the tests — describe what "done" looks like, not how you'll check it

Example (from the ssh-key-pair tutorial):

# SSH Key Pair Generation

Generate an SSH key pair in the files `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`.

Don't make them password protected.

Step 3: Build the environment

Edit environment/Dockerfile to install dependencies the task needs. The agent works inside this container.

FROM ubuntu:24.04
WORKDIR /app

# Install what the task requires — NOT the solution
RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/lists/*

For multi-container setups, use environment/docker-compose.yaml instead (note: most cloud sandbox providers only support Dockerfile).

Test the environment interactively before writing the solution or tests:

harbor task start-env -p "<task-path>" -e docker -a -i

This is usually where task authors realize something is missing from the Dockerfile.

Step 4: Decide how to verify

This is the most important decision. Ask the user: "How do you want to grade this task?" Then help them pick:

Option A: Reward Kit (recommended for most cases)

Use when the verifier has multiple criteria, needs partial credit, uses an LLM/agent judge, or would benefit from composable reusable checks. See the rewardkit skill.

Good fit signals:

• Multiple things to check (file exists + content correct + command works)
• Subjective quality dimensions (readability, correctness of prose)
• Want partial credit rather than pass/fail
• Want to compose built-ins like file_contains, command_succeeds, json_key_equals

tests/test.sh:

#!/bin/bash
uvx --from harbor-rewardkit==0.1.* rewardkit /tests

Note: the package is named harbor-rewardkit but the executable is rewardkit, hence --from. Running uvx harbor-rewardkit directly will fail.

Then add tests/checks.py and/or tests/judge.toml. Invoke the rewardkit skill to design the criteria.

Option B: pytest (good for deterministic unit-style checks)

Use when the verification is straightforward assertion-style Python. Default template if --no-pytest wasn't passed.

tests/test.sh:

#!/bin/bash
apt-get update && apt-get install -y curl
curl -LsSf https://astral.sh/uv/0.9.7/install.sh | sh
source $HOME/.local/bin/env

uvx --with pytest==8.4.1 pytest /tests/test_outputs.py

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

Example tests/test_outputs.py:

from pathlib import Path

def test_file_exists():
    assert (Path.home() / ".ssh" / "id_rsa").exists()

Option C: Custom shell

For simple single-command checks (e.g. a binary pass/fail from one command):

#!/bin/bash
if diff -q /app/output.txt /tests/expected.txt; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

Reward file format (all options)

• /logs/verifier/reward.txt — single number (usually 0 or 1)
• /logs/verifier/reward.json — {"accuracy": 0.95, "runtime_sec": 1.2} for multiple metrics

Always use absolute paths in test.sh.

Step 5: Write the solution

Write solution/solve.sh — a script that actually solves the task. The Oracle agent runs this to sanity-check that the task is solvable and the tests pass on a correct solution.

#!/bin/bash
ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ""

Make it executable: chmod +x solution/solve.sh.

Step 6: Configure task.toml

Walk through the important fields:

[task]
name = "<org>/<task-name>"
description = "One-line description"
keywords = ["jax", "mnist", "rewardkit"]  # always populate — used for search/filtering

[metadata]
difficulty = "easy" | "medium" | "hard"
category = "programming" | "machine-learning" | "gpu" | ...
tags = ["..."]

[environment]
cpus = 1               # CPU cores
memory_mb = 2048       # RAM in MB
storage_mb = 10240     # Disk in MB
allow_internet = true  # Network access

[agent]
timeout_sec = 120.0    # How long the agent has

[verifier]
timeout_sec = 600.0    # How long tests have

Always populate keywords. Pick 3–8 lowercase tokens covering the domain (language/framework/benchmark family), the verifier style (rewardkit, judge-grading, pytest), and any notable hardware (gpu). They're surfaced in harbor datasets list and registry search.

For Reward Kit judges needing API keys:

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

Step 7: Verify with the Oracle agent

harbor run -p "<task-path>" -a oracle

Oracle runs solution/solve.sh and then the verifier. Reward should be 1.0. If it's not, debug in this order:

1. Does solve.sh actually solve it? (start-env -a -i and run it manually)
2. Does the verifier correctly detect success? (check /logs/verifier/ output)
3. Are paths correct? (absolute vs relative)
4. Are dependencies installed in the Dockerfile?

Step 8: Test with a real agent (optional)

harbor run -p "<task-path>" -a terminus-2 -m anthropic/claude-sonnet-4-6

If the task is too easy (every model 1.0) or impossible (every model 0.0), consider adjusting difficulty.

Step 9: Update README.md (always the final step)

harbor task init leaves README.md as a stub. Before wrapping up, populate it so future humans (and agents) can understand the task without reading every file. Include:

• What the agent does — one paragraph, link to instruction.md.
• Environment — base image, key installed packages, cached data, hardware (GPU/CPU/RAM), agent timeout.
• Verifier — for Reward Kit tasks, a table of reward dimensions with type (programmatic / LLM judge / agent judge) and what each measures; how they're aggregated.
• Layout — a tree of the task directory with one-line annotations.
• Running — the concrete harbor run commands (Oracle + real agent), with the right provider flag if the task needs a GPU.

Treat this as docs, not marketing — the reader wants to know what they'd need to change to modify the task.

Special features (mention if relevant)

• MCP servers: Add [[environment.mcp_servers]] in task.toml for agent tooling
• Healthcheck: Add [environment.healthcheck] for services that need to be ready
• GPU: Set environment.gpus and optionally environment.gpu_types
• Pre-built image: Set environment.docker_image instead of building from Dockerfile
• Non-root user: Set agent.user / verifier.user for isolation

Common pitfalls

• Forgetting to write the reward file → task "passes" silently with reward 0
• Using relative paths in test.sh → breaks when Harbor runs it from a different cwd
• Installing the solution into the Dockerfile → agent already gets the answer
• Test script leaks into instruction.md → agent sees the rubric and gaming becomes trivial
• Forgetting chmod +x solution/solve.sh → Oracle agent fails
• Leaving keywords = [] in task.toml → task is invisible to registry search
• Leaving README.md as a stub → teammates have no way to understand the task at a glance