Harbor CLI command reference and usage patterns. Covers harbor run, harbor jobs, harbor trials, harbor datasets, harbor adapters, harbor tasks, harbor view, harbor sweeps, harbor traces, harbor cache, and harbor admin commands. Use this skill whenever running Harbor evaluations, managing datasets, viewing results, debugging tasks, exporting traces, or working with any harbor CLI command. Also use when constructing harbor command lines, looking up flag names, or troubleshooting CLI errors.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Harbor CLI command reference and usage patterns. Covers harbor run, harbor jobs, harbor trials, harbor datasets, harbor adapters, harbor tasks, harbor view, harbor sweeps, harbor traces, harbor cache, and harbor admin commands. Use this skill whenever running Harbor evaluations, managing datasets, viewing results, debugging tasks, exporting traces, or working with any harbor CLI command. Also use when constructing harbor command lines, looking up flag names, or troubleshooting CLI errors.
Harbor CLI Reference
Harbor CLI manages the full evaluation lifecycle: creating tasks, running agents, viewing results. Install with uv tool install harbor. Global option: --version / -v.
For complete flag tables with types and defaults for every command, read references/flags.md.
Quick Reference
Command
Description
harbor run
Run evaluations (alias for harbor jobs start)
harbor jobs start
Start an evaluation job
harbor jobs resume
Resume an interrupted job
harbor jobs summarize
AI-powered failure summaries for a job
harbor trials start
Run a single trial (debugging)
harbor trials summarize
AI-powered summary of a single trial
harbor datasets list
List available datasets
harbor datasets download
Download a dataset
harbor adapters init
Scaffold a new benchmark adapter
harbor adapters review
Structural + AI review of an adapter
harbor tasks init
Scaffold a new task
harbor tasks check
AI quality assessment of a task
harbor tasks start-env
Launch task environment interactively
harbor tasks debug
Analyze failing trials for a task
harbor tasks migrate
Convert Terminal-Bench tasks to Harbor
harbor view
Browse job/trial results in web UI
harbor sweeps run
Run successive evaluation sweeps
harbor traces export
Export trace data in ATIF format
harbor cache clean
Clean Docker images and cache directory
harbor admin
Administrative commands (hidden)
harbor run / harbor jobs start
The primary command. harbor run is an alias for harbor jobs start.
Additional flags for environment kwargs (--ek), environment env vars (--ee), verifier kwargs (--vk), orchestrator kwargs (--ok), agent images, retry config, trace export, and per-phase timeout multipliers are in references/flags.md.
Examples
# Run local task with oracle (validates solution + tests)
harbor run -p ./my-task
# Run with a real agent
harbor run -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1
# Dataset evaluation with concurrency
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 -n 8
# Cloud environment
harbor run -d my-dataset -a claude-code -m anthropic/claude-sonnet-4-1 -e daytona
# From config file
harbor run -c eval-config.yaml
# Pass API key to agent
harbor run -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 \
--ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY# Run subset of tasks
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 \
-t "bash-*" -l 10
# Dry run
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 --dry-run
harbor jobs resume
Resume an interrupted or partially completed job.
harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError
Flag
Short
Description
--job-path
-p
Path to job directory with config.json (required)
--filter-error-type
-f
Remove trials matching this error type before resuming. Repeatable. Default: CancelledError
harbor jobs summarize
Generate AI-powered failure summaries for trials in a job.
harbor jobs summarize ./jobs/my-job
harbor jobs summarize ./jobs/my-job -m sonnet --all --overwrite
Flag
Short
Description
job_path
Path to job dir or parent (positional)
--model
-m
Model: haiku, sonnet, opus (default: haiku)
--n-concurrent
-n
Max concurrent queries (default: 5)
--all / --failed
Analyze all or only failed trials (default: --failed)
--overwrite
Overwrite existing summary.md files
harbor trials start
Run a single trial. Useful for debugging and task development.
Key difference from harbor jobs start: the environment flag is --environment-type (not --env), and output goes to --trials-dir (default: ./trials).
Flag
Short
Description
--path
-p
Path to local task directory
--config
-c
Trial config YAML/JSON
--agent
-a
Agent name (default: oracle)
--model
-m
Model for the agent
--environment-type
-e
Environment type (default: docker)
--trials-dir
Output directory (default: ./trials)
--agent-env
--ae
Env var for agent: KEY=VALUE. Repeatable
--agent-kwarg
--ak
Agent kwarg: key=value. Repeatable
--no-cleanup
Keep environment after trial
--no-verify
Skip running tests
Full flag list (task kwargs, git options, etc.) in references/flags.md.
# Test with oracle (validates solution + tests)
harbor trials start -p ./my-task
# Test with real agent
harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1
# Keep container for inspection
harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 --no-cleanup
The oracle agent runs solution/solve.sh inside the environment, then the verifier runs tests/test.sh. If oracle does not get reward 1.0, your tests or solution have a bug.
harbor datasets list
harbor datasets list --registry-url https://custom-registry.example.com
harbor datasets list --registry-path ./local-registry.json
Flags --registry-url and --registry-path are mutually exclusive. Default: Harbor's public registry.
Interactive wizard that prompts for benchmark name, adapter ID, class name, description, source URL, and license. Creates the adapter directory with template files.
Build and push task Docker images to a container registry. Updates task.toml with docker_image for pre-built image workflows. Hidden from harbor --help but fully functional.
harbor datasets list
harbor datasets download terminal-bench@2.0
harbor run -d terminal-bench@2.0 -a claude-code -m anthropic/claude-sonnet-4-1 -n 8
harbor view ./jobs
harbor traces export -p ./jobs/my-job --push --repo my-org/traces
Debug a failing task
harbor tasks start-env -p ./my-task # interactive shell in the container# Inside: bash /tests/test.sh && bash /solution/solve.sh
harbor tasks debug my-task-id -m sonnet # AI analysis of failures
harbor trials start -p ./my-task # re-test with oracle
Resume a failed job
harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError
harbor jobs summarize ./jobs/my-job
Common Gotchas
API keys: Most agents need keys passed via --ae. Claude Code needs ANTHROPIC_API_KEY, OpenHands needs LLM_API_KEY, Goose needs provider-specific keys. If the agent fails immediately, check the key.
Docker must be running. Harbor uses Docker for sandboxed environments. Network exhaustion from many concurrent trials can cause failures -- harbor cache clean helps.
Model name format varies by agent. Claude Code uses anthropic/claude-sonnet-4-1, Cline CLI uses provider:model-id (e.g., openrouter:anthropic/claude-opus-4.5), Goose uses provider/model_name. A ValueError about model format usually means the wrong format for that agent.
--env vs --environment-type:harbor run uses --env / -e. harbor trials start uses --environment-type / -e. Same short flag, different long name.
Default agent is oracle. Forgetting -a gives you the oracle agent, which just runs solve.sh. Fine for task validation, not for real evaluations.
--no-cleanup for debugging. Pass this to harbor trials start to keep the container running after a failed trial so you can inspect it.
Retry defaults skip common errors.--retry-exclude defaults to AgentTimeoutError, VerifierTimeoutError, RewardFileNotFoundError, RewardFileEmptyError, VerifierOutputParseError. These usually indicate task bugs, not transient failures.
