harbor

name	harbor
description	Run Terminal-Bench or any Harbor dataset with Ante as the agent, using the Harbor adapter in the ante repo's ante-harbor/ directory. Use when the user wants to benchmark Ante on Harbor or reproduce Ante's published eval results.
metadata	{"argument-hint":"<provider> <model> [task ...]"}

Run Harbor with Ante

Run Harbor with Ante as the agent. The adapter (ante_agent.py) lives in the ante-harbor/ directory of the ante repo. Harbor imports it and installs Ante inside each task sandbox from the published install script.

Prerequisites

• A checkout of AntigmaLabs/ante, for ante-harbor/
• uv with Python 3.12
• Docker running: Harbor executes each task in a container
• The provider API key exported in the shell

Run

Resolve the model and provider from the user's request. From the repo's ante-harbor/ directory (so ante_agent:AnteAgent is importable):

uv run --python 3.12 --with harbor harbor run \
  --agent ante_agent:AnteAgent \
  --model "<model_name>" \
  --ak provider=anthropic \
  --ak install_args= \
  --ae 'ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}' \
  --dataset terminal-bench/terminal-bench-2-1 \
  --n-attempts 1

Adjust for the request:

• provider selects which key Ante reads inside the sandbox. For openai or gemini, forward OPENAI_API_KEY or GEMINI_API_KEY with --ae instead.
• install_args picks the Ante build installed in each sandbox: empty for the latest release, or a version to pin.
• Scope: add -i <task-id> (repeatable) to run specific tasks. Smoke-test one task before a full run unless the user asks otherwise.
• Throughput: --n-concurrent <n> caps parallel sandboxes. --n-attempts <n> sets attempts per task; published leaderboard runs use 5.
• Custom endpoint: add --ae 'MODEL_BASE_URL=${MODEL_BASE_URL}' when routing through a proxy.

Read results

Harbor prints a per-task summary and writes a run directory with per-trial output; each trial captures Ante's raw event log from /logs/agent/ante.txt. Task failures do not make harbor exit non-zero, so judge the run by the summary, not the exit code.