benchflow

# BenchFlow — Agent Benchmarking BenchFlow runs AI coding agents against tasks in sandboxed environments and scores their output. It combines Harbor (environments, verifier) with ACP (multi-turn agent communication). Arguments passed: `$ARGUMENTS` --- ## Dispatch on arguments ### No args or `status` — show current state 1. Check if benchflow is installed: `uv tool list | grep benchflow` 2. Check if `.env` exists with API keys 3. Check available agents: `benchflow agents` 4. Show recent job results if any exist in `jobs/` 5. Point to next action based on state ### `eval run --tasks-dir <task-path>` — run a task ```bash source .env bench eval run --tasks-dir <task-path> --agent claude-agent-acp --sandbox daytona --model claude-haiku-4-5-20251001 ``` Or via SDK: ```python import asyncio from benchflow import SDK async def main(): sdk = SDK() result = await sdk.run( task_path="<task-path>", agent="claude-agent-acp", model="claude-haiku-4-5-20251001", environment="daytona", ) print(f"Reward: {result.rewards}, Tools: {result.n_tool_calls}") asyncio.run(main()) ``` API keys are auto-inherited from `os.environ`. No need to pass `agent_env`. ### `job <tasks-dir>` — run a benchmark suite ```bash benchflow job --tasks-dir <tasks-dir> --agent claude-agent-acp --sandbox daytona --concurrency 64 ``` Or via YAML config: ```bash benchflow job --config examples/configs/tb2-haiku.yaml ``` YAML format (benchflow-native): ```yaml source: repo: harbor-framework/terminal-bench-2 jobs_dir: jobs/tb2-haiku agent: claude-agent-acp model: claude-haiku-4-5-20251001 environment: daytona concurrency: 64 max_retries: 1 ``` Harbor-compatible YAML also works: ```yaml jobs_dir: jobs n_attempts: 2 orchestrator: n_concurrent_trials: 8 environment: type: daytona agents: - name: claude-agent-acp model_name: anthropic/claude-haiku-4-5-20251001 datasets: - path: harbor-framework/terminal-bench-2 ``` Multi-turn (adds a recheck prompt): ```yaml source: repo: harbor-framework/terminal-bench-2 jobs_dir: jobs/tb2-multiturn agent: claude-agent-acp model: claude-haiku-4-5-20251001 environment: daytona concurrency: 64 prompts: - null # uses instruction.md - "Review your solution. Check for errors, test it, and fix any issues." ``` ### `metrics <jobs-dir>` — analyze results ```bash benchflow metrics jobs/tb2-haiku/ benchflow metrics jobs/tb2-haiku/ --json ``` SDK: ```python from benchflow import collect_metrics metrics = collect_metrics("jobs/tb2-haiku", benchmark="TB2", agent="claude-agent-acp") print(metrics.summary()) ``` ### `view <trial-dir>` — view a trajectory ```bash benchflow view jobs/tb2-haiku/<trial-name>/ ``` Opens HTML viewer at `http://localhost:8888`. ### `create-task` — create a new benchmark task See `skills/benchflow/references/create-task.md` for the full guide. Quick structure: ``` my-task/ ├── task.toml # timeouts, resources, metadata ├── instruction.md # what the agent should do ├── environment/ │ └── Dockerfile # sandbox setup ├── tests/ │ └── test.sh # verifier → writes to /logs/verifier/reward.txt └── solution/ # optional reference solution ``` ### `agents` — list available agents ```bash benchflow agents ``` | Agent | Status | Skills | |-------|--------|--------| | `claude-agent-acp` | Working | `~/.claude/skills/` | | `pi-acp` | Working | `~/.claude/skills/` | | `openclaw` | Working (via shim) | copies to `<workspace>/skills/` | | `codex-acp` | Registered | needs OPENAI_API_KEY | | `gemini` | Registered | needs GOOGLE_API_KEY | ### `compare` — multi-agent comparison ```python import asyncio from benchflow import Job, JobConfig async def main(): for agent in ["claude-agent-acp", "pi-acp", "openclaw"]: job = Job( tasks_dir="path/to/tasks", jobs_dir=f"jobs/compare-{agent}", config=JobConfig(agent=agent, environment="daytona", concurrency=64), ) result = await job.run() print(f"{agent}: {result.passed}/{result.total} ({result.score:.1%})") asyncio.run(main()) ``` --- ## Setup ```bash uv tool install --python 3.12 benchflow # or: uv tool install --python 3.12 -e . (from source) source .env # ANTHROPIC_API_KEY, DAYTONA_API_KEY ``` ## Environments | Environment | Concurrency | Setup | |-------------|-------------|-------| | `daytona` | 64+ | Set `DAYTONA_API_KEY` in `.env` | | `docker` | ~4 | Docker must be running locally | Use `daytona` for benchmarks. Docker is limited by network exhaustion. ## Skills in tasks SkillsBench tasks bake skills into Docker images: ```dockerfile COPY skills /root/.claude/skills ``` - `claude-agent-acp` / `pi-acp`: auto-discover `~/.claude/skills/` - `openclaw`: shim copies from `.claude/skills/` → `<workspace>/skills/` - Skills must load from the environment, never injected into prompts ## Output structure ``` jobs/{job_name}/{trial_name}/ ├── result.json # rewards, agent, timing ├── prompts.json # prompts sent ├── trajectory/ │ └── acp_trajectory.jsonl # tool calls + agent thoughts └── verifier/ ├── reward.txt # reward value └── ctrf.json # test results ``` ## Tips - Use `claude-haiku-4-5-20251001` for testing. Use Sonnet for real benchmarks. - Jobs resume — re-running the same `jobs_dir` skips completed tasks. - `None` in prompts list gets replaced with `instruction.md` content. - Partial rewards work (verifier can write `0.5` to reward.txt).