orbit-bench

# Orbit bench: adapters and task resources Orbit runs every benchmark trial as a **solve** run (the method's Agent works on the task) and a separate **verify** run (the benchmark's grader), and averages only scores it can trust. A benchmark is connected through an **adapter**: a small executable that describes those two runs and reads the score. Orbit is not modified or rebuilt. Read the [adapter guide](https://github.com/zjunlp/Orbit/blob/main/docs/guides/bench-adapter.md); [ResearchClawBench](https://github.com/zjunlp/Orbit/tree/main/examples/bench-adapters/researchclawbench) is a complete example. In a checkout these are `docs/guides/bench-adapter.md` and `examples/bench-adapters/researchclawbench/`. Use `orbit` when it is on PATH. Otherwise replace `orbit` in the commands below with `npx -y @zjunlp/orbit` (requires Node.js 24.18+ in the 24.x line); no source checkout is needed to use the CLI. Run `doctor` before the first benchmark job. ## 1. Study the benchmark first Before writing anything, find out from the benchmark's own code and docs (and write it down for the user in a few lines): - Where the task list comes from, and which tasks cannot run (say why; never drop them silently). - What the Agent receives (instruction text, files, working directory) and what it must produce. - How the original grader runs: command, environment (Python, packages), inputs, where it writes the score, its scale, and its failure modes (missing output, API errors, internal errors). - Whether a reference solution exists (`oracle`), and what an empty submission should score. - Secrets the grader needs (API keys): they are passed by name with `--pass-env`, never stored. Reuse the original code for instructions and grading instead of re-implementing it: scores must stay comparable with results produced by the benchmark's own tools. ## 2. Write the adapter Any language; usually one Python file with the standard library only. Orbit runs `<adapter> tasks|solve|verify|score` in the benchmark directory, one JSON request on stdin, one JSON reply on stdout; on error write stderr and exit non-zero. Each call must finish within 120 s: no experiments or grading inside the adapter. | op | reply | | --- | --- | | `tasks` | `{"tasks": [{"id", "title", "supported", "reason", "tags"}]}` | | `solve` | `{"spec": run, "issues": []}` — request has `trial` (`task`, `method`, `seed`, `options`, `staging`, `timeoutMin`) | | `verify` | `{"spec": run}` — request adds `solve.output` (host path of the solve run's `$ORBIT_OUTPUT`) | | `score` | `{"score": n or null, "maxScore": m, "invalid": bool, "issues": [], "review": {...}}` — request adds `verify.output` | `run` uses `orbit run` fields: `image` or `dockerfile`, `resources`, `mounts` (`{source, target, mode}`), `env`, `network`, `workdir`, `collect` (container paths copied to `output/collected/<path>`), `timeoutMin`, or `backend: "process"` with `command` for a host process. Rules: - For `codex` / `claude-code` methods the solve spec has `"agentTask": {"instruction": "..."}` and no `command`. Orbit builds the Agent CLI command, auth, bundle and network. Never write Agent CLI flags. - Never set `name group params project snapshot auth agentBundle agent parentId priority passEnv`. - `oracle` / `empty` get a plain `command`; without a reference solution, raise an error for `oracle`. - Writable copies and mount points go under `trial.staging` (create directories before mounting). The benchmark checkout stays read-only; set `PYTHONDONTWRITEBYTECODE=1` for Python. - Give `score` a `review` so the trial page shows the right things: `files` (`label`, `phase` solve|verify, `path` relative to that run's output, `kind` markdown|image|text|json|table), `items` (graded criteria: `criterion`, `score`, `maxScore`, `weight`, `reasoning`) and short `details`. Never send HTML or styles; unknown kinds need a `fallback` from that list. - `invalid: true` only when the grader itself failed (API error, internal error, unusable model). A method that produced nothing gets a valid 0. Cancel, timeout, not-started and out-of-memory are judged by Orbit. ## 3. Write the resources file Per-task solve resources live in one JSON file next to the benchmark (for example `<benchmark>/orbit/resources.json`, under git), registered with `--resources`. It overrides what the adapter asks for, so the adapter can keep simple defaults. ```json { "version": 1, "default": { "cpus": 4, "memGB": 16, "gpus": 0, "timeoutMin": 240 }, "tasks": { "Material_002": { "gpus": 1, "gpuMemGB": 12, "memGB": 32, "timeoutMin": 360, "evidence": "trains a GNN on CUDA in 3 of 3 earlier runs; peak RAM 21 GB; longest 3.9 h", "confidence": "medium" } } } ``` - Fields: `cpus`, `memGB` (also the hard memory limit), `gpus`, `gpuMemGB` (minimum GPU memory size), `timeoutMin`; `evidence`, `confidence`, `note` are for people. Unknown fields are errors. - List only tasks that differ from `default`, and only the fields that differ. - `gpus: 1` only when the task really trains or infers on a GPU and would not finish on CPU; importing torch is not evidence. Memory: generous (too little kills the run); CPU: modest (too little only slows it). Timeout: longest known run × 1.5, rounded up to the hour. - Every exception carries `evidence` (what was measured or seen, where) and `confidence`. - Evidence sources, best first: `orbit bench usage` of earlier Orbit jobs; earlier runs of the benchmark's own tools (durations, CUDA use in code and logs, `Killed` / exit 137, input sizes); the task description. ## 4. Register and validate ```bash orbit bench add <name> <checkout> --adapter /abs/path/adapter.py \ --resources /abs/path/resources.json [-c key=value] [--pass-env JUDGE_KEY] orbit bench tasks <name> # registration already listed them once orbit bench run <name> -t <a few tasks> -m empty -n smoke (and -m oracle if there is one) orbit bench show <job> # empty: valid 0; oracle: full score ``` Then compare a sample with the original grader on the same outputs (an LLM judge varies between calls: measure that spread first) and report the differences to the user. Only then run real Agent methods. Editing the adapter or the resources file needs no re-registration. ## 5. Revise resources from measured runs Every run records peak memory, CPU and GPU memory. After jobs with real Agent methods: ```bash orbit bench usage <job>... # peak vs request per task, suggestions, notes orbit bench usage <job>... --suggest # suggested changes as a resources-file fragment ``` Suggestions are never applied automatically. Merge them into the resources file yourself when the user agrees, keep the `evidence`, and decide GPU needs from the task (a run that did not use its GPU may only mean the Agent chose not to train). Out-of-memory solve runs are counted invalid, not as the method's zeros: raise the task's `memGB` and rerun those tasks.