<div align="center">
<img src="assets/ego-browser-benchmark-header.svg" width="100%" alt="Ego Browser Benchmark Framework"/>
An open-source, reproducible benchmark harness for comparing how different models perform
browser tasks under a controlled agent and browser-tool setup.
Model comparison is the default protocol: vary the model while holding the provider route,
browser runtime and skill, task set, judge, limits, and concurrency fixed. Other
experimental dimensions are supported, but should be changed independently.
[简体中文](README.zh-CN.md)
[](https://github.com/citrolabs/ego-browser-benchmark-framework/releases/tag/v0.1.0-beta.1)
[](https://www.python.org/)
[](https://github.com/citrolabs/ego-browser-benchmark-framework/actions/workflows/ci.yml)
[](LICENSE)
[](https://github.com/citrolabs/ego-browser-benchmark-framework/stargazers)
[](https://github.com/citrolabs)
</div>
> [!WARNING]
> **Public beta.** Interfaces, configuration, dataset snapshots, and report schemas may
> change before 1.0. Results are intended for controlled comparisons, not as a universal
> leaderboard.
## Evaluation contract
| Layer | Included surface |
| --- | --- |
| Variable under test | Model selected with `--model` or `config.yaml` |
| Agent providers | `pi` and `codex` |
| Browser tool | `ego-browser` by default; additional tools can register through the browser-tool protocol |
| Task input | A bundled dataset, repeated `--task` arguments, or an Odysseys-shaped JSON file |
| Datasets | `real-world-bench`, `odysseys`, `bu-bench`, `stealth-bench`, `online-mind2web` |
| Judge | Pi-based, image-capable agent or single-turn judge |
| Interfaces | Three independent CLI stages and a local WebUI |
For a model comparison, keep the provider route, browser runtime and skill, dataset
snapshot, judge configuration, limits, and concurrency unchanged. Engine-level
observation and teardown are expressed through CDP semantics so every browser-tool
adapter presents the same control surface to the harness.
## Pipeline
```text
dataset or ad-hoc tasks
│
▼
run_eval.py · Stage A · agent execution + browser evidence
│
▼
run_judge.py · Stage B · verdicts and rubric results
│
▼
run_report.py · Stage C · aggregate report + session HTML
```
The stages share a run directory but remain independently callable:
- Stage A writes task records, run metadata, and deduplicated screenshots.
- Stage B reads the agent session and screenshots, then writes judge artifacts and a
summary. A negative verdict does not trigger another Stage A attempt.
- Stage C renders existing artifacts without changing the evaluation result.
The agent's session JSONL is the source for duration, turns, tokens, cost, tool usage, and
final output. `tasks.jsonl` stores task state and artifact references rather than copying
those derived metrics.
## Requirements
- Python 3.13 or newer and [uv](https://docs.astral.sh/uv/)
- Node.js 22.13 or newer, pnpm, and npm for the WebUI and local benchmark sites
- `pi` on `PATH` when Stage A or the Judge uses the Pi provider:
[pi-coding-agent](https://www.npmjs.com/package/@earendil-works/pi-coding-agent)
- `fd` for Pi's `find` tool
- For the default `ego-browser` adapter: macOS, the ego-lite app, and the
`ego-browser` CLI
## Quick start
### 1. Clone and configure an agent
```bash
git clone https://github.com/citrolabs/ego-browser-benchmark-framework.git
cd ego-browser-benchmark-framework
# Pi is the default Stage A provider and the built-in Judge provider.
cp agent-home/.pi/agent/auth.json.example agent-home/.pi/agent/auth.json
# Add this when Codex runs Stage A.
cp agent-home/.codex/auth.json.example agent-home/.codex/auth.json
```
Set the provider and model in `config.yaml`, or override them per run with
`--provider` and `--model`.
### 2. Start the WebUI
Install and finish onboarding in [ego lite](https://lite.ego.app/) first. It prepares the
`ego-browser` CLI, Skill, and latest release SDK. Then start the console:
```bash
make serve
```
Before a run, use **ego-browser version** in New run to inspect or switch the active
release. Use `scripts/bootstrap.sh` only for a local SDK build or to pin a release from
the command line before opening the WebUI.
### 3. Run a smoke task
```bash
uv run python run_eval.py \
--task "Open https://example.com and report the page title."
```
The command prints the generated run ID and writes artifacts under `runs/<run_id>/`.
Plain `--task` inputs are not judged. To judge an ad-hoc batch, use `--tasks-json` with
non-empty rubrics and enable `--auto-judge`.
## CLI workflows
### Run tasks
```bash
# Entire dataset
uv run python run_eval.py --benchmark real-world-bench
# One dataset task
uv run python run_eval.py \
--benchmark real-world-bench \
--task-ids <task-id>
# Multiple ad-hoc tasks
uv run python run_eval.py \
--task "task one" \
--task "task two" \
--concurrency 2
# Odysseys-shaped ad-hoc batch with rubrics
uv run python run_eval.py \
--tasks-json path/to/tasks.json \
--auto-judge
```
### Judge an existing run
```bash
# Judge tasks that do not yet have a result
uv run python run_judge.py --run-id <run_id>
# Retry only judge infrastructure errors
uv run python run_judge.py --run-id <run_id> --rejudge-errors
```
Stage B also supports whole-run, selected-task, single-task, and rubric-level rejudging.
Run `uv run python run_judge.py --help` for the selectors.
### Generate reports
```bash
# Aggregate report; zh is the default locale
uv run python run_report.py --single runs/<run_id>
uv run python run_report.py --single runs/<run_id> --locale en
# Session HTML only
uv run python run_report.py --export-sessions runs/<run_id>
```
Each entry point exposes its full option set through `--help`.
## WebUI
```bash
make serve # WebUI :8365 and benchmark sites :4374
make dev # WebUI :8365, Vite :5373, benchmark sites :4374
```
The WebUI can:
- configure dataset and ad-hoc runs;
- queue, monitor, stop, resume, and selectively rerun tasks;
- inspect Judge progress and rejudge a run, task, or rubric subset;
- export aggregate reports, agent sessions, judge sessions, and screenshot galleries.
CLI- and WebUI-started runs use the same `runs/` layout and remain available in both
interfaces. Local-site tasks require the benchmark-site service started by `make serve`
or `make dev`.
## Configuration
Configuration precedence is:
```text
CLI arguments > config.yaml > code defaults
```
`config.yaml` groups the run, provider, sampling, limit, judge, screenshot, WebUI, and
browser-tool settings. The WebUI creates per-run overrides and records the resolved
configuration in `run_metadata.json`; it does not rewrite `config.yaml`.
## Artifacts and result validity
A completed run commonly contains:
```text
runs/<run_id>/
├── run_metadata.json
├── tasks.jsonl
├── summary.json
├── screenshots/<task_id>/frames/
├── judges/
├── sessions_html/
└── judge_sessions_html/
```
Use the raw session JSONL, task records, judge JSON, screenshots, and accessibility trees
when auditing a result. The HTML files are presentation artifacts.
Most bundled tasks target live websites. Site changes and anti-bot challenges can prevent
a task from measuring the model at all. Sign the browser into required sites before a
comparison run, and separate intercepted or stale-site tasks from genuine agent failures.
## Add a browser tool
The built-in implementation is `ego-browser`, but the harness does not require callers to
branch on browser-tool names.
1. Implement the
[`BrowserTool`](ego_bench/browser_tools/base.py) protocol:
`AgentSurface` describes how the agent invokes the CLI, and `EngineControl` provides
screenshot and teardown operations.
2. Register the class with
[`register_browser_tool`](ego_bench/browser_tools/__init__.py) before a run starts.
3. Add `browser_tools.<name>` to `config.yaml` with `bin_env`, `bin_default`, and
`skill_name`.
4. If the CLI has task-space management subcommands, register them with
`register_task_management_subcommands`.
The WebUI reads the registered options from the backend, so an adapter does not require a
frontend-specific tool list.
## Datasets and task resources
Dataset adapters live in [`ego_bench/datasets/`](ego_bench/datasets/), and their snapshots
live in [`data/`](data/). See [data/README.md](data/README.md) for schemas, task counts,
resource fields, and redistribution notes.
Tasks can reference named attachments under `data/attachments/` or deterministic local
sites under `benchmark_sites/app/`. See
[benchmark_sites/README.md](benchmark_sites/README.md) before adding a local site.
## Project map
| Path | Purpose |
| --- | --- |
| `ego_bench/` | Core package: providers, datasets, browser tools, judging, reports, and WebUI backend |
| `run_eval.py` | Stage A entry point |
| `run_judge.py` | Stage B entry point |
| `run_report.py` | Stage C entry point |
| `config.yaml` | Repository defaults and local overrides |
| `agent-home/` | Pi and Codex runtime templates |
| `frontend/` | React WebUI |
| `benchmark_sites/` | Deterministic local benchmark sites |
| `tests/` | Python tests and repository checks |
## Development
```bash
uv sync
uv run ruff check .
uv run pytest -q
make build
```
## License
The framework code is available under the [MIT License](LICENSE). Some bundled task texts
come from external benchmark datasets and retain their upstream terms; review
[data/README.md](data/README.md) before redistributing them.
Leer el README original