browser-bench-task

# Browser benchmark task author ## Why this skill exists You are building tasks for `data/real_world_bench.json` — a browser-agent benchmark where each task is graded checkpoint-by-checkpoint (per-rubric). A good task is **real-world, completable, and free of any risk-control block the autonomous run can't get past**: an agent driving ego-browser must be able to actually finish it, and a grader must be able to verify each checkpoint from screenshots, trajectory, or final output. The one rule that matters most, and the reason this skill exists: > **Never write a task you have not run through ego-browser first.** > Web research and model priors *lie* about bot-friendliness — sites that look "low risk" on > paper hard-block real automation (Product Hunt → Cloudflare "Just a moment"; Etsy → DataDome > + forced login at checkout). The only proof is a live ego-browser probe. Then calibrate the > rubrics to what you actually observed, not to what you imagined. This operationalizes the repository's core authoring rule: calibrate every task from a real ego-browser run. ## The loop Do **one task at a time**. Read the referenced file the first time you need its detail. 1. **Load context.** Read `data/real_world_bench.json` (tail for the live schema; scan all `website` + `task_id` values to avoid duplicating a site or play you already have). Skim `references/schema.md` for the schema and `references/authoring.md` for rubric discipline. Recall the bot-friendliness memory if present. 2. **Pick a candidate.** Honor the user's constraints (US mainstream site, required `level`, cross-site wanted?, a mandated site such as reddit?). Avoid sites already in the dataset unless the *play* is genuinely new. To cover breadth fast you may fan out research with a `Workflow` (parallel web-search lanes → shortlist ranked by bot-risk), but research only **prioritizes probe order — it never replaces the probe.** → `references/site-selection.md` 3. **Probe with ego-browser — this is the gate.** Drive the EXACT interactions the task will need: reach the start page, confirm the autonomous run won't be walled off, apply every filter/sort/form step, and confirm the target data is actually extractable. Capture the real trajectory and concrete values. An **automated hard wall** (Cloudflare / DataDome / PerimeterX that won't clear) or an unreachable key interaction → **drop it and pick another**, do not write the task. A **human-verification / captcha** is the one carve-out: pause and hand off to the user to solve it **once** — if it then clears and does **not** return on repeated probing, treat it as a one-time trust gate (not a per-request wall) and the site is usable; if the SAME site re-challenges **after** that solve, **skip it** — a per-entry captcha can't be driven by the agent at run time. Probes share one real browser, so run them **sequentially, never in parallel.** → `references/ego-browser-testing.md` 4. **Design + write `confirmed_task`.** Fit a proven, gradeable pattern (aggregate-and-compute / form-fill-then-stop / calculator / cross-site or cross-app), with concrete filters, fields, computations, and stop-boundaries. Write it in **natural user voice** — first person, the motivation woven in, like the odysseys medium/hard tasks. **No operator meta-instructions** ("read-only", "don't log in", "must use old.reddit.com") — a real user would not say those; put any such rationale in the `note` field. → `references/authoring.md` 5. **Write rubrics** (3–6). Each rubric = one independently verifiable checkpoint; verification is **state-based** (what a grader sees in a screenshot / final output, not an action sequence); operation rubrics must be "applied AND confirmed"; all of it calibrated to your real run. → `references/authoring.md` 6. **Append + validate.** Append the object **surgically** (do not reformat or rewrite existing entries — keep the diff to an append). Then validate: ```bash uv run python -c "import json; json.load(open('data/real_world_bench.json')); print('json ok')" uv run python -c "from ego_bench.datasets import _ADAPTERS; ts=_ADAPTERS['real-world-bench']().load(); t=[x for x in ts if x.task_id==NEW_ID][0]; print(len(ts), 'rubrics:', len(t.metadata['rubrics']))" ``` Missing/empty rubrics fail-loud in Stage B, so confirm the new entry carries them. 7. **Capture learnings.** Update the bot-friendliness memory with any new pass/fail site and its evidence. ## Definition of done - [ ] Ran end-to-end through ego-browser — no automated hard wall, and any human-verification cleared on a **single** handoff and did **not** recur; key interactions reached; data extracted. - [ ] `confirmed_task` reads like a real user request — first person, motivated, **zero operator meta-instructions**. - [ ] 3–6 rubrics, each one verifiable checkpoint, **calibrated to the real run** (not invented). - [ ] JSON valid, adapter loads, new entry has rubrics. - [ ] `level` / `site_mode` / `site_count` / `categories` / `reference_length` filled; `note` carries operator hints + the calibration record. ## Reference files - `references/schema.md` — the Odysseys schema fields, semantics, and a worked JSON example. - `references/ego-browser-testing.md` — **the gate**: risk-control signals, copy-paste probe recipes, special cases (React inputs, cross-origin editors, captcha handoff), and the known bot-friendly / bot-hostile site table. - `references/authoring.md` — the proven task patterns, the natural-phrasing rules (with before/after), rubric discipline with examples, and the anti-patterns to avoid.