browser-harness

<!-- VIBE-SELLER CUSTOMIZATIONS: adapted from the upstream 0.13 skill (browser_use/skills/browser-use/SKILL.md in the wheel). If re-syncing from upstream, re-apply: (1) the Store/No-store task banners, (2) the wrapper env-injection contract (BU_NAME/BU_CDP_WS auto-injected, agent overrides blocked), (3) removal of cloud/remote-daemon and local-profile sections we don't use, (4) the "js() does NOT parse JSON" bullet, (5) the "a screenshot is not a data source" rule. See docs/browser-use-0.13-migration.md. --> > **browser-use 0.13 changed everything.** There are **no subcommands**. You > no longer run `browser-use open <url>`. Instead you pipe Python helper code > to `browser-use` via a heredoc; helpers are pre-imported and a background > daemon is attached automatically. > **Store Tasks:** `browser-use` is a per-store wrapper script that > auto-injects `BU_NAME` (the store session) and `BU_CDP_WS` (the store's CDP > proxy). You **cannot** set `BU_NAME`, `BU_CDP_URL`, `BU_CDP_WS`, or `--mcp` > yourself — the wrapper blocks them. Use the default session for the store's > seller center, or `--session <slug>-aux` for non-seller-center sites (the > wrapper maps this to the aux session; no other `--session` value is > allowed). > > **No-store (orchestrator) Tasks:** `browser-use` is the store-less `web` > wrapper (`bin/_web`). Use it only for neutral public web work (search, > tracking/logistics, research) — NEVER for a store's seller center or to log > into store/platform accounts (create a store sub-task for those). # Browser Automation with browser-use (0.13, heredoc interface) Drive the browser by piping Python to `browser-use`. Helpers are pre-imported; the harness calls `ensure_daemon()` before running your code, so the browser attaches automatically to the store's CDP endpoint. ```bash browser-use <<'PY' new_tab("https://example.com") # first navigation is new_tab(), NOT goto wait_for_load() print(page_info()) PY ``` The wrapper takes the heredoc form **only** — there is no `-c` flag (passing one just prints usage). Put every statement inside the heredoc. ## Prerequisites ```bash browser-use --doctor # verify installation / CDP connectivity ``` ## Budget every invocation: the wrapper kills it at 120 s The store wrapper runs the real `browser-use` under a **120-second alarm**, and a timeout is treated as evidence the *browser* is wedged: the wrapper bumps a strike counter, force-`--reload`s the daemon, and on the **second consecutive** timeout prints `UNRECOVERABLE`, exits 75, and tells you to stop retrying and report a gap. Any other outcome — even an ordinary Python error — resets the counter. **A slow-but-healthy call is indistinguishable from a wedged browser.** So a legitimately long heredoc does not merely get cut off: two of them in a row will convince the wrapper (and then you) that a perfectly good browser is dead, and the honest-reporting rule then makes you abandon real work. Keep each invocation comfortably under the limit: - **One unit of work per invocation** — one page, one SKU, one export. Drive the loop from the **shell**, not inside the heredoc, and append results to a file under `/tmp/<task>/` so progress survives. - **Count your polls.** A render-wait of `range(15)` with `sleep(3)` is 45 s on its own; two of those plus navigation overruns 120 s. - If you genuinely need a long single operation (a slow export), poll it across **separate** invocations rather than sleeping inside one. A timeout also leaves the tab mid-operation, so re-read state at the start of the next invocation instead of assuming where you left off. ## Core Workflow 1. **Navigate**: `new_tab(url)` — for the first page **and every later navigation**. There is **no `page` object** in the heredoc scope, so `page.goto(url)` raises `NameError`; use `new_tab(url)` (or click a link) to move around. 2. **Understand state — via the DOM (PREFERRED, no vision needed):** `page_info()` for page-level facts (url/title/size), and **`js(...)` to read the DOM** — text content AND every element's on-screen coordinates via `getBoundingClientRect` (see "Locate & click without vision"). This is the primary way to drive the browser; a non-vision model completes the whole flow this way. 3. **See the layout (OPTIONAL — vision models only):** `capture_screenshot()` returns a PNG path (`~/.vibe-seller/bh-tmp/shot.png`, overwritten each call); `print()` it and **Read that PNG** to view it. Use it only to disambiguate a crowded layout — never *depend* on it. If your model can't view images, skip screenshots entirely and use step 2. **A screenshot is not a data source.** Never take a value you will act on or report — an ID, a number, a row, a column that exists, a count — from an image. Read it from the DOM with `js(...)`. A screenshot answers "where is it on screen", nothing else. This is not caution about edge cases; it is the observed default. On a page holding exactly two rows, models on two different providers each described a table that did not exist — invented columns (`ACOS`, `CPC`, `Bid`), invented campaigns, invented statuses — then spent a dozen turns hunting for the data they had "seen", and one named a nonexistent campaign in its final report to the user. Nothing was wrong with the PNG. **When the DOM and the picture disagree, the DOM is right and the picture is a hallucination** — do not go looking for the difference, and never reconcile by trusting the image. 4. **Interact**: get an element's centre coords from step 2, then `click_at_xy(x, y)`; set input values with `js(...)`. Re-read with `page_info()` / `js(...)` after to confirm. 5. **After navigation**: `wait_for_load()`; if the tab is stale/internal, `ensure_real_tab()`. ## Helper API Helpers are pre-imported into the heredoc namespace: ```python new_tab(url) # open a new tab and navigate (use for EVERY navigation) page_info() # structured summary of the current page capture_screenshot() # → PNG path (~/.vibe-seller/bh-tmp/shot.png); LAYOUT only, never data click_at_xy(x, y) # click at pixel coordinates wait_for_load() # wait for navigation/network to settle ensure_real_tab() # switch off a stale/internal (chrome://) tab js('<javascript>') # run JS; returns the SERIALIZABLE result only cdp('Domain.method', **params) # raw CDP — params are KEYWORDS, not a dict # e.g. cdp('Page.navigate', url='...') ``` - **`js()` returns serializable values only.** `js("document.title")` and `js("return 1+1")` work; but `js("document.querySelector(...)")` returns a useless `{}` (a DOM node can't serialize). Return **numbers, strings, or plain objects/arrays** — e.g. an element's coordinates (below), not the element itself. For an element *reference* (to set a file input) use `cdp('Runtime.evaluate', expression=..., returnByValue=False)` → `objectId` (note: `cdp()` params are **keyword args**, never a positional dict — see "Uploading a file"). - **`js()` does NOT parse JSON — so do not `JSON.stringify` your result.** The value comes straight from CDP `returnByValue`, so the JS type maps to the Python type: `return [{a:1}]` gives you a **list of dicts** already, while `return JSON.stringify([{a:1}])` gives you a **`str`** that you must then `json.loads` yourself. Both directions bite: stringifying and *not* parsing makes `for row in data` iterate **characters**; not stringifying and *then* parsing raises `TypeError: the JSON object must be str, bytes or bytearray, not list`. Return the object directly and use it as-is. If you inherit code whose shape you can't be sure of, normalise once rather than guessing: ```python import json # stdlib modules: import them, don't rely on the namespace def jsjson(expr): d = js(expr) return json.loads(d) if isinstance(d, str) else d ``` Only the **helpers** above are guaranteed pre-imported. `json`, `time` and friends do happen to be reachable in the heredoc namespace today (they leak in through the harness's own `import *`), but that is an implementation detail of the wheel, not a contract — one `__all__` upstream and it stops. Import the stdlib you use. ## Locate & click an element WITHOUT vision (the preferred path) You do not need to see the page. Read the DOM and compute click coordinates from `getBoundingClientRect`, then `click_at_xy`. This drives any page — buttons, links, shadow-DOM `kat-*` components — with no screenshot: ```bash browser-use <<'PY' # one element by selector → its centre coords + text (None if not found): box = js(""" var el = document.querySelector('button.submit'); // any CSS selector if(!el) return null; var r = el.getBoundingClientRect(); return {text:(el.innerText||el.value||'').slice(0,40), x:Math.round(r.x+r.width/2), y:Math.round(r.y+r.height/2)}; """) print("target:", box) if box: click_at_xy(box["x"], box["y"]) # OR enumerate all clickables to find the right one by its text: els = js(""" return [].slice.call(document.querySelectorAll('a,button,input,[role=button],kat-button')) .map(function(el){var r=el.getBoundingClientRect(); return {text:(el.innerText||el.value||'').slice(0,40), x:Math.round(r.x+r.width/2), y:Math.round(r.y+r.height/2)};}) .filter(function(e){return e.x>0 && e.y>0;}); """) print(els) # pick the one whose text matches, then click_at_xy(it.x, it.y) PY ``` **A plain `<a href>` is a navigation, not a click.** Read the href and `new_tab(href)`. Clicking one means landing inside the *anchor's* own rect, and an anchor is usually a small target inside a much larger parent: the centre of the table row or cell holding it is typically not on the anchor at all, so `click_at_xy` hits dead space and silently does nothing — no error, no navigation, and `page_info()` still shows the old page. Reserve clicking for controls that have no href (buttons, `kat-*`, JS handlers). ### A control BELOW the fold that won't scroll into view Some pages (e.g. Amazon's "Generate Spreadsheet" popover) put the button you need **below the viewport**, inside a `kat-popover`/panel that `scrollTo`/`scrollIntoView` can't bring up — the window is short (e.g. 839px tall) and `click_at_xy` can't hit an off-screen y. Don't fight the scroll: **grow the layout viewport with CDP so the whole panel fits**, then click the (now on-screen) coordinate, then restore: ```bash browser-use <<'PY' import time cdp("Emulation.setDeviceMetricsOverride", width=1920, height=2400, deviceScaleFactor=1, mobile=False) # tall viewport time.sleep(1) box = js(""" var els=document.querySelectorAll('kat-button,button,[role=button]'); for(var i=0;i<els.length;i++){var t=(els[i].innerText||els[i].getAttribute('label')||'').trim(); if(/generate spreadsheet/i.test(t)){var r=els[i].getBoundingClientRect(); return {x:Math.round(r.x+r.width/2), y:Math.round(r.y+r.height/2)};}} return null; """) if box: click_at_xy(box["x"], box["y"]) # trusted click, now on-screen time.sleep(3) cdp("Emulation.clearDeviceMetricsOverride") # restore the real viewport PY ``` (Verified: the override raises `window.innerHeight` to the set value, so a below-fold button becomes reachable; `clearDeviceMetricsOverride` undoes it. A plain JS `.click()` still no-ops on `kat-*` — you need the trusted `click_at_xy`.) For an element inside an **open shadow root** (Amazon `kat-*`), pierce it in the selector: `document.querySelector('kat-file-upload').shadowRoot.querySelector('input')`. Set an input's value with `js("document.querySelector('#q').value='socks'")` (then click its search icon — some inputs need the click to fire events). Only the helpers above (plus Python builtins) are in scope — the heredoc runs as a plain Python script. **`time`, `json`, `re`, etc. are NOT pre-imported; `import` them yourself.** Bare `sleep 3` is a `SyntaxError` and `time.sleep(3)` without `import time` is a `NameError`. **Prefer `wait_for_load()` over sleeping** — reach for `import time; time.sleep(n)` only when you must wait on something `wait_for_load()` can't observe (e.g. an async in-page render after a click). Multiple statements run in one heredoc (this replaces `&&` chaining): ```bash browser-use <<'PY' new_tab("https://example.com/login") wait_for_load() js("document.querySelector('#email').value = 'user@example.com'") js("document.querySelector('#password').value = 'secret'") click_at_xy(640, 480) wait_for_load() print(page_info()) PY ``` ## Uploading a file to a web `<input type=file>` > **The file must be in a path the BROWSER PROCESS can read, passed in > that process's path form — NOT `/tmp`.** `setFileInputFiles` reads the > file in the browser, not the agent. Put it in the store's downloads dir > (`~/.vibe-seller/downloads/<slug>/`) — the one location guaranteed > readable on every backend. Otherwise it **silently no-ops**: > `files.length` stays 0, Submit never enables, no error. Per backend: > macOS Ziniao's Chrome is sandboxed and can't read `/tmp` (so a `/tmp` > file fails — this was the whole "the widget won't accept my file" > bug, not the widget); native-Windows Chrome has no such sandbox but the > winchrome case (WSL agent → native Windows Chrome) must pass the > **Windows-form** path (`C:\…\downloads\<slug>\file`, via `wslpath -w`) — > a `/mnt/c` or WSL path is unreadable by native Chrome. There is **no native upload helper**. Never coordinate-click the visible "Browse"/"Upload file" button expecting to then drive the OS file picker