browser-control

# Browser Control Browser Control is a **driver**, not an agent. The calling agent decides what to do; Browser Control runs deterministic Playwright code in the user's visible browser. When Browser Control is installed as an OpenCode plugin, the plugin already registers this MCP server and exposes the packaged workflow as the `@browser` skill. Do not ask the user to install this skill separately or add an MCP configuration. Browser extension approval remains a separate user step. Use one loop throughout: **inspect, act, verify**. Inspect the real page before choosing locators, act through the narrowest stable control, then verify the result through a URL or fresh page read. Never treat a successful click or human acknowledgment as proof that the task succeeded. ## Core Workflow ### 1. Run The Task Directly Start with the requested browser work. Relay-backed commands start the detached relay and wait for the extension; do not start `browser-control serve` first. ```bash browser-control execute 'return { url: page.url(), title: await page.title() }' ``` Use `browser-control doctor` only when setup or runtime behavior is unclear. `status` and `doctor` are observational and never start the relay. MCP startup, tool discovery, `skill`, and `session_current` do not contact the relay. The first operational tool call starts it if needed; relay-backed observational tools report unavailability instead of starting it. For an externally supervised relay, set `BROWSER_CONTROL_AUTOSTART=false` to make ordinary calls fail when that relay is absent instead of launching another process. Existing relay connections and explicit `relay restart` still work. Ordinary CLI/MCP/SDK calls never replace a running relay. On a build mismatch, coordinate with other agents before running `browser-control relay restart`. It preserves browser tabs and durable sessions but resets JavaScript state and snapshot refs. A busy or timed-out drain leaves the old relay running; finish recordings/captures and disconnect raw CDP clients rather than forcing a stop. ```bash browser-control doctor browser-control status --json ``` Completion: one execute returns a page result and a readable session id, or `doctor` identifies the concrete setup failure. ### 2. Choose The Page Deliberately The bundled shim 0.0.25 verifies debugger ownership during reconnection. Reload the unpacked extension after installing this shim update. DevTools-attached tabs are excluded from Browser Control's inventory. `page.title()` reads time out after five seconds if the page execution context remains unavailable; the read timeout does not close or replace the tab. A bare CLI execute creates a fresh session-owned page and prints the exact `--session <id>` continuation command, or you can pass a descriptive session name (including an optional emoji, which appears directly on the browser tab group and in-page status pill). Every later CLI call must pass that id or set `BROWSER_CONTROL_SESSION`; bare execute never guesses from human-shell current state. ```bash browser-control session new "🎙️ elevenlabs" browser-control execute --session "🎙️ elevenlabs" 'return page.url()' browser-control execute 'return page.url()' ``` MCP keeps one implicit process session. Omit `session` for that normal path, or call `session_new` and pass an explicit id when one MCP process needs multiple sessions. To control a tab already open in the user's browser, ask the user to click the Browser Control toolbar button on that tab. Select it for one execute or adopt it for sticky reuse (omit `--target-url` / `targetUrl` when only one user tab is attached): ```bash browser-control session adopt --session github browser-control session adopt --target-url github.com --session github browser-control execute --target-url github.com 'return page.url()' ``` `execute --target-url` selects a page for that call only. Continuing with just `--session` uses the session's default page, which may still be `about:blank`. For a multi-step task in an existing user tab, adopt it first. Always include `page.url()` when diagnosing an empty snapshot. Snapshot labels are compact descriptions; use `ref()` for actions rather than assuming their text is an exact Playwright accessible name. `targetUrl` and `targetIndex` select existing attached pages; they never navigate. A URL selector must match exactly one page, and URL and index selectors cannot be combined. Adoption makes that tab the session default, closes the session's previous relay-created page, and is exclusive to one Browser Control session. Reset or delete releases an adopted user tab without closing it. Adoption binds ownership and exact target identity before initializing automation. Until the first execute resolves the page, session status reports `connected: false` and `pageUrl: null`; `adoptedUrl` is the registry-selected URL, not a fresh page read. A busy page can produce `session-page/adopted-initialization-timeout` on execute: user code did not run, and the exact adopted target is retained. Retry after the page settles; do not reset or adopt a different tab to recover. Prefer adoption for authenticated browser state rather than reproducing login in a fresh page. Each relay controls one browser/profile at a time. A second extension connection cannot replace a healthy active connection. If `status` or `doctor` reports rejected competing connections, keep the extension enabled only in the intended browser/profile. To switch browsers, disconnect the incumbent extension first; creating a new execute session does not switch browsers. Completion: the selected page URL is the intended page, and later work either retains the returned session id or intentionally uses the MCP process session. ### 3. Inspect, Act, Verify Inspect before guessing roles or selectors: ```js return await snapshot() ``` Then act from the returned structure and verify the destination: ```js await ref("e12").click() await page.getByRole("heading", { name: "Settings" }).waitFor() if (!page.url().includes("/settings")) { throw new Error(`Unexpected destination: ${page.url()}`) } return { url: page.url(), heading: await page.getByRole("heading").first().innerText() } ``` Use normal Playwright first. Keep dependent interactions in one execute when they rely on transient UI such as an open menu, selected rows, hover state, or an in-progress form. If native `locator.fill()` hangs, inspect the failure before using the explicit input, textarea, or contenteditable fallback. A `target/cross-extension-page` diagnostic requires human dismissal or completion first; `fillInput` is not a way around protected extension UI: ```js await fillInput(page.getByPlaceholder("Username"), "standard_user") ``` Completion: the final return value contains evidence of the requested outcome, not merely evidence that an action was attempted. #### Forms and outward mutations A click returning, HTTP 200, a redirect, or cleared fields do not prove delivery. Register any expected navigation before clicking once, then read the destination and assert the workflow's specific receipt or confirmation. Coordinate rejection and confirmation checks with the actual page you inspected; the driver cannot infer business success from generic page text. ```ts await Promise.all([ page.waitForURL(expectedDestination, { waitUntil: "domcontentloaded", timeout: 15_000 }), ref(sendRef).click({ timeout: 15_000 }), ]) const observed = await snapshot() await expectedReceipt.waitFor({ state: "visible", timeout: 5_000 }) return { url: page.url(), observed, receipt: await expectedReceipt.innerText() } ``` `expectedDestination`, `sendRef`, and `expectedReceipt` come from the inspected workflow, not guessed selectors. For same-URL navigation, register a main-frame navigation or response wait instead. A mouse coordinate click need not wait for navigation; an immediate snapshot can still be the old document. If submission timed out or the destination cannot be read, report **unverified** and inspect once without clicking again. Re-read only after the exact tab/context settles. Never replay an uncertain send, booking, purchase, or payment. An explicit site rejection means **rejected**, not delivered; protection requiring human interaction means hand off the ordinary page to the user. Do not modify protection tokens, spoof human signals, or dispatch DOM clicks to evade a security boundary. ### 4. Continue Or Finish Cleanly Named sessions preserve their default page across short-lived CLI and MCP processes. They also survive relay restarts: Browser Control restores the id, read-only mode, and exact default target. JavaScript `state` and snapshot refs are process-local and reset after a relay restart with an explicit warning. ```bash browser-control session list browser-control session reset github browser-control session delete github ``` Deletion is idempotent for an explicit session id, so cleanup can be safely retried when that session is already absent. Every execute is journaled under `~/.browser-control/sessions/<id>/journal.jsonl`. The journal records code, status, duration, URL movement, warnings, handoffs, and bounded diagnostics. Never place credentials directly in execute source. ```bash browser-control journal --session github --limit 50 ``` Completion: retain the session only when follow-up work is expected; otherwise reset or delete session-owned pages and report any warnings that affect later work. ## Canonical Authenticated Flow The distinguishing Browser Control workflow is an authenticated tab plus a human-only prompt: Attach and adopt the existing tab, inspect its real UI, fill ordinary fields, then register `handoff` before triggering WebAuthn, 2FA, CAPTCHA, or payment UI. After the user completes it, verify the authenticated destination. The same session can continue after an MCP process or relay restart. When the prompt-triggering action may itself block, put only that action in `start`. Browser Control presents and acknowledges WAIT before invoking it: ```js await handoff("Complete the security-key prompt, then continue", { timeoutMs: 600_000, start: () => page .getByRole("button", { name: /passkey|security key|sign in/i }) .click({ timeout: 600_000 }), }) await page.waitForURL((url) => !url.pathname.startsWith("/login")) await page.getByRole("heading", { name: /account|dashboard/i }).waitFor() return { authenticatedUrl: page.url(), title: await page.title() } ``` After a resolved handoff, Browser Control waits through transient destination context replacement before returning, so this verification can remain in the same execute. With `start`, the handoff deadline and target cancellation remain active until the action settles, even if the user has already pressed Continue. An early acknowledgment does not authorize an indefinitely pending action. For a handoff on another page, pass `{ page: otherPage }`. Readiness checks that page, not the session default. If a non-default page was replaced or closed, inspect the remaining pages rather than assuming an old Playwright reference now identifies its replacement. Tell the user what action is waiting. Human acknowledgment is not verification: always assert the expected URL or stable element after `handoff`. If the action was already completed and only the human step remains, call `handoff(message)` without `start`. The default timeout is ten minutes. Completion: the prompt was presented only after WAIT was registered, the action settled, and the authenticated result was independently verified. To turn a user-demonstrated flow into reusable Playwright, use `demonstrate()`. It uses the same exact-tab handoff, records clicks, edits, checkbox/select changes, and same-tab navigations, then returns editable code: ```ts return await demonstrate("Perform the workflow once, then continue") ``` Password fields become explicit secret-source comments rather than copied values. Review selectors and add outcome assertions before reusing generated code; a demonstration records actions, not proof that the workflow succeeded. ### Password Manager Prompts Ordinary webpage fields and accessible open-shadow-root controls remain usable. 1Password's inline menus are extension-owned iframes, not ordinary webpage DOM. Chromium blocks one extension from debugging another extension's pages; toolbar popups and native unlock, Touch ID, and Windows Hello prompts are not supported Playwright control surfaces. Focusing or filling a card-number or credential field can open the inline menu by itself, even inside a third-party payment iframe. While it is open, Chrome rejects every automation command for that tab; Playwright shows this as "Execution context was destroyed" or a locator timeout, so the execute result carries the `target/cross-extension-page` diagnostic and warning, and `browser-control status` marks the tab `protected-ui=true`. `target/cross-extension-page` means a permission boundary. Ask the user to finish or dismiss the prompt and retry; do not reset the page, read vault contents, or weaken browser security to get around it. The page itself is healthy: do not treat the failure as an unresponsive tab or create a new page. Register `handoff` on the originating webpage before triggering a human-only prompt when possible. If the prompt already prevents attachment, give the user the required action directly rather than assuming an in-page handoff can be displayed. Verify the intended webpage state after the prompt is completed. A page's leftover password-manager status text is not evidence that its menu is still open. Use current relay diagnostics and a fresh ordinary-page read after human dismissal, without inspecting the extension's private frame contents. ## Inspection Tools Use the least expensive view that answers the question: - `snapshot()` is the compact read-before-act default. It prioritizes semantic groups, alerts, lists, tables, headings, links, and controls. Text input and textarea values are omitted. - The default snapshot scopes to a single visible modal when present; portal dialogs outside `main` remain discoverable. Use an explicit `within` scope when you intentionally need background content. Repeated list wrappers have a bounded reservation so product links can still fit in a dense page. - Native number/search inputs are `spinbutton`/`searchbox`. Native disclosure controls are labeled `summary`, which is an element kind rather than an ARIA role; use the returned `ref()` to operate them. - `ref("e12")` resolves a control from the latest snapshot. Refs fail closed after navigation or incompatible DOM drift. Compatible refs keep the same id across repeated same-document captures. - `snapshot({ diff: true })` reports semantic changes from the compatible prior baseline. `snapshot({ delta: true })` returns a full first baseline and compact deltas afterward. Existing compatible refs remain usable. - `snapshot({ find: "checkout", context: 2 })` searches the bounded semantic snapshot and returns matching lines with nearby context and actionable refs. - `ariaSnapshot(target?, { timeout })` returns Playwright's detailed YAML aria tree when the compact snapshot omits needed structure. Native text-control values, custom ARIA range values, and editable content are omitted so they do not enter tool output. Await it separately; do not run other operations on the same page concurrently. - `screenshotWithLabels({ page?, path? })` adds visual labels and metadata when layout matters, and registers its `e1..eN` labels for `ref()`. - `screenshotDiff({ baseline, path?, threshold?, fullPage? })` compares a saved PNG (absolute path or Buffer) with the current session page at CSS-pixel scale. It returns `matches`, `changedPixels`, `changedRatio` (0..1), dimensions, and a red-highlighted PNG. Omit `path` to return the image as execute media; otherwise supply a fresh absolute `.png` path. Existing output files are never overwritten. ```js return await snapshot({ within: "main", maxItems: 200 }) return await snapshot({ find: /checkout|payment/i, context: 2 }) return await snapshot({ delta: true }) // When layout matters, return the image through MCP so it can be inspected. return await screenshotWithLabels({ page }) ``` Saving an image and returning only `"ok"` proves file creation, not visual correctness. Return screenshot buffers through MCP when visual evidence matters. For visual regression checks, save a baseline before changing the UI: ```ts await page.screenshot({ path: "/absolute/before.png", scale: "css" }) // After the intended UI change, in the same viewport: return await screenshotDiff({ baseline: "/absolute/before.png" }) ``` The threshold defaults to 0.1 and controls per-pixel color tolerance, not the allowed changed area. Set it to 0 for exact pixels. Antialiasing changes count. Settle animations yourself and use the same viewport and `fullPage` setting for both captures. Dimension mismatches fail explicitly; images are never resized. Comparisons are limited to PNGs of 32 MiB / 16 megapixels each. Screenshots and diffs include visible page content: inspect for private information before sharing. ## Execute Interface Execute code can use `page`, `context`, `browser`, persistent `state`, selected Node modules through `modules` and aliases such as `fs` and `path`, plus the Browser Control helpers documented here. Execute code runs in Node. Use `page.evaluate` for `window`, `document`, storage, and same-origin `fetch` with page cookies. Single expressions auto-return; multi-statement scripts need `return`. Use `--file` for longer scripts: ```bash browser-control execute --session github --file ./perform-flow.js ``` Human CLI output includes logs, warnings, and a concise aftermath. Use `--json` when another command needs to branch on `ok`, `value`, `error`, `warnings`, or `aftermath`: ```bash browser-control execute --json --session github '({ url: page.url() })' | jq .value.url ``` `page.waitForEvent("download")` intercepts both `Content-Disposition: attachment` responses (via CDP `Fetch`) and `<a download>` / `blob:` / `data:` links without opening Chrome's native Save confirmation dialog, returning a `Download` with `suggestedFilename()`, `await dl.path()`, and `await dl.saveAs(targetPath)`: ```ts const dl = page.waitForEvent("download", { timeout: 30_000 }) await page.getByRole("button", { name: /Generate a private key/i }).click() const download = await dl await download.saveAs("/tmp/private-key.pem") return { filename: download.suggestedFilename(), savedPath: await download.path() } ``` Pages with WebMCP enabled can expose structured page tools. Discover and call them through the execute helper; names, descriptions, schemas, and results come from the page: ```ts const tools = await webmcp.list() const result = await webmcp.call("search_catalog", { query: "adapter" }) return { tools, result } ``` Discovery covers all frames. If the same name appears in multiple frames, pass the exact reported frame label as `{ frame }`. Browser Control re-discovers the tool immediately before invoking it, so stale registrations fail directly. ## Safety Browser Control blocks CDP commands that would destroy shared browser state, including browser close and cookie/cache clearing. Never work around those guardrails. For inspect-only work, use a read-only session: ```bash browser-control session new inspect-prod --read-only browser-control execute --session inspect-prod 'await page.goto("https://example.com"); return page.title()' ``` Read-only sessions reject `Input.*`, so they cannot click or type through Playwright. `page.evaluate` can still mutate the DOM; read-only prevents trusted mistakes, not malicious code. For destructive UI work, use a two-phase **read, confirm, verify** flow: 1. Read candidates and return exact stable identifiers or row text. 2. Obtain user approval for those exact items. 3. Re-select only approved items and assert the selected count. 4. Read the confirmation dialog and throw unless it matches the approved action. 5. Confirm, then verify through a fresh page read or independent CLI/API path. Do not discover and confirm destructive candidates in one script unless the user already approved exact stable identifiers. Never globally auto-accept native dialogs; wait for the expected dialog and assert its type and message before accepting it. ## TypeScript Client Applications and CLI tools can use `BrowserControlClient.origin()` for direct same-origin JSON requests authenticated by a session page (with optional default headers and automatic in-page `handoff` recovery when a session expires or redirects to login): ```ts import { BrowserControlClient } from "@opencode-ai/browser-control" const ubereats = BrowserControlClient.origin("https://www.ubereats.com", { session: "🥤 karma-cafe", startUrl: "/feed", headers: { "x-csrf-token": "x" }, handoffOnAuthFailure: true, }) const store = await ubereats.post("/_p/api/getStoreV1", {