browser

# Browser Control Control Chrome browser via `agbrowse` commands. Uses ref-based snapshots to identify page elements, then click/type by ref ID. ## Positioning (first-run note) agbrowse is a **local Chrome / CDP** runtime. It deliberately does **not** offer: - hosted / cloud / managed browser sessions (see Browserbase, Browser Use Cloud, Vercel agent-browser cloud-session flag for that shape) - remote / external CDP endpoints (deferred — see `docs/EXTERNAL_CDP.md`) - stealth, anti-detection, CAPTCHA bypass, or Cloudflare bypass - benchmark / leaderboard score claims If you need hosted infrastructure or detection evasion, agbrowse is not the tool — use a hosted browser provider. Compare positioning in [`docs/comparison.md`](../../docs/comparison.md). The release gate `gate:no-cloud-claims` enforces this in CI. ## Prerequisites - Node.js 18+ - Google Chrome (or Chromium/Brave) installed - `playwright-core` installed: ```bash cd <project-root> npm install playwright-core ``` ## Quick Start ```bash agbrowse start # Start Chrome (CDP auto port) agbrowse start --headless # Headless mode (server/CI/WSL) agbrowse navigate "https://example.com" # Go to URL agbrowse search "query" --json # Standalone deep search agbrowse fetch "https://example.com" --json # Read one candidate URL agbrowse extract "https://example.com/table" --schema spec.json --json # Structured extraction agbrowse snapshot --interactive # Interactive elements with ref IDs agbrowse click e3 # Click ref e3 agbrowse type e5 "hello" --submit # Type + Enter agbrowse screenshot # Save screenshot agbrowse reload # Reload current page ``` For AI provider websites, use the bundled `web-ai` skill/command instead of raw click/type sequences when possible: ```bash agbrowse web-ai status --vendor chatgpt agbrowse web-ai query --vendor gemini --url https://gemini.google.com/app --inline-only --prompt "Reply exactly OK" ``` For Runway, use the dedicated task-runner surface instead of `web-ai`. The initial Runway command is read-only except navigation, focuses Apps and Custom/tools, and never clicks `Generate`, `Run all`, payment, destructive, or submit-like controls. For live smoke tests where the user explicitly submits generation jobs, treat Runway Unlimited as a queue-capped task runner: allow at most 2 active jobs, then poll completion signals for up to 10 minutes per model. The poller is Playwright/CDP-based: it reads Runway DOM state, not Computer Use state. Treat `In queue`, `Generating`, `Processing`, `loading animation`, and right-rail percentage labels such as `18 50%` as active generation signals. Do not treat two active jobs as terminal completion. Record `queue_full` only when Runway shows the explicit `You're on a roll` / `Credits Mode` gate. ```bash agbrowse runway selectors --surface apps agbrowse runway status --surface auto --json agbrowse runway preflight --surface custom-tools --json agbrowse runway poll --timeout 600000 --interval 5000 --queue-limit 2 --after-count 17 --expected-item "18" --json ``` ## Core Workflow > **Always follow this pattern:** > 1. `snapshot --interactive` → See elements + ref IDs > 2. `click`/`type`/`press`/`select` → Interact using ref > 3. `snapshot` → Verify result → Repeat ## Commands ### Browser Management ```bash agbrowse start [--port <9222>] [--headless|--headed] [--chrome-path /path/to/chrome] agbrowse stop agbrowse status agbrowse reset [--force] ``` ### Observe ```bash agbrowse snapshot # Ref snapshot (all elements) agbrowse snapshot --interactive # Interactive elements only (recommended) agbrowse snapshot --max-nodes 30 # Limit output for token budget agbrowse screenshot # Current viewport agbrowse screenshot --full-page # Full page agbrowse screenshot --ref e5 # Specific ref element only agbrowse screenshot --clip 0 0 320 180 # Clipped region in CSS pixels agbrowse screenshot --json # JSON output (path, dpr, viewport) agbrowse text # Page text content agbrowse text --format html # HTML source agbrowse get-dom # Full DOM HTML agbrowse get-dom --selector ".card" --max-chars 2000 agbrowse console --clear --reload --duration 3000 # Buffered console logs agbrowse network --reload --duration 1000 # Fresh page-load + async requests ``` ### Adaptive URL Fetch (v2) Use `agbrowse fetch` after a candidate URL already exists. Do not use it as the first step for broad generic search. See Command Routing below for when to use `fetch` vs `search` vs `extract`. ```bash agbrowse fetch "https://example.com/article" agbrowse fetch "https://example.com/article" --json --trace agbrowse fetch "https://example.com/article" --browser never agbrowse fetch "https://example.com/article" --browser required agbrowse fetch "https://example.com/article" --allow-third-party-reader agbrowse fetch "https://example.com/article" --browser-session user agbrowse fetch "https://example.com/article" --identity chrome ``` #### Command Routing Pick the right command by what you have and what you need: ```text generic search request -> agbrowse search (or external search tool) known / candidate URL to read -> agbrowse fetch structured data from URL + JSON schema -> agbrowse extract prompting AI providers (ChatGPT, Gemini, Grok) with a question -> agbrowse web-ai (web-ai skill) ``` #### QA / Playwright Intent Routing When a user asks to "run Playwright", "플레이라이트로 확인", "browser QA", "브라우저 테스트", or equivalent without naming an existing test file, treat the phrase as an intent to verify the page, not as a dependency requirement. Use agbrowse first: ```text page opens and renders -> navigate -> snapshot -> screenshot critical interaction works -> snapshot --interactive -> click/type -> snapshot runtime errors or failed requests -> console/network with --reload public URL/source claim is correct -> search skill -> agbrowse search --verify/fetch maintained project E2E regression -> run/extend the repository's existing Playwright suite ``` Do not install a separate Playwright runner, Playwright MCP, Puppeteer, or browser driver for ad-hoc QA. agbrowse already exposes the required browser control through its CLI and uses `playwright-core` internally. Preserve an existing project-owned Playwright E2E suite when the request explicitly targets that regression surface. #### Escalation Ladder (code execution order in index.mjs) 1. **Public endpoints + direct fetch** — known API resolvers (GitHub, Reddit, HN, Wikipedia, npm, PyPI, arXiv, Bluesky, Mastodon, Stack Exchange, dev.to, DOI/CrossRef, OpenLibrary, Wayback CDX, YouTube/X oEmbed, HN Algolia, V2EX, Lobsters, generic oEmbed), direct HTTP with identity headers, discovered RSS/Atom feeds, metadata extraction 2. **Third-party readers** — opt-in public readers like Jina (`--allow-third-party-reader`) 3. **Isolated Chrome render** — fresh Chrome profile + in-page Defuddle main-content→markdown extraction (vendored bundle; evidence `browser-defuddle`) + network API JSON discovery 4. **User session** — user's authenticated browser session (`--browser-session user`, explicit opt-in) 5. **Human-in-the-loop** — human resolves challenges (`--browser-session interactive`, 5-minute timeout) Content scoring runs after each phase to decide whether to escalate. #### Key Flags | Flag | Values | Default | Description | |------|--------|---------|-------------| | `--browser` | `auto\|never\|required` | `auto` | Browser escalation mode | | `--browser-session` | `none\|isolated\|existing\|user\|interactive` | `isolated` | Session/cookie boundary | | `--identity` | `auto\|minimal\|chrome` | `auto` | Request identity headers (`auto` and `chrome` send browser-grade headers; `minimal` sends only Accept) | | `--no-browser` | — | — | Alias for `--browser never` | | `--allow-third-party-reader` | — | — | Enable Jina Reader | | `--no-public-endpoints` | — | — | Skip known public endpoint resolvers | | `--max-bytes` | number | `1048576` | Maximum response bytes per read (1 MB) | | `--timeout-ms` | number | `15000` | Per-attempt timeout | | `--selector` | CSS selector | — | Browser text extraction selector | | `--allow-archive` | — | — | Accepted but deferred; emits a warning | | `--trace` | — | — | Include all attempt traces | | `--json` | — | — | JSON output | #### JSON Contract `--json` output is always intended to parse as one JSON object. Large selected content is compacted before serialization and annotated with: - `contentBytes`: original selected content size in UTF-8 bytes - `contentLimitBytes`: content bytes retained in CLI JSON output - `contentTruncated`: true when `content` was shortened for output safety Do not treat `contentTruncated` as a failed fetch. Use `verdict`, `source`, `finalUrl`, `warnings`, and `attempts` to decide whether another escalation is needed. #### Agent Workflow When an agent needs to read a URL: 1. Use a search tool to discover candidate URLs first. 2. Run `agbrowse fetch <url> --json --trace --browser never` for HTTP-only reading. 3. If `verdict` is `weak_ok` or `blocked`, inspect `attempts` before escalating. 4. Public endpoints, RSS/Atom, oEmbed, and metadata are tried before any browser step — do not skip them. 5. Use `--allow-third-party-reader` only when the user or task allows a public third-party reader. 6. Use `--browser auto --browser-session isolated` for JS-rendered pages when a browser is needed. 7. Use `--browser-session user` or `interactive` only for the user's own authenticated browser state and human-supervised challenge handling. 8. Report boundaries plainly: `blocked`, `auth_required`, `paywall`, `challenge`, or `browser_required`. These are observations, not immediate stops — continue the legitimate ladder before the final boundary verdict. Do not treat CAPTCHA, login, or paywall markers as "stop immediately" signals. Do not claim that browser session mode "bypasses" paywalls — it uses the user's own already-authorized browser state. Do not claim challenge resolution is automated — it is human-supervised with a timeout. #### Safety Model - Automated CAPTCHA solving, credential stuffing, stealth libraries: **forbidden** - Human assistance (browser-grade headers, user session, human resolves): **allowed with explicit opt-in** - DNS rebinding guard enforced in fetch path and redirect chain — blocks hostnames resolving to private/loopback IPs (both A and AAAA records) - User session final URL validated — redirects to private networks are rejected - `safetyFlags` in result track which elevated capabilities were used (`user_session_used`, `human_action_taken`) #### WAF Detection Detects Cloudflare (managed challenge + Turnstile), Akamai Bot Manager, AWS WAF, Imperva/Incapsula, DataDome, and PerimeterX from response headers before browser escalation. WAF profile informs challenge classification and wait strategies. ### Schema-bound Structured Extraction `agbrowse extract` pulls structured data from a page into a validated JSON envelope, driven by a JSON schema you supply. It is LLM-free by default (Tier 1) and fail-closed: when no candidate structure in the page satisfies the schema, it returns `verdict: "no_mappable_structure"` with a summary of what was considered. It never silently returns partial data. ```bash # Tier 1 (LLM-free, default) -- fetch HTML, find tables/JSON-LD, map to schema agbrowse extract "https://example.com/pricing" --schema pricing.json --json # From a local HTML file instead of fetching agbrowse extract --from-file page.html --schema pricing.json --json # Restrict source type agbrowse extract "https://example.com" --schema spec.json --source table --json agbrowse extract "https://example.com" --schema spec.json --source jsonld --json # Tier 2 opt-in: on Tier 1 failure, escalate to a web-ai vendor session agbrowse extract "https://example.com" --schema spec.json --escalate-web-ai --json agbrowse extract "https://example.com" --schema spec.json --escalate-web-ai --vendor chatgpt --json ``` #### Options | Flag | Values | Default | Description | |------|--------|---------|-------------| | `--schema` | file path | *(required)* | JSON schema file (`extract-schema-v1` subset) | | `--from-file` | file path | -- | Read local HTML instead of fetching a URL | | `--source` | `auto\|table\|jsonld` | `auto` | Which page structures to consider | | `--json` | -- | -- | Machine-readable JSON envelope | | `--escalate-web-ai` | -- | -- | Tier 2: web-ai fallback on Tier 1 failure | | `--vendor` | `chatgpt\|gemini\|grok` | `grok` | Tier 2 vendor | | `--timeout` | ms | -- | Fetch timeout | #### Tier Architecture **Tier 1 (default, LLM-free):** `fetchTextCandidate` raw HTML -> structured extractor (tables, JSON-LD) -> schema mapping (header-to-property matching, conservative type coercion) -> `validateExtraction`. Fail-closed with `verdict: "no_mappable_structure"` when no candidate satisfies all required schema properties. **Tier 2 (`--escalate-web-ai`, opt-in):** Activated only on Tier 1 failure. Sends page text + schema to a logged-in web-ai vendor session (default: grok) and validates the response JSON through the same validator. JS-rendered pages are a Tier 1 non-goal. Client-rendered content that never appears in the raw HTML is reported as fail-closed rather than guessed. #### JSON Envelope `--json` output uses the `agbrowse-extract-v1` envelope: | Field | Description | |-------|-------------| | `schemaVersion` | `"agbrowse-extract-v1"` | | `ok` | `true` if extraction succeeded | | `tier` | `1` or `2` | | `verdict` | e.g. `"ok"`, `"no_mappable_structure"` | | `data` | Extracted array/object or `null` | | `errors` | Validation errors (if any) | | `candidatesConsidered` | Number of structures evaluated | | `source` | `"table"`, `"jsonld"`, or `"web-ai"` | | `finalUrl` | Resolved URL after redirects | | `evidence` | Extraction trace entries | ### Snapshot Output Example ``` e1 link "Gmail" e2 link "Images" e3 textbox "Search" ← To type here: type e3 "query" e4 button "Google Search" ← To click: click e4 e5 button "I'm Feeling Lucky" ``` ### Act ```bash agbrowse click e3 # Click element agbrowse click e3 --double # Double-click agbrowse click e3 --right # Right-click / context menu agbrowse type e3 "hello" # Type text agbrowse type e3 "hello" --submit # Type + press Enter agbrowse press Enter # Press key agbrowse press Escape agbrowse press Tab