web-pilot

# Web Pilot ## Tools | Tool | Purpose | |------|---------| | `sandbox_browser_observe` | Text snapshot: URL, title, headings, textExcerpt, elements with refs | | `sandbox_browser_navigate` | Go to URL, wait for ready, return snapshot with textExcerpt | | `sandbox_browser_click` | Click by ref (e.g. "e3") or visible text | | `sandbox_browser_type` | Type into input by ref (e.g. "e5") | | `sandbox_browser_content` | Page as markdown (optional CSS selector) | | `sandbox_browser_evaluate` | Run JS, reports side effects | | `sandbox_browser_screenshot` | PNG screenshot (expensive, last resort) | | `sandbox_web_search` | Search DuckDuckGo for URLs | ## Workflow: Observe → Act → Observe 1. **Navigate** → returns snapshot with element refs 2. **Act** using refs from snapshot: `click({ ref: "e3" })` or `type({ ref: "e5", text: "hello" })` 3. **Observe** → verify action worked 4. Repeat. Screenshot only if text can't answer your question. ## Refs Observe output lists elements like: ``` - button "Submit" [ref=e1] - textbox [ref=e2] name=email placeholder="Enter email" - link "Home" [ref=e3] ``` Use refs in click/type: `{ "ref": "e1" }`. Refs update on each observe — always use latest. ## Data Extraction Strategy 1. **Check textExcerpt first** — observe/navigate returns the first 2000 chars of visible text. For many tasks this is enough. 2. **Use content(selector)** over content() — target a specific section (e.g. `{ selector: "main" }` or `{ selector: ".repo-list" }`) to avoid huge dumps. 3. **Use evaluate() for structured extraction** — `evaluate({ expression: "..." })` with JS that returns exactly the data you need (e.g. `[...document.querySelectorAll('.repo')].map(...)`) 4. **Prefer web_fetch for read-only pages** — if you just need page content without interaction, `sandbox_web_fetch` is cheaper than navigating. 5. **Never repeat a failed approach** — if a tool call returns nothing useful, switch to a different strategy immediately. ## Patterns - **Click by text:** `{ "text": "Accept cookies" }` (when ref unavailable) - **Scroll:** evaluate `window.scrollBy(0, 500)` then observe - **Dropdown:** evaluate with value set + change event dispatch