firefox-browser

# Firefox Browser Agent Bridge Control the user's actual Firefox browser session via WebSocket. This uses their real browser with existing logins and cookies - **not** a headless browser. ## Quick Start ```bash # 0. If Firefox isn't running, start it first nohup firefox &>/dev/null & # 1. Check connection browser ping # 2. See what tabs are open browser listTabs '{}' # 3. Start a new session (recommended) browser newSession '{"url": "https://example.com"}' # 4. Read the page with interactable elements marked browser getContent '{"format": "annotated"}' ``` ## Client Usage ```bash browser <action> '<json_params>' ``` ## Actions Reference ### Session & Tab Management | Action | Description | Key Params | |--------|-------------|------------| | `listTabs` | List all open tabs across windows | - | | `newSession` | Create new tab to work in | `url` (optional) | | `newWindow` | Create a new normal window to work in | `url` (optional), `focus` (defaults true), `sandbox` (private window) | | `setActiveTab` | Switch which tab agent works on | `tabId`, `focus` | | `getActiveTab` | Get current tab info | - | ### Navigation & Page Info | Action | Description | Key Params | |--------|-------------|------------| | `navigate` | Go to URL in current tab | `url`, `wait`, `newTab` | | `getContent` | Get page content | `format`: `annotated`, `text`, `html` | | `getInteractables` | List clickable elements and inputs | `selector` (optional scope) | | `screenshot` | Capture visible area as PNG | `filename` (optional) | ### Interaction | Action | Description | Key Params | |--------|-------------|------------| | `click` | Click element | `selector`, `text`, or `x`/`y` coords | | `type` | Type into input | `selector`, `text`, `submit`, `clear` | | `fillForm` | Fill multiple fields | `fields[]` with selector/value pairs | | `waitFor` | Wait for element/text | `selector`, `text`, `timeout` | | `scroll` | Scroll the page | `y`/`x`, `selector`, `position` | | `evaluate` | Execute JavaScript and return result | `script` | ### Control Flow | Action | Description | Key Params | |--------|-------------|------------| | `fork` | Duplicate tab into multiple paths | `paths[]` with name + commands | | `killFork` | Close a fork | `fork` (name) | | `listForks` | List active forks | - | | `tryUntil` | Try alternatives until one succeeds | `alternatives[]`, `timeout` | | `parallel` | Run commands on multiple URLs | `branches[]` with url + commands | ### Cross-Origin Iframe Support Many login pages (Apple, Google, Microsoft) embed their sign-in forms inside cross-origin iframes. Standard `click`/`type`/`getContent` only target the main frame by default. Use these to work inside iframes: | Action | Description | Key Params | |--------|-------------|------------| | `listFrames` | List all frames in current tab with their URLs, inputs, and clickable elements | - | | Any action | Target a specific frame | Add `"frameId": N` to params | | Any action | Try all frames | Add `"allFrames": true` to params | **Workflow for login iframes (Apple, Google, etc.):** ```bash # 1. Discover frames and find the one with the login form browser listFrames '{}' # Returns frames with frameId, url, inputs[], clickables[] # 2. Get content from the login iframe specifically browser getContent '{"frameId": 3, "format": "annotated"}' # 3. Click/type inside the iframe browser click '{"selector": "#account_name_text_field", "frameId": 3}' browser type '{"selector": "#account_name_text_field", "text": "user@example.com", "frameId": 3}' # Alternative: let the bridge try all frames automatically browser click '{"selector": "#account_name_text_field", "allFrames": true}' browser type '{"selector": "#account_name_text_field", "text": "user@example.com", "allFrames": true}' ``` ### Authentication & Vault | Action | Description | Key Params | |--------|-------------|------------| | `autoLogin` | Auto-fill credentials from Bitwarden vault and optionally submit | `domain`, `submit` (default false) | | `vaultStatus` | Check vault lock state and credential count | - | | `vaultSync` | Re-sync vault from Bitwarden server via API key | - | | `getAuthContext` | Detect login pages, available accounts | - | | `requestAuth` | Request user approval for auth | `reason` | --- ## Rich Text Editors (ProseMirror, Lexical, Slate, etc.) The `type` and `fillForm` actions automatically handle rich text editors (ProseMirror, Lexical/Reddit, Draft.js, Tiptap, Slate, CKEditor, Quill). They use `document.execCommand("insertText")` in the page world, which works with any `contenteditable`-based editor. ```bash # Works on any rich text editor — ProseMirror, Lexical, etc. browser type '{"selector": "div[contenteditable=true]", "text": "Hello world!"}' # Response includes richEditor: true when execCommand path was used # Clear existing content and replace browser type '{"selector": ".ProseMirror", "text": "New content", "clear": true}' # Fill multiple rich text fields in a form browser fillForm '{"fields": [ {"selector": "#title", "value": "My Title"}, {"selector": "#body .ProseMirror", "value": "Article body text"} ]}' ``` No special handling needed — just use `type` or `fillForm` as normal. Falls back to `textContent` assignment if `execCommand` isn't available. --- ### 1. Start by Inspecting Available Tabs ```bash browser listTabs '{}' ``` Returns: ```json { "activeTabId": 123, "windows": [ { "windowId": 1, "focused": true, "tabs": [ {"tabId": 123, "url": "https://...", "title": "...", "active": true} ] } ], "totalTabs": 5 } ``` ### 2. Start Fresh or Pick Existing Tab ```bash # Start fresh in a tab browser newSession '{"url": "https://amazon.com"}' # Start fresh in a new normal window browser newWindow '{"url": "https://example.com"}' # Or switch to existing tab browser setActiveTab '{"tabId": 456}' ``` ### 3. Read Page with Annotated Format (Recommended) ```bash browser getContent '{"format": "annotated"}' ``` Returns content with interactive elements marked inline: ``` Product Name Here $4.99 [button: "Add to cart" | selector: #add-btn] [input:text: "search" | value: "" | selector: #search-box] [link: "View details" | href: /product/123 | selector: a.details-link] ``` This shows **what's clickable** and **where it is in context**. ### 4. Interact Using Selectors ```bash # Click using selector from annotated output browser click '{"selector": "#add-btn"}' # Or by text (prefers visible elements) browser click '{"text": "Add to cart"}' # Type into input browser type '{"selector": "#search-box", "text": "query", "submit": true}' ``` --- ## Fork: Speculative Parallel Execution When you're not sure which path is right, fork the tab and try both: ```bash # Create forks browser fork '{ "paths": [ { "name": "google-auth", "commands": [{"action": "click", "params": {"text": "Sign in with Google"}}] }, { "name": "email-auth", "commands": [{"action": "click", "params": {"text": "Sign in with Email"}}] } ] }' ``` Returns: ```json { "forked": true, "sourceTabId": 123, "forks": [ {"name": "google-auth", "tabId": 456, "url": "...", "commandResults": [...]}, {"name": "email-auth", "tabId": 789, "url": "...", "commandResults": [...]} ] } ``` Work on specific fork: ```bash browser getContent '{"format": "annotated", "fork": "google-auth"}' browser click '{"text": "Continue", "fork": "google-auth"}' ``` Kill the wrong path: ```bash browser killFork '{"fork": "email-auth"}' ``` --- ## TryUntil: Handle Uncertain UI When the exact button varies (cookie banners, A/B tests): ```bash browser tryUntil '{ "alternatives": [ {"action": "click", "params": {"selector": "#accept-cookies"}}, {"action": "click", "params": {"text": "Accept All"}}, {"action": "click", "params": {"selector": ".cookie-dismiss"}} ], "timeout": 3000 }' ``` Tries each until one succeeds. --- ## Parallel: Multiple URLs at Once Compare prices across sites: ```bash browser parallel '{ "branches": [ {"url": "https://amazon.com/product", "commands": [{"action": "getContent", "params": {"format": "text"}}]}, {"url": "https://walmart.com/product", "commands": [{"action": "getContent", "params": {"format": "text"}}]} ] }' ``` --- ## Authentication (Autonomous Login) The bridge integrates with a Bitwarden vault (via bronzewarden) for fully autonomous credential fill. No human interaction needed. ### Auto-Login Flow ```bash # 1. Navigate to the site browser navigate '{"url": "https://github.com"}' # 2. Auto-fill credentials (looks up domain in vault, fills form) browser autoLogin '{"domain": "github.com", "submit": false}'