agentbrowser-desktop-mcp

name	agentbrowser-desktop-mcp
description	Use the local AgentBrowser MCP server to control the browser and full virtual desktop. Trigger: When the AI needs to operate the browser appliance, use Firefox via desktop control, use Chromium via CDP/browser tools, or drive the desktop autonomously from OpenCode or another MCP client.
license	Apache-2.0
metadata	{"author":"aitorroma","version":"2.0"}

When to Use

• When the task requires controlling the local browser appliance through MCP
• When the agent must operate Firefox without CDP
• When the agent must use mouse, keyboard, clipboard, or windows on the virtual desktop
• When OpenCode or another MCP client needs a ready-to-use config for this project

Critical Patterns

• MCP endpoint: http://127.0.0.1:18787/mcp
• Use browser tools for Chromium/CDP workflows:
  • goto, fill, click, eval, screenshot, get_markdown
• Use desktop tools for Firefox or whole-desktop workflows:
  • screen_shot, mouse_move, mouse_click, mouse_drag, mouse_scroll
  • key_type, key_press
  • clipboard_get, clipboard_set
  • window_list, window_focus
  • app_list, app_launch, app_status
• Prefer Firefox + desktop tools for "human-like" browsing
• Prefer Chromium + browser tools when deterministic DOM control or CDP is needed
• Before typing, usually do:
  1. app_launch() if needed
  2. window_list()
  3. window_focus()
  4. key_press("ctrl+l") or click target
  5. key_type(...)
• For desktop validation, always finish with screen_shot()

Decision Table

Goal	Preferred toolset
Navigate browser reliably by DOM	Browser MCP tools
Extract rendered markdown	get_markdown(url)
Use Firefox	Desktop MCP tools
Handle popups, file pickers, dialogs, downloads	Desktop MCP tools
Operate full GUI like a human	Desktop MCP tools

File Manager (Dolphin)

Launch (via MCP)

niri_msg(args=["action", "spawn", "--", "dolphin"])
niri_msg(args=["action", "spawn", "--", "dolphin", "/path/to/dir"])
niri_msg(args=["action", "spawn", "--", "dolphin", "--new-window", "/path"])
niri_msg(args=["action", "spawn", "--", "dolphin", "--split", "/path"])
niri_msg(args=["action", "spawn", "--", "dolphin", "--select", "file.txt"])

D-Bus Control (via MCP)

First focus Dolphin, then send keyboard shortcuts:

1. niri_windows() → find org.kde.dolphin → get id
2. niri_msg(args=["action", "focus-window", "--id", "<id>"])
3. agentbrowser_key_press(keys="<shortcut>")

Shortcut	Action
ctrl+l	Open location bar (type path)
ctrl+shift+n	New window
ctrl+shift+t	New tab
F5	Reload
alt+Up	Go to parent
alt+Left	Go back
alt+Right	Go forward
F3	Toggle split view
ctrl+shift+c	Copy
ctrl+shift+v	Paste
Delete	Move to trash

Navigate to Path (via MCP)

1. niri_windows() → find org.kde.dolphin → get id
2. niri_msg(args=["action", "focus-window", "--id", "<id>"])
3. agentbrowser_key_press(keys="ctrl+l")
4. agentbrowser_key_type(text="/path/to/dir")
5. agentbrowser_key_press(keys="Return")

Window Management (via MCP)

niri_windows()                                        # List all windows
niri_msg(args=["action", "close-window", "--id", "<id>"])   # Close
niri_msg(args=["action", "focus-window", "--id", "<id>"])   # Focus
niri_msg(args=["action", "fullscreen-window"])              # Fullscreen
niri_msg(args=["action", "center-column"])                 # Center

Accessibility Tree (via MCP)

agentbrowser_cua_call(tool="get_accessibility_tree")
agentbrowser_cua_call(tool="get_window_state", input={"pid": <dolphin_pid>})

Window Management (niri)

All window operations use niri_msg(args=[...]). List windows first to get IDs.

List Windows

niri_windows()

Returns: {id, app_id, title, pid, workspace_id, is_focused, layout:{tile_size}}

Close Window

niri_msg(args=["action", "close-window", "--id", "<id>"])

Focus Window

niri_msg(args=["action", "focus-window", "--id", "<id>"])

Fullscreen / Toggle

niri_msg(args=["action", "fullscreen-window"])
niri_msg(args=["action", "toggle-windowed-fullscreen"])

Move Window

niri_msg(args=["action", "move-column-left"])
niri_msg(args=["action", "move-column-right"])
niri_msg(args=["action", "move-column-to-first"])
niri_msg(args=["action", "move-column-to-last"])
niri_msg(args=["action", "move-column-to-index", "--index", "<n>"])
niri_msg(args=["action", "move-window-down"])
niri_msg(args=["action", "move-window-up"])
niri_msg(args=["action", "move-window-down-or-to-workspace-down"])
niri_msg(args=["action", "move-window-up-or-to-workspace-up"])
niri_msg(args=["action", "swap-window-right"])
niri_msg(args=["action", "swap-window-left"])
niri_msg(args=["action", "consume-or-expel-window-left"])
niri_msg(args=["action", "consume-or-expel-window-right"])
niri_msg(args=["action", "consume-window-into-column"])
niri_msg(args=["action", "expel-window-from-column"])

Resize Window

niri_msg(args=["action", "set-window-width", "--width", "<px>"])
niri_msg(args=["action", "set-window-height", "--height", "<px>"])
niri_msg(args=["action", "reset-window-height"])
niri_msg(args=["action", "set-column-width", "--width", "<px>"])
niri_msg(args=["action", "expand-column-to-available-width"])

Preset Sizes

niri_msg(args=["action", "switch-preset-column-width"])
niri_msg(args=["action", "switch-preset-column-width-back"])
niri_msg(args=["action", "switch-preset-window-width"])
niri_msg(args=["action", "switch-preset-window-width-back"])
niri_msg(args=["action", "switch-preset-window-height"])
niri_msg(args=["action", "switch-preset-window-height-back"])

Maximize

niri_msg(args=["action", "maximize-column"])
niri_msg(args=["action", "maximize-window-to-edges"])

Center

niri_msg(args=["action", "center-column"])
niri_msg(args=["action", "center-window"])
niri_msg(args=["action", "center-visible-columns"])

Floating vs Tiling

niri_msg(args=["action", "toggle-window-floating"])
niri_msg(args=["action", "move-window-to-floating"])
niri_msg(args=["action", "move-window-to-tiling"])
niri_msg(args=["action", "focus-floating"])
niri_msg(args=["action", "focus-tiling"])
niri_msg(args=["action", "switch-focus-between-floating-and-tiling"])
niri_msg(args=["action", "move-floating-window", "--x", "<px>", "--y", "<px>"])

Tabbed Display

niri_msg(args=["action", "toggle-column-tabbed-display"])
niri_msg(args=["action", "set-column-display", "--display", "tabbed|normal"])

Opacity

niri_msg(args=["action", "toggle-window-rule-opacity"])

Urgent Status

niri_msg(args=["action", "toggle-window-urgent"])
niri_msg(args=["action", "set-window-urgent"])
niri_msg(args=["action", "unset-window-urgent"])

Focus Navigation

Between Columns

niri_msg(args=["action", "focus-column-left"])
niri_msg(args=["action", "focus-column-right"])
niri_msg(args=["action", "focus-column-first"])
niri_msg(args=["action", "focus-column-last"])
niri_msg(args=["action", "focus-column-right-or-first"])
niri_msg(args=["action", "focus-column-left-or-last"])
niri_msg(args=["action", "focus-column", "--index", "<n>"])

Between Windows

niri_msg(args=["action", "focus-window-down"])
niri_msg(args=["action", "focus-window-up"])
niri_msg(args=["action", "focus-window-down-or-column-left"])
niri_msg(args=["action", "focus-window-down-or-column-right"])
niri_msg(args=["action", "focus-window-up-or-column-left"])
niri_msg(args=["action", "focus-window-up-or-column-right"])
niri_msg(args=["action", "focus-window-top"])
niri_msg(args=["action", "focus-window-bottom"])
niri_msg(args=["action", "focus-window-down-or-top"])
niri_msg(args=["action", "focus-window-up-or-bottom"])
niri_msg(args=["action", "focus-window-previous"])
niri_msg(args=["action", "focus-window-in-column", "--index", "<n>"])

Cross-monitor / Cross-workspace

niri_msg(args=["action", "focus-window-or-monitor-up"])
niri_msg(args=["action", "focus-window-or-monitor-down"])
niri_msg(args=["action", "focus-column-or-monitor-left"])
niri_msg(args=["action", "focus-column-or-monitor-right"])
niri_msg(args=["action", "focus-window-or-workspace-down"])
niri_msg(args=["action", "focus-window-or-workspace-up"])

Workspace Management

List / Focus

niri_workspaces()
niri_msg(args=["action", "focus-workspace-down"])
niri_msg(args=["action", "focus-workspace-up"])
niri_msg(args=["action", "focus-workspace", "--reference", "<index|name>"])
niri_msg(args=["action", "focus-workspace-previous"])

Move Windows to Workspace

niri_msg(args=["action", "move-window-to-workspace-down"])
niri_msg(args=["action", "move-window-to-workspace-up"])
niri_msg(args=["action", "move-window-to-workspace", "--reference", "<index|name>"])
niri_msg(args=["action", "move-column-to-workspace-down"])
niri_msg(args=["action", "move-column-to-workspace-up"])
niri_msg(args=["action", "move-column-to-workspace", "--reference", "<index|name>"])

Reorder / Name

niri_msg(args=["action", "move-workspace-down"])
niri_msg(args=["action", "move-workspace-up"])
niri_msg(args=["action", "move-workspace-to-index", "--index", "<n>"])
niri_msg(args=["action", "set-workspace-name", "--name", "<name>"])
niri_msg(args=["action", "unset-workspace-name"])

Monitor Management

List / Focus

niri_outputs()
niri_msg(args=["action", "focus-monitor-left"])
niri_msg(args=["action", "focus-monitor-right"])
niri_msg(args=["action", "focus-monitor-down"])
niri_msg(args=["action", "focus-monitor-up"])
niri_msg(args=["action", "focus-monitor-previous"])
niri_msg(args=["action", "focus-monitor-next"])
niri_msg(args=["action", "focus-monitor", "--id", "<name>"])

Move Windows to Monitor

niri_msg(args=["action", "move-window-to-monitor-left"])
niri_msg(args=["action", "move-window-to-monitor-right"])
niri_msg(args=["action", "move-window-to-monitor-down"])
niri_msg(args=["action", "move-window-to-monitor-up"])
niri_msg(args=["action", "move-window-to-monitor-previous"])
niri_msg(args=["action", "move-window-to-monitor-next"])
niri_msg(args=["action", "move-window-to-monitor", "--id", "<name>"])
niri_msg(args=["action", "move-column-to-monitor-left"])
niri_msg(args=["action", "move-column-to-monitor-right"])
niri_msg(args=["action", "move-column-to-monitor-down"])
niri_msg(args=["action", "move-column-to-monitor-up"])
niri_msg(args=["action", "move-column-to-monitor-previous"])
niri_msg(args=["action", "move-column-to-monitor-next"])
niri_msg(args=["action", "move-column-to-monitor", "--id", "<name>"])
niri_msg(args=["action", "move-workspace-to-monitor-left"])
niri_msg(args=["action", "move-workspace-to-monitor-right"])
niri_msg(args=["action", "move-workspace-to-monitor-down"])
niri_msg(args=["action", "move-workspace-to-monitor-up"])
niri_msg(args=["action", "move-workspace-to-monitor-previous"])
niri_msg(args=["action", "move-workspace-to-monitor-next"])
niri_msg(args=["action", "move-workspace-to-monitor", "--id", "<name>"])

App Management

List Apps

agentbrowser_app_list()

Launch App

agentbrowser_app_launch(command="<binary>")

Kill App by PID

niri_msg(args=["action", "spawn", "--", "kill", "<pid>"])

Or use CUA:

agentbrowser_cua_call(tool="kill_app", input={"pid": <pid>})

Screenshots

Full Desktop

agentbrowser_screen_shot(path="/tmp/screenshot.png")

Focused Window

agentbrowser_screenshot_window(query="<app_id|title|window_id>")

Cropped Region

agentbrowser_cua_call(tool="zoom", input={"x1": 0, "y1": 0, "x2": 500, "y2": 500})

Niri Built-in

niri_msg(args=["action", "screenshot"])
niri_msg(args=["action", "screenshot-screen"])
niri_msg(args=["action", "screenshot-window"])

Mouse & Keyboard

Mouse Click

agentbrowser_mouse_click(x=<int>, y=<int>, button=<1|2|3>)

Mouse Move

agentbrowser_mouse_move(x=<int>, y=<int>)

Mouse Drag

agentbrowser_mouse_drag(x1=<int>, y1=<int>, x2=<int>, y2=<int>, button=<1|2|3>)

Mouse Scroll

agentbrowser_mouse_scroll(clicks=<int>, direction="up|down|left|right")

Key Press (combo)

agentbrowser_key_press(keys="ctrl+c")
agentbrowser_key_press(keys="alt+Tab")
agentbrowser_key_press(keys="Return")

Key Type (text)

agentbrowser_key_type(text="Hello world")

CUA Low-level

agentbrowser_cua_call(tool="press_key", input={"key": "Return"})
agentbrowser_cua_call(tool="hotkey", input={"keys": ["ctrl", "c"]})
agentbrowser_cua_call(tool="type_text", input={"text": "hello"})
agentbrowser_cua_call(tool="type_text_chars", input={"text": "hello", "delay_ms": 30})
agentbrowser_cua_call(tool="double_click", input={"x": 100, "y": 200})
agentbrowser_cua_call(tool="right_click", input={"x": 100, "y": 200})
agentbrowser_cua_call(tool="scroll", input={"pid": <pid>, "direction": "down"})

Clipboard

agentbrowser_clipboard_get()
agentbrowser_clipboard_set(text="copied text")

Workspaces (niri-specific helpers)

agentbrowser_workspace_goto(reference="1")
agentbrowser_workspace_next()
agentbrowser_workspace_prev()
agentbrowser_workspace_set_name(name="dev")

Window Helpers (agentbrowser wrappers)

agentbrowser_window_list()
agentbrowser_window_focus(query="<app_id|title>")
agentbrowser_window_minimize(query="<app_id|title>")
agentbrowser_window_maximize(query="<app_id|title>")
agentbrowser_window_restore(query="<app_id|title>")
agentbrowser_niri_click_in_window(window_query="<app_id>", x=<int>, y=<int>)
agentbrowser_niri_type_in_window(window_query="<app_id>", text="hello")

Power & System

niri_msg(args=["action", "power-off-monitors"])
niri_msg(args=["action", "power-on-monitors"])
niri_msg(args=["action", "quit"])
niri_msg(args=["action", "spawn", "--", "<command>", "<args>"])
niri_msg(args=["action", "spawn-sh", "--", "<shell command>"])
niri_msg(args=["action", "load-config-file"])
niri_msg(args=["action", "toggle-debug-tint"])

Screencast / Recording

niri_msg(args=["action", "set-dynamic-cast-window", "--id", "<window_id>"])
niri_msg(args=["action", "set-dynamic-cast-monitor", "--id", "<monitor_id>"])
niri_msg(args=["action", "clear-dynamic-cast-target"])
niri_msg(args=["action", "stop-cast", "--id", "<cast_id>"])
niri_msg(args=["action", "toggle-overview"])
niri_msg(args=["action", "open-overview"])
niri_msg(args=["action", "close-overview"])

CUA Advanced Tools

agentbrowser_cua_call(tool="get_desktop_state")        # Full desktop screenshot
agentbrowser_cua_call(tool="get_accessibility_tree")    # AT-SPI tree
agentbrowser_cua_call(tool="get_window_state", input={"pid": <pid>})  # Window AT-SPI
agentbrowser_cua_call(tool="get_cursor_position")       # Mouse position
agentbrowser_cua_call(tool="get_screen_size")           # Display size
agentbrowser_cua_call(tool="bring_to_front", input={"pid": <pid>})
agentbrowser_cua_call(tool="move_cursor", input={"x": 100, "y": 200})
agentbrowser_cua_call(tool="set_value", input={"element_index": <n>, "value": "text"})
agentbrowser_cua_call(tool="click", input={"pid": <pid>, "element_index": <n>})
agentbrowser_cua_call(tool="health_report")

Python CLI (agentbrowser-cli.py)

Located at skills/browser-automation/agentbrowser-cli.py. Standalone Python script for browser + desktop + Bitwarden automation.

Browser Commands

python3 agentbrowser-cli.py login <site>              # Auto-login with Bitwarden
python3 agentbrowser-cli.py login <site> -a "account" # Prefer specific account
python3 agentbrowser-cli.py login <site> --no-submit  # Fill only, don't submit
python3 agentbrowser-cli.py login <site> --no-totp    # Skip TOTP auto-fill

python3 agentbrowser-cli.py google "search query"    # Search Google
python3 agentbrowser-cli.py navigate <url>            # Go to URL
python3 agentbrowser-cli.py tabs                      # List browser tabs
python3 agentbrowser-cli.py fill "#selector" "value"  # Fill input field
python3 agentbrowser-cli.py click "button"            # Click element
python3 agentbrowser-cli.py eval "document.title"     # Run JavaScript
python3 agentbrowser-cli.py screenshot [output.png]   # Take screenshot
python3 agentbrowser-cli.py snapshot                  # Get page accessibility tree
python3 agentbrowser-cli.py markdown [url]            # Get page as markdown
python3 agentbrowser-cli.py close-blank               # Close all blank tabs

Bitwarden Commands

python3 agentbrowser-cli.py bw list                   # List vault items
python3 agentbrowser-cli.py bw list -s "github"       # Search vault
python3 agentbrowser-cli.py bw get "github"           # Get item details + TOTP
python3 agentbrowser-cli.py bw generate -l 20         # Generate password
python3 agentbrowser-cli.py bw status                 # Check vault status

Desktop Commands

python3 agentbrowser-cli.py desktop click 100 200     # Click at coordinates
python3 agentbrowser-cli.py desktop type "hello"      # Type text
python3 agentbrowser-cli.py desktop type "hello" -d 50 # Type with delay (ms)
python3 agentbrowser-cli.py desktop key "ctrl+c"      # Press key combo
python3 agentbrowser-cli.py desktop screenshot        # Desktop screenshot
python3 agentbrowser-cli.py desktop windows           # List windows
python3 agentbrowser-cli.py desktop focus "Firefox"   # Focus window by name

Environment Variables

CDP_URL=http://localhost:9222            # Chrome DevTools Protocol endpoint
MCP_URL=http://127.0.0.1:18787/mcp       # Go MCP endpoint (streamable-http)
CDP_CLI=cdp-cli                          # CDP CLI binary path
CONTAINER=agentbrowser-arch-browser      # Docker container name
BW_SESSION=...                      # Bitwarden session token (optional)

Python Scripts (scripts/)

# Accessibility tree dump (dogtail/AT-SPI)
python3 scripts/test_dogtail.py                       # Dump all apps
python3 scripts/test_dogtail.py --app firefox          # Filter by app
python3 scripts/test_dogtail.py --json                 # JSON output
python3 scripts/test_dogtail.py --max-depth 6          # Deeper tree

# Dolphin file manager (via MCP)
niri_msg(args=["action", "spawn", "--", "dolphin"])                    # Launch
niri_msg(args=["action", "spawn", "--", "dolphin", "/path/to/dir"])   # Open path
niri_msg(args=["action", "spawn", "--", "dolphin", "--new-window"])   # New window

# Focus and send shortcuts
niri_windows()  # → find org.kde.dolphin → id
niri_msg(args=["action", "focus-window", "--id", "<id>"])
agentbrowser_key_press(keys="ctrl+l")          # Location bar
agentbrowser_key_type(text="/home/user")       # Type path
agentbrowser_key_press(keys="Return")          # Navigate

# Close
niri_msg(args=["action", "close-window", "--id", "<id>"])

# Accessibility tree
python3 scripts/test_dogtail.py --app dolphin
python3 scripts/test_dogtail.py --app dolphin --json

# CDP browser tests
python3 scripts/test_cdp.py                            # Test CDP connection
python3 scripts/test_turnstile.py                      # Test Turnstile solver

System Commands

# Rebuild and restart the appliance after image or launcher changes
docker compose up -d --build

# Quick health check: the MCP server answers an initialize handshake
curl -s -X POST http://127.0.0.1:18787/mcp \
  -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"probe","version":"1"}}}'

# List desktop apps / windows (via the CLI, which speaks MCP)
python3 agentbrowser-cli.py desktop windows

# OpenCode: guided MCP setup
opencode mcp add

OpenCode Configuration

OpenCode supports MCP under the mcp key in opencode.json, and also provides opencode mcp add for guided setup.

Minimal remote MCP entry:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "agentbrowser": {
      "type": "remote",
      "url": "http://127.0.0.1:18787/mcp"
    }
  }
}

Resources

• Templates: See assets/ for OpenCode config snippets