cua-driver

# Qwen Cua Driver Orchestrates cross-platform app automation via `qwen-cua-driver`. Whenever a user asks to drive a native app, follow the loop in this skill rather than calling tools ad-hoc — the snapshot-before-action invariant is not optional and silently breaks if you skip it. ## Platform-specific reading — read this first This file is the **cross-platform core**: snapshot invariant, CLI vs MCP choice, tool surface naming, behavior matrix, canonical loop, pixel-click contract, common failure modes. The platform-specific material (forbidden-list, accessibility tree implementation, launch semantics, click dispatch) lives in companion files in this same directory: - **macOS** — read `MACOS.md` (no-foreground contract, forbidden `open`/`osascript`/`cliclick` invocations, AXMenuBar navigation, SkyLight pixel-click dispatch). - **Windows** — read `WINDOWS.md` (UIA tree vs AX, UWP / ApplicationFrameHost hosting, layered UIA+PostMessage click chain, Session 0 isolation, Windows-specific focus-steal vectors). - **Linux** — read `LINUX.md` (X11 background input via AT-SPI + XSendEvent and compositor-specific Wayland capabilities). Cross-cutting topics also have their own files: - `BROWSER.md` — exact native-window binding, explicit browser preparation, typed Chromium/Electron page tools, input trust classes, and native fallbacks for browser chrome and unsupported engines. - `RECORDING.md` — session recording + `replay_trajectory`. Use whichever combination matches the host. When in doubt, run `qwen-cua-driver doctor` — it reports the platform and the right entry point. ## Start with the narrowest semantic route Before opening or operating an application, name the desired postcondition and use the first applicable route below. Verify the result in the same domain before stopping or advancing: 0. **Caller-provided headless/background operation for a non-GUI outcome.** Prefer an exact application API/SDK, service or database client, CLI, or filesystem operation over imitating a user. This includes batch-safe file moves, renames, copies, directory creation, archive extraction, data conversion, and process inspection. Read the resulting semantic state back; a zero exit status alone is not proof. 1. **Typed Cua operation for an application or window outcome.** Use `set_window_frame` for exact geometry, `invoke_menu` for a known native application-menu path, typed browser tools for supported page content, and clipboard tools for clipboard state. Verify with `list_windows`, `get_browser_state`, or `clipboard_read`, respectively. 2. **Background accessibility action.** Use a fresh AX/UIA/AT-SPI target. 3. **Background pixel action.** Use the pixels from the same state snapshot. 4. **Foreground delivery.** Retry only the action that evidence says could not land in the background. 5. **Desktop fallback.** Enter this explicit, one-way session phase last. Use Cua Driver when the outcome lives in an application's UI or window state, or when the user explicitly asks to operate that GUI. Once the task crosses that boundary, do not replace Cua's targeted and verified actions with shell scripts that mutate the app UI. A shell is a capability of the calling agent, not of the Cua Driver MCP server; an MCP-only client must not assume one exists. ### Filesystem outcomes and GUI fallbacks When the requested outcome is a filesystem change and the caller has a headless filesystem or command capability, keep it on rung 0. Enumerate the exact source set, decide the destination-conflict policy before changing anything, perform one batch-safe operation, then independently read back both source and destination manifests. Do not open a file manager merely to mimic a move, copy, or rename that the caller can execute and verify directly. If the caller has no such capability, use the file manager as a GUI fallback and keep each claim narrow: 1. After entering an inline rename and setting its value, commit it with the platform's confirmation key, then take a fresh snapshot. Value readback from the inline editor proves only that the editor changed; it does not prove the filesystem rename committed. 2. For a multi-selection, use the platform modifier (`cmd` on macOS, `ctrl` on Windows/Linux). On macOS and Windows, issue that modified click with `delivery_mode:"foreground"` so the target observes physical modifier state; a refused background attempt is an escalation signal, not a failed action to trust or repeat. Re-snapshot before the next operation. Continue only when every intended item is selected and the prior selection was preserved. 3. After a cross-window drag or paste, verify the destination contains the complete expected set and the source reflects copy-versus-move semantics. A delivered drag, keypress, or menu action is not file-operation proof. 4. If a destination conflict presents an unrecognized policy or ambiguous partial result, stop that GUI path and surface the unresolved state instead of retrying blindly. ## The no-foreground principle (window phase) In a strict `window` session, and during the initial window phase of an `auto` session, **the user's frontmost app MUST NOT change.** Every platform has its own list of forbidden commands: - macOS: any `open` invocation, any `osascript` that mutates GUI state, `cliclick`, `cghidEventTap` writes targeting another app's window. Full list in `MACOS.md`. - Windows: any `Start-Process` that triggers a `ShowWindow`/`SetForegroundWindow` on the target, `WScript.Shell.AppActivate`, attaching to the foreground thread for input forwarding. Full list in `WINDOWS.md`. If you reach for a command that says "activate", "foreground", "raise", or "make key", stop and translate to the cua-driver tool that does the same intent without focus-stealing. A strict `desktop` session is an explicit user choice to operate the visible desktop and therefore uses foreground/system input. An `auto` session may enter that phase only after the complete window ladder below has been attempted and verified, followed by `escalate_session`. Never infer desktop permission from a failed action or a proxy/transport session id. ## GUI transport defaults — prefer cua-driver over GUI shell shims **Default transport is the `qwen-cua-driver` CLI** — `Bash` shelling out to `qwen-cua-driver <tool-name> '<JSON-args>'`. MCP tools (prefix `mcp__cua-driver__*`) only when the user explicitly asks for them. CLI wins because it picks up rebuilds instantly, failures are easier to diagnose, and there's no per-tool schema-load overhead. Every reference to `click(...)`, `get_window_state(...)` etc. in this skill means `qwen-cua-driver click '{...}'` — translate to MCP form only when MCP is requested. ### Claude Code computer-use compatibility mode For normal Claude Code use, keep the default CLI or `qwen-cua-driver` MCP server path above. If the user explicitly wants Claude Code's vision/computer-use-style flow, they can register: ```bash qwen-cua-driver mcp-config --client claude # then paste + run the printed line ``` Observation: Claude Code vision flows appear to treat a screenshot MCP tool as the image-grounding anchor. This compatibility mode keeps the normal CuaDriver tools and changes only `screenshot`. The compatibility `screenshot` requires `pid` and `window_id`, captures only that target window, and returns the window-local pixel coordinate frame. Start with `launch_app` or `list_windows`, then call `screenshot({pid, window_id})`; do not assume desktop coordinates or a full-screen capture. Use MCP for this Claude Code vision/computer-use-style path. Do not shell out to `qwen-cua-driver screenshot` as a substitute: CLI screenshots still work as CuaDriver calls, but they do not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue. ## Using cua-driver from the shell Tool names are `snake_case`, management subcommands are `kebab-case` — no ambiguity. Tools invoked as `qwen-cua-driver <tool-name> '<JSON-args>'`. Management subcommands: - `qwen-cua-driver serve` — start an explicit persistent service when short-lived clients must share runtime state or a platform identity. Bare MCP owns its runtime directly on Windows/Linux and uses the signed app service on macOS; `qwen-cua-driver mcp --socket <endpoint>` selects a service explicitly. One-shot CLI tool calls still use the service path. macOS users: see `MACOS.md` for the LaunchServices-routed launch form. - `qwen-cua-driver stop` / `status` - `qwen-cua-driver list-tools`, `describe <tool>` - `qwen-cua-driver recording start|stop|status` — see `RECORDING.md` - `qwen-cua-driver check-update [--json] [--no-cache]` — read-only "is a newer release available?" probe. Same payload as the `check_for_update` MCP tool; pair with `qwen-cua-driver update --apply` to install. Canonical multi-step workflow (example shape — platform-specific launch idioms in the per-OS companion file): ```bash qwen-cua-driver serve qwen-cua-driver launch_app '{"bundle_id":"..."}' # → {pid: 844, windows: [{window_id: 10725, ...}]} qwen-cua-driver get_window_state '{"pid":844,"window_id":10725}' # Use the returned structuredContent.elements[].element_token: qwen-cua-driver click '{"pid":844,"element_token":"s0000002a:14"}' qwen-cua-driver verify_state '{"pid":844,"window_id":10725,"expect":[{"element":{"selector":{"label_contains":"Saved"},"exists":true}}]}' qwen-cua-driver stop ``` For Chromium page content, keep the same native window selection but switch to the browser capability loop: `start_session`, bind `(pid, window_id)` with `get_browser_state`, snapshot the returned tab, then use `browser_click`, `browser_type`, or `browser_navigate`. Read `BROWSER.md` before using this route. Browser target ids, tab ids, and refs are session-scoped and stale refs must be replaced by a fresh snapshot. ## Agent cursor overlay Visual cursor overlay for demos and screen recordings. It is enabled by default for declared sessions; anonymous actions remain cursor-less. Toggle with `set_agent_cursor_enabled` to hide or re-show it. The embedded `cua.default` theme uses a session-colored pointer over a larger, cursor-shaped glow in the same session color. The glow fades to transparent around the full silhouette. Action marks use the same session-colored center and white-outline treatment, plus a tighter, softer glow. This pairing preserves contrast across varied backgrounds. It provides animations for idle, observe, click, drag, scroll, text, key, navigation, app, transfer, recording, and system activity. Motion knobs: `set_agent_cursor_motion` takes any subset of `start_handle`, `end_handle`, `arc_size`, `arc_flow`, `spring` — tuneable at runtime, persisted to config. Delivery and target context is shown as host-owned chips inside the session badge. Themes own the twelve action animations only. The session name and context chips fade independently, so an active tool can show its execution context without revealing a session name that has already faded. **Per-session cursors.** Each MCP session automatically owns its own cursor, keyed by the session's id (the proxy mints one session id per MCP connection and the daemon scopes the cursor, config overrides, and recording to it). The CLI and SDK contracts take the declared `session` explicitly. Cursor-theme controls no longer accept `cursor_id` or the legacy shape/color/image fields. Input-delivery tools may still use `cursor_id` to name a virtual pointer; it never selects artwork. The default cursor is Cua blue, while each named session receives a stable fill from the built-in palette. Select only preinstalled themes with `set_agent_cursor_theme`; theme source paths and inline animation data are never accepted through an agent tool. Use the trusted local `qwen-cua-driver cursor-theme` workflow to validate, compile, preview, install, list, or remove custom themes. **Visibility caveat (AX runs).** On a pure accessibility-action run (clicking by `element_index`), the first action **seeds the cursor on-screen a short distance from the target and plays a brief glide + pulse** — not the long Bezier sweep a cursor already on-screen would trace from its previous spot. It's subtle and easy to miss in a recording. If you want a clearly _gliding_ cursor for a demo or screen recording, do a pixel click (`click({pid,x,y})`) or a `move_cursor` first to put the cursor on-screen; subsequent AX actions then glide the full path normally. Requires a suitable UI event loop. Service and private-worker runtimes provide one. On macOS, a same-process SDK runtime or `qwen-cua-driver mcp --direct` without a certified host main-thread adapter returns a structured `facility_unavailable` result for overlay operations; do not treat that as a successful cursor move. One-shot CLI adapters do not own an overlay themselves. ## The core invariant — snapshot before and verify after every action **Every action MUST be bracketed by observation for the session's effective scope.** Use `get_window_state(pid, window_id)` before a window action (or `get_desktop_state(session)` in desktop scope), then use `verify_state` for an expressible window-scoped postcondition. In effective desktop scope, `verify_state` is intentionally refused with `window_scope_disabled`; verify with a fresh `get_desktop_state` result and agent-owned visual/semantic reading. - **Before** — the pre-action snapshot resolves the `element_index` you're about to use. Indices from previous turns are stale; the server replaces the element index map on every snapshot, keyed on `(pid, window_id)`. Indices from turn N don't resolve in turn N+1, and indices from window A don't resolve against window B of the same app. Skip this and element-indexed actions fail with `No cached AX state`. - **After** — `verify_state(pid, window_id, expect)` checks a bounded, deterministic postcondition. Results are `satisfied`, `unsatisfied`, or `unknown`; `unknown` never means success. Set `include_screenshot:true` when the outcome also needs visual reading. The driver returns that final image without interpreting it. A multimodal agent harness reads the image and owns the stop/retry/ladder decision. `unknown_reason` distinguishes invalid/unsupported predicates, untrusted web content, ambiguous matches, missing targets, unavailable observations, and `stability_unproven`. A positive final sample that was not observed for the requested consecutive sample count is `stability_unproven`, not success. Negative element existence is conservative: when an accessibility projection cannot prove its search domain exhaustive, absence remains `unknown`. Do not make the driver invent task meaning or retry actions automatically. For postconditions not expressible by `verify_state`, take a fresh state snapshot and let the agent judge the tree and/or image explicitly. This applies to pixel clicks and desktop actions too. ### Read action facts without confusing them with task success A successful action returns `effect` and `route`, with optional typed `delivery`, `evidence`, and `escalation`. These fields describe the actuator; they do not declare the user's task complete. - `confirmed` means the driver has publishable value readback or window-change evidence for that action. - `partial` means only `delivery.delivered_count` was delivered. - `unverifiable` means the driver cannot prove the effect. - `suspected_noop` means available evidence suggests no useful change. - `refused` means the selected route deliberately did not deliver. The route vocabulary is intentionally cross-platform: `accessibility`, `synthetic_events`, `global_input`, `dom`, and `trusted_input`. Do not branch on private OS transport names. An optional escalation is a harness instruction, never an automatic retry: - `pixel`: refresh visual state and choose an exact pixel target; - `foreground`: explicitly select foreground delivery if session policy allows; - `page`: bind the native window to a supported browser page route; - `session`: prepare or explicitly widen the session only when policy permits. Branch on the closed reason vocabulary: `route_unavailable`, `delivery_failed`, `effect_unconfirmed`, `suspected_noop`, and `permission_required`. After any action, keep using `verify_state` or a fresh state snapshot for the actual task postcondition. The multimodal harness owns visual reading and the decision to stop, retry, or advance the ladder. ## Choose capture scope when the session starts `capture_scope` is a per-session policy, not persistent configuration. Declare it with `start_session`; it is immutable until that session ends. Concurrent sessions may choose different policies safely. - `auto` (default): begins with effective scope `window`. Desktop perception and actions are locked until the window ladder is exhausted, each attempted action is verified, and the caller explicitly invokes `escalate_session`. Escalation is one-way for the live session. - `window`: strict window-only perception and actions. Desktop tools are always rejected with `desktop_scope_disabled`. - `desktop`: strict full-desktop perception and foreground/system actions. Window-scoped perception and actions are rejected with `window_scope_disabled`. ```bash qwen-cua-driver start_session '{"session":"research-1","capture_scope":"auto"}' qwen-cua-driver get_session_state '{"session":"research-1"}' ``` Do not use `config set capture_scope` or `set_config`; that key is retired and stale values on disk are ignored. Always pass the public `session` field on state and action calls. Reserved fields such as `_session_id` are transport metadata and cannot create or change policy. During a mixed-version rollout, require `tools/list` to advertise `session.capture_scope` (and `session.capture_scope.escalate` for `auto`). If an older daemon does not advertise them, fail closed and ask for an upgrade; never fall back to the retired global config key. ### Why window selection is the caller's job now `get_app_state` used to pick a window for you via a max-area heuristic that returned the wrong surface on apps with large off-screen utility panels. Concrete reproducer: IINA's OpenSubtitles helper (600×432 off-screen) out-area'd the visible 320×240 player window, so `get_app_state(pid)` screenshot'd the invisible panel and clicks landed there silently. The new `get_window_state(pid, window_id)` makes the caller name the window explicitly — the driver validates that the window belongs to the pid and is on the current Space/desktop, then snapshots exactly what was asked for. Enumerate candidates via `list_windows` or read the `windows` array `launch_app` already returns. ## Behavior matrix ### Perception is mode-agnostic — `get_window_state` returns BOTH `get_window_state(pid, window_id)` **returns both the accessibility tree AND a screenshot by default.** There is no capture mode to pick and nothing to configure — you ground on the tree and the screenshot together, and you cross-check one against the other. This matters because the tree **lies** on some surfaces: - **Electron** echo-confirms a `set_value` / `type_text` against the AX shim while the rendered text view never changed. - **Catalyst** (iOSAppOnMac) exposes null / placeholder `AXValue`s. - **Virtualized / off-viewport list rows** report bogus frames (an `h:1` height, an off-screen origin) for rows that aren't actually laid out. A grounding screenshot is present by default, so when the tree looks wrong you look at the pixels **in the same response** — no second capture, no mode flip. > **Perf opt-out — `include_screenshot`.** `include_screenshot` > (boolean, default `true`) is the one knob, and it is a **perf** knob, > not a modality choice. Default returns both (grounding-first). Pass > `include_screenshot:false` to skip the screen grab and get the tree > only — the cheap path when you're just **re-indexing before an > element ax action** and don't need to re-ground on pixels. The > `ax`/`px` decision still lives at action time, not here. > **`capture_mode` is DEPRECATED and ignored.** It is still _accepted_ > on `get_window_state` so old callers don't error, but it has **no > effect** — both the tree and the screenshot come back regardless of > what you pass (`ax`, `vision`, `som`, anything). There is no