cua-driver

name

cua-driver

description

Drive native macOS apps in the background via cua-driver — snapshot AX trees, click/type/scroll by element_index, verify via re-snapshot. Use when the user asks to operate, automate, or perform a GUI task in a real macOS application without stealing focus or cursor (e.g. "open a file in Finder", "fill a form in Safari", "click Save in Numbers"). Background-first — the user keeps typing in their editor while the agent drives another app.

cua-driver skill

Orchestrates macOS app automation via the cua-driver tools. When a user asks to drive a native macOS app, follow this skill's loop rather than calling tools ad-hoc — the snapshot-before-action invariant is not optional and silently breaks if you skip it.

The no-foreground contract

The user's frontmost app MUST NOT change. This is the whole reason cua-driver exists. Users keep typing in their editor while an agent drives another app in the background.

Before running any shell command, ask: "does this raise, activate, foreground, or make-key any app?" If yes, don't run it.

Forbidden commands (every one activates the target):

• Every form of open — open -a, open -b, open <file>, open <url> — always activates through LaunchServices.
• osascript -e 'tell application "X" to activate'
• cliclick — warps the real cursor.
• CGEventPost with cghidEventTap targeting coordinates over another app's window.
• Focus-stealing shortcuts — ⌘L in browsers (activates omnibox), ⌘⇧G in Finder.
• Tab-switching shortcuts — ⌘1-9, ⌘], ⌘[ visibly flip tabs even on backgrounded apps.

The only exception: osascript activate when the user explicitly asked for frontmost state ("bring Chrome to the front"). Always name the focus steal in your response.

Prefer windows over tabs in browsers. Use cua_launch_app({bundle_id, urls: [url]}) — each URL opens in a new window with its own AX tree, all backgrounded.

Defaults — prefer cua-driver tools over shell shims

Intent	Use	Don't use
Open/launch an app	cua_launch_app({bundle_id})	open -a, osascript 'tell app … to activate'
Find a pid	cua_launch_app (returns pid) or cua_list_apps	pgrep, ps
Enumerate windows	cua_list_windows({pid}) or cua_launch_app response	osascript 'every window'
Click / type / scroll	cua_click, cua_type_text, cua_scroll	osascript, cliclick
Screenshot	cua_get_window_state or cua_screenshot	screencapture
Quit an app	Ask user first, then cua_hotkey({pid, keys: ["cmd","q"]})	kill, killall
Hand a file/URL to an app	cua_launch_app({bundle_id, urls:[path]})	open -a <App> <path>

Prerequisites — check before starting

1. cua-driver on $PATH (which cua-driver). If not found, point the user to the install script.
2. Run /cua-status to check daemon and permissions. If either grant is missing, tell the user to grant Accessibility and Screen Recording in System Settings → Privacy & Security.

The core invariant — snapshot before AND after every action

Every action MUST be bracketed by cua_get_window_state(pid, window_id):

• Before — the pre-action snapshot resolves the element_index you're about to use. Indices from previous turns are stale. The server replaces the element index map on every snapshot, keyed on (pid, window_id). Skip this and element-indexed actions fail.
• After — the post-action snapshot verifies the action landed. If nothing changed in the AX tree, the action probably failed silently — say so, don't assume success.

Capture modes

capture_mode	cua_get_window_state returns	Use for actions
som (default)	tree + screenshot	element_index preferred; pixel fallback
ax	tree only (no PNG)	element_index only
vision	PNG only (no tree)	pixel only

Switch with cua_set_config({key: "capture_mode", value: "som"}).

The canonical loop

cua_launch_app({bundle_id})
  → pick window_id from the returned windows array
  → cua_get_window_state({pid, window_id})
    → [act]  # every action also takes pid + window_id
  → cua_get_window_state({pid, window_id}) → verify

1. Resolve target — always via cua_launch_app

Always start with cua_launch_app, whether or not the app is already running. It's idempotent and gives you the pid + windows in one call.

• cua_launch_app({bundle_id: "com.apple.finder"}) — preferred
• cua_launch_app({name: "Calculator"}) — fallback

2. Snapshot and act by element_index

Call cua_get_window_state({pid, window_id}). In som mode (default) the response carries:

• tree_markdown — every actionable element tagged [N]. That N is the element_index.
• screenshot — window screenshot for visual context.

Reason over both the tree AND the screenshot — they're complementary:

• The tree tells you what's clickable (roles, labels, element_index).
• The screenshot tells you which one (visual context disambiguates similar labels).

3. Dispatch table

Intent	Tool
Left click	cua_click({pid, window_id, element_index})
Double-click	cua_double_click({pid, window_id, element_index})
Right click	cua_right_click({pid, window_id, element_index})
Type text	cua_type_text({pid, text, window_id, element_index})
Type Unicode	cua_type_text_chars({pid, text}) — for Chromium/Electron
Set value	cua_set_value({pid, window_id, element_index, value})
Scroll	cua_scroll({pid, direction, amount})
Press key	cua_press_key({pid, key, modifiers})
Send shortcut	cua_hotkey({pid, keys: ["cmd", "c"]})
Pixel click	cua_click({pid, x, y}) — canvas/WebView fallback

4. Re-snapshot and verify — mandatory

Always call cua_get_window_state after the action. Check the AX tree diff. If nothing changed, the action likely failed silently — tell the user.

Pixel-coordinate clicks

For surfaces the AX tree doesn't reach (canvas, video, WebGL, custom-drawn controls):

1. cua_get_window_state({pid, window_id}) to get the screenshot
2. Pick the target pixel from the screenshot (window-local, top-left origin, y-down)
3. cua_click({pid, x, y}) — optionally pass window_id to anchor the coordinate conversion

Pixel coordinates match the screenshot at default max_image_dimension (1568px long-side) — no scaling math needed.

Menu bars — only when frontmost

Only drive the menu bar (AXMenuBarItem) when the target app is already frontmost. When backgrounded, use in-window element_index or pixel clicks instead. Two reasons:

1. Menu items go DISABLED when their app isn't key window — silent no-ops.
2. macOS renders the menu bar for the frontmost app — visually confusing.

Menu flow (frontmost only):

1. Find [N] AXMenuBarItem "Menu Name" in tree
2. cua_click({pid, element_index: N, action: "pick"}) — menu items use AXPick
3. Re-snapshot — expanded items now have element_index values
4. Click the target item

Web apps (Chrome, Electron, Safari, etc.)

Chromium/Electron apps need special handling:

• Sparse AX tree? Retry cua_get_window_state once — first call triggers AX population.
• Navigate to URL: Use cua_launch_app({bundle_id, urls: [url]}) — fully backgrounded.
• Don't ⌘L — it steals focus even on backgrounded pids. Use cua_set_value on the omnibox element instead.
• Pixel right-click on Chromium coerces to left — use element_index for AX-addressable targets.
• type_text silently drops on some web inputs — use cua_type_text_chars instead.
• Minimized windows: Return/Space keyboard commits silently no-op. Use cua_set_value or click a submit button instead.

Common errors

Error	Fix
No cached AX state for pid X window_id W	Call cua_get_window_state with same (pid, window_id) first
Invalid element_index N	Re-snapshot — indices are stale
window_id W belongs to pid P, not …	Use cua_list_windows to get correct windows
AX action AXPress failed	Try show_menu, confirm, cancel, or pick
System beep on cua_press_key	Window minimized — use cua_set_value instead
Permission errors	Run /cua-status and grant in System Settings

Things to avoid

• Never reuse an element_index across re-snapshots.
• Never translate screenshot pixels into a click when element_index is available.
• Prefer AX over pixels. Exhaust AX paths before dropping to coordinates.
• Never drive destructive actions without explicit user intent.
• Never launch apps autonomously without user confirmation.