| name | macos-cua |
| description | Use for macos-cua setup, troubleshooting, MCP/CLI operation, or desktop automation only when Senpi/pi-extension registered tools are unavailable. In Senpi registered-tool mode, do not load this skill just to operate the UI; use the tool guides on `get_app_state`, `click`, `type_text`, `press_keys`, `scroll`, `drag`, `set_value`, `select_text`, `perform_secondary_action`, and `zoom`. Triggers: macos-cua setup, macos-cua troubleshooting, macos-cua CLI, macos-cua MCP, macos computer use without registered tools, host-native computer use, ScreenCaptureKit, CGEvent, macOS GUI automation fallback. |
macos-cua
macos-cua is a TypeScript-native computer-use automation framework for macOS. It drives the mouse, keyboard, screen, and accessibility tree through native macOS APIs (CoreGraphics CGEvent, AXUIElement, screencapture/sips, and SkyLight/AppKit FFI) without requiring Python, a VM sandbox, or a helper binary.
When to reach for macos-cua
Use macos-cua whenever the user wants the agent to operate their actual macOS desktop as a human would. Specifically:
- Take a screenshot of the user's Mac desktop or a specific app window.
- Click, double-click, drag, type text, press key chords, or scroll at coordinates.
- Query the current cursor position or screen dimensions.
- Drive a macOS app through a sequence of UI actions.
- Run automation that needs low-latency, host-native execution (no Docker/QEMU/Lume overhead).
If the user only wants to read a file, run a normal CLI, write code, or operate UI through Senpi's registered get_app_state/click/type_text tools, this skill does not apply. Registered tools carry their own guides.
Operating modes
macos-cua has three entry points. Registered Senpi/pi-extension tools are self-guiding and do not require loading this skill. When this skill is already loaded for setup, troubleshooting, MCP, or CLI work, prefer the highest available mode in this order:
| Mode | How to invoke | When to pick it |
|---|
| pi-extension | pi install file://... then restart pi | Senpi has registered Codex-compatible list_apps, get_app_state, click, drag, scroll, type_text, press_keys, set_value, select_text, perform_secondary_action, and zoom tools. |
| MCP server | macos-cua-mcp over stdio | The host loads MCP servers automatically. The server exposes the same verbs as tools. |
| CLI | macos-cua <verb> directly in pi's bash | The discrete tools are unavailable, you are setting up/debugging the package, or the user explicitly asks for the CLI. |
All three modes share the same underlying MacOSHostComputer implementation.
Senpi/pi-extension fallback note
If this skill is already loaded while Senpi/pi-extension tools are visible, use get_app_state first for visible UI work, then act with the discrete tools. Use bash/CLI only when the discrete tools are unavailable, for setup, or when the user explicitly asks for the CLI.
- Start with
get_app_state for the target app. It returns a screenshot, numbered element_index boxes, the accessibility tree, and remembers the target window.
- Prefer
click element_index=<number>, set_value, select_text, or perform_secondary_action when the accessibility tree exposes the target. Use zoom for small or ambiguous targets before clicking.
- Use
type_text, press_keys, scroll, or drag for direct input. Keep using the same app name or bundle id.
- Coordinates are screenshot pixels in the latest screenshot frame. Use screenshot metadata when present; after any coordinate error, get a fresh screenshot with
get_app_state before retrying. Do not guess outside the visible frame.
- Call
get_app_state again only after a UI-changing action when you need to verify or choose the next target.
When the @macos-cua/pi-extension is loaded, Anthropic Messages and direct OpenAI Responses models automatically receive native computer-use. Anthropic gets the computer-use-2025-01-24 beta tool plus header/body fields; direct OpenAI Responses gets { type: "computer" } in the provider payload. OpenAI Chat/proxy models, including Kimi in Senpi, use the discrete tools instead. All flows keep the Codex-compatible app tools available and all native-computer flows are disabled only by MACOS_CUA_DISABLE_COMPUTER_USE_BETA=1 (true, yes, and on also work).
The extension uses adaptive screenshot fidelity for model-facing images. Provider hard limits still apply, byte-budget downgrades are surfaced through screenshot metadata, and coordinates returned by the model are interpreted in that latest screenshot frame before dispatch.
Deferred from gajae learnings: precise two-axis scroll fields and broad schema/casing refactors are intentionally not shipped here. Keep the semantic page/element scroll API and existing public argument shapes unless a separate plan changes them.
macOS permissions
macOS needs host consent before macos-cua can see or control UI:
- Screen Recording for screenshots and window discovery.
- Accessibility for click, type, key, scroll, drag, and AX actions.
- Apple Events when resolving
--target-bundle-id through System Events.
Full installation walkthrough: references/installation.md.
App targeting
When you want the agent to drive a specific app, call get_app_state first in MCP/pi-extension mode, or pass --target-pid/--target-bundle-id in CLI mode. Targeted input uses a remembered visible app window and routes through CoreGraphics plus SkyLight/AppKit FFI. If no target window is known, targeted input fails loudly rather than falling back to global cursor-moving input.
CLI surface — macos-cua <verb>
When using CLI mode, desktop actions go through macos-cua <verb>. The verb taxonomy:
| Action | Command shape |
|---|
| Screenshot | macos-cua screenshot -o /tmp/macos-cua-<unix-ts>.png |
| Click | macos-cua click <x> <y> (or --button right) |
| Double-click | macos-cua double-click <x> <y> |
| Move cursor | macos-cua move <x> <y> |
| Drag | macos-cua drag <fromX> <fromY> <toX> <toY> |
| Type text | macos-cua type "<text>" |
| Key chord | macos-cua key <key> -m <modifiers> (e.g. cmd,shift) |
| Keypress sequence | macos-cua keypress cmd shift t |
| Scroll | macos-cua scroll -d <up|down|left|right> -a <amount> |
| Wait | macos-cua wait <ms> |
| Cursor position | macos-cua cursor |
| Screen size | macos-cua screen |
| Check permission | macos-cua permissions check <screen|accessibility|input-monitoring|apple-events> |
| Request permission | macos-cua permissions request <kind> |
| Active window | macos-cua --json windows active |
| List windows | macos-cua --json windows list (requires Screen Recording; fails fast after 5s if missing) |
Add --json to any subcommand for machine-readable output. If a call fails with an unknown flag, run macos-cua <verb> --help to see the exact flag set for that verb.
CLI screenshot → Read pattern
In CLI mode, macos-cua writes the PNG to disk; pi's Read tool ingests it as inline image content.
TS=$(date +%s%N)
SHOT="/tmp/macos-cua-${TS}.png"
macos-cua screenshot -o "${SHOT}"
Then in the same agent turn, call pi's Read tool with the absolute path /tmp/macos-cua-<ts>.png. The PNG is attached to the next assistant message as inline image content. Do not base64 it manually, do not pipe it through stdin. Read already does the right thing.
Shell on localhost — prefer pi's bash
On localhost, prefer pi's built-in bash tool for shell commands. It has the same shell environment and you get pi's standard output capture. If pi-extension computer-use tools are visible, use them for desktop actions; otherwise use macos-cua only for computer-use actions (screenshot, click, type, etc.).
Reference files — load only what you need
Critical conventions
Never auto-drive a destructive GUI
Before executing any irreversible UI action (deleting files in Finder, confirming a system dialog, submitting a form, purchasing something), pause and ask the user for explicit confirmation. The agent can see the screen, but the user is the one who bears the consequences.
Coordinate spaces
MacOSHostComputer input uses macOS logical points. The pi-extension native computer tool shows models adaptive screenshot images within provider limits and unscales model-space coordinates back to logical points before input dispatch. Raw CLI screenshots may still be Retina physical PNGs, so convert screenshot pixels to logical points when driving the CLI directly.
Permission troubleshooting
Only check permissions after a black screenshot, missing window, or ignored input. Permission is per-binary, so switching terminal apps may require re-granting. In CLI mode, verify with:
macos-cua permissions check screen
macos-cua permissions check accessibility
macos-cua permissions check input-monitoring
Each should print authorized. If any prints denied or not-determined, run macos-cua permissions request <kind> to trigger the macOS system dialog, then re-check. Falling back to System Settings → Privacy & Security works too.
