| name | cua-skill |
| description | MUST USE whenever the user wants to automate a real desktop or sandbox - clicking, typing, scrolling, screenshotting, running an OS shell command, or handing a high-level 'open browser and do X' task to an autonomous computer-use agent. Wraps the trycua/cua Python toolkit via its `cua` CLI - pynput-based, cross-platform (macOS / Linux / Windows). NO custom tools are registered; you call `cua` through pi's built-in bash and read screenshots back through the Read tool. Triggers: cua, computer use, computer-use, GUI automation, screenshot the desktop, click on the screen, type into the active app, scroll the page, control my computer, drive my browser, sandbox, docker sandbox, QEMU sandbox, Lume sandbox, ComputerAgent, cua do, cua sandbox, 컴퓨터 자동화, 스크린샷 찍어, 내 컴퓨터 조작, 브라우저 열어서, 샌드박스, 화면 자동화, 마우스로 클릭, 키보드 타이핑, computer use 위임, 자동으로 클릭, 자율 에이전트로 처리. |
cua-skill
cua (from trycua/cua) is a Python-native computer-use automation framework. The cua CLI is a single binary that drives the mouse, keyboard, screen, shell, and an autonomous ComputerAgent loop. This skill teaches you how to invoke it directly from pi's built-in bash - no extension tools are registered, so every action is shell-mediated.
When to reach for cua
Use cua whenever the user wants the agent to operate a GUI as a human would, or to delegate a multi-step "do this in the browser / in this app" task. Specifically:
- Take a screenshot of the user's desktop (or a sandbox display).
- Click, double-click, drag, type text, press key chords, scroll at coordinates.
- Run a shell command on the host or inside a sandbox.
- Spin up an isolated Docker / QEMU / Lume / Tart sandbox to run untrusted automation.
- Hand a natural-language goal ("open Firefox, search for X, click the first result") to cua's
ComputerAgent and wait for the trajectory.
If the user only wants to read a file, run a normal CLI, or write code, this skill does not apply - use pi's existing read/bash/edit tools.
Operating modes
cua has three execution targets. Decide which one applies before issuing commands; the surface is mostly identical but the safety implications are not.
| Mode | Where actions actually run | Sandboxed? | When to pick it |
|---|
| localhost (default) | the user's real machine | no | The user explicitly wants their actual desktop driven. macOS, Linux, and Windows all work via pynput. |
| sandbox | a Docker container or VM (QEMU / Lume / Tart) | yes | The user wants risky automation isolated, or wants a clean Linux/Windows display. Requires cua sandbox start first. |
| cloud | a cua.ai-hosted VM | yes | The user has CUA_API_KEY and wants ephemeral cloud execution. |
If the user did not specify and the action is destructive (rm-anything, writing to system locations, irreversible UI clicks), pause and confirm: "Run on your real machine, or in a fresh sandbox?"
First-time host consent
Localhost mode needs OS-level input + screen-recording permission. The very first time cua touches the host, run:
cua do-host-consent
This is a one-shot grant. macOS will prompt for Accessibility + Screen Recording permission on the controlling terminal/IDE. After approval, subsequent cua do … calls work silently. If cua do click silently does nothing or cua do screenshot produces a black image, the user hasn't granted permission yet.
Full installation walkthrough: references/installation.md.
Core surface — cua do <verb>
Every desktop action goes through cua do. The verb taxonomy:
| Action | Command shape |
|---|
| Click | cua do click <x> <y> |
| Type text | cua do type "<text>" |
| Key chord | cua do key <chord> (e.g. cmd+s, Return, ctrl+shift+t) |
| Scroll | cua do scroll <x> <y> --dy <amount> |
| Screenshot | cua do screenshot --output /tmp/cua-<unix-ts>.png |
| Shell exec | cua do shell "<command>" (use pi's own bash on localhost — see note below) |
| Agent task | cua do task "<natural-language goal>" [--model <id>] [--max-turns <N>] |
Flag names sometimes drift across cua minor versions; if the first call fails with an argparse error, fall back to cua do <verb> --help immediately, capture the actual flag set, and continue. Don't guess.
Concrete recipes for each verb: references/localhost-usage.md.
The screenshot → Read pattern
cua writes screenshots to disk; you ingest them into context by calling pi's Read tool on that path. Do NOT try to base64-encode the file yourself - Read already attaches PNG/JPEG payloads as image content blocks.
TS=$(date +%s%N)
cua do screenshot --output "/tmp/cua-${TS}.png"
This is the entire reason cua is worth a skill rather than a tool - the Read tool already does image-inline injection, so we get the same vision capability with zero custom plumbing.
Shell on localhost - prefer pi's bash
On localhost mode, cua do shell "ls" and pi's built-in bash run in the same shell environment. Use pi's bash directly unless you specifically need cua's per-call timeout semantics or the same syntax to also work when the user switches to a sandbox. The cua do shell form really earns its place only on sandbox / cloud modes, where it routes the command into the isolated environment.
Reference files - load only what you need
Each file in references/ is independently consumable. Read only the one that matches the current request to keep context lean.
| File | Read when |
|---|
references/installation.md | Setting up cua for the first time, fixing a missing-dependency error, configuring the venv that pi will exec, or diagnosing PEP 668 issues on Homebrew Python. |
references/localhost-usage.md | Driving the user's real machine — mouse/keyboard/screenshot/shell recipes with exact flag forms. |
references/sandbox-usage.md | Starting / stopping Docker / QEMU / Lume / Tart sandboxes, targeting commands at a specific sandbox, capturing sandbox-side screenshots. |
references/computer-agent.md | Delegating a multi-step goal to ComputerAgent via cua do task, picking a model, reading trajectory output, capping cost with --max-turns. |
references/troubleshooting.md | Common errors — ModuleNotFoundError: No module named 'agent', permission-denied on macOS, pynput backend issues on Wayland, runaway clicks, etc. |
Critical conventions
Some defaults are non-obvious and have severe consequences when violated, so they're worth stating in the body rather than buried in a reference file.
Never auto-drive a destructive GUI
Mouse and keyboard on localhost is privileged: a single misclick can confirm a destructive dialog, send a message, or fill a payment form. If the user said "automate something" but did not specify the exact target, stop and confirm before driving the real GUI. A short screenshot-and-confirm exchange is much cheaper than an unrecoverable click.
Logical points, not physical pixels
cua do click 100 100 operates in logical points, the same coordinate system you see in macOS System Settings or in browser devtools. cua_auto already reads NSScreen.backingScaleFactor() (macOS), GDK_SCALE / QT_SCALE_FACTOR (Linux), and the Windows DPI scaling factor, and converts internally. Do not pre-multiply by 2 for Retina displays.
One screenshot per decision, not per turn
Screenshots are large image tokens. Take one, decide on a sequence of actions from it, execute them, then re-screenshot only if the next decision actually depends on the visual state. Re-screenshotting after every click is wasteful and slow.
When the user names a sandbox or mode, respect it
If the user said "in a sandbox" or "in a Docker", do not collapse to localhost just because it's the default. Start the sandbox first, target it explicitly, and only fall back to localhost with the user's confirmation. The opposite is also true: if they said "on my actual machine", do not spawn a sandbox.
Validation reminder
After editing this skill or any reference file under references/, run the senpi/Claude Skills validator before declaring work done:
python3 ~/.agents/skills/skill-creator/validate-skills.py ~/.agents/skills/cua-skill
Fix every violation until the validator prints OK: 1 skill(s) valid. Silently-rejected skills are worse than no skill.
