菜单
Use when you need to visually interact with a GUI — test buttons, fill forms, verify visual layouts, fuzz web pages, automate user flows, take screenshots, or perform end-to-end QA on any application. Works on cloud VMs, Docker containers, local machines, and sandboxes. Install: pip install cua.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/am-will/gooey-pi --skill gui-automation命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
Launch and drive the GooeyPi desktop (Electron) app to prove user-facing behavior — sidebar navigation, Settings, and Capabilities — capturing screenshots and ARIA snapshots. Reach for this when verifying a GooeyPi UI change or confirming the app still works end to end.
Sketch types, signatures, and module structure before code, then stay in the loop while implementation fills in. Use for /architect, 'architect this', 'design this', or non-trivial work where jumping to code would lock in the wrong shape.
Spawn N parallel candidates at the same task, pick a base, graft the strongest parts of the losers into it. Use for /arena, 'arena this', 'throw it in the arena', or when one attempt at a non-trivial artifact would lock in the wrong shape.
| name | gui-automation |
| description | Use when you need to visually interact with a GUI — test buttons, fill forms, verify visual layouts, fuzz web pages, automate user flows, take screenshots, or perform end-to-end QA on any application. Works on cloud VMs, Docker containers, local machines, and sandboxes. Install: pip install cua. |
CUA gives you eyes and hands on a real computer: see the screen, move the mouse, click, type, drag, and manage windows — like a human at the keyboard.
Use this skill for visual interaction that can't be done via shell or API.
cua --version # check install; if missing: pip install cua
# Connect to target (pick one)
cua do switch cloud my-vm
cua do switch docker my-container
cua do-host-consent && cua do switch host # local machine (one-time consent)
ANTHROPIC_API_KEYis optional. With it,cua do snapshotreturns an AI-annotated screen with element coordinates. Without it, usescreenshotand read the image yourself.
Look → Act → Verify — repeat until done, then share:
cua do screenshot # look
cua do click 450 280 # act
cua do screenshot # verify
cua trajectory share # share replay link with user
Re-screenshot after every UI change — coordinates go stale when the screen changes.
cua do screenshot
cua do click 450 280
cua do screenshot
cua do screenshot
cua do click 400 200 && cua do type "Jane Doe"
cua do key tab && cua do type "jane@example.com"
cua do key tab && cua do type "SecureP@ss123"
cua do click 400 500
cua do screenshot
cua do click 350 400 # "Choose File"
cua do type "/home/user/report.pdf"
cua do key enter
cua do screenshot
When clicking small or dense UI elements — especially on the host machine — zoom into the target window first. Coordinates become window-relative and screenshots show only that window, giving you higher effective resolution.
cua do zoom "Google Chrome" # crop to Chrome window; coords are now window-relative
cua do screenshot # zoomed view — easier to locate small elements
cua do click 112 44 # precise click on a small tab or button
cua do screenshot # verify
cua do unzoom # restore full-screen coords when done
cua do screenshot # back to full desktop view
Use
zoomany time click accuracy is uncertain.unzoombefore switching windows or when you need to see the full desktop again.
cua do window ls # list open windows
cua do drag 150 300 650 400 # source → destination
cua do screenshot
cua do screenshot
cua do click 400 200
cua do type "<script>alert(1)</script>"
cua do key tab && cua do type "'; DROP TABLE users; --"
cua do key tab && cua do type "AAAAAAAAAAAAAAAAAAAAAAA"
cua do click 400 500
cua do screenshot # check for errors, crashes, unexpected behavior
Every action is auto-recorded to ~/.cua/trajectories/{machine}/{session}/.
cua trajectory share # upload and get shareable HTTPS link (always do this at end)
cua trajectory ls # list sessions
cua trajectory export # generate HTML report
cua do --no-record click 100 200 # disable recording for a single action
Tell the user: "Here is the trajectory of my session: {url}"
| Action | Command |
|---|---|
| Connect to target | cua do switch <provider> [name] |
| Screenshot | cua do screenshot |
| AI-annotated screen | cua do snapshot ["instructions"] |
| Click | `cua do click [left |
| Double-click | cua do dclick <x> <y> |
| Type text | cua do type "text" |
| Press key | cua do key <key> |
| Hotkey | cua do hotkey <combo> (e.g. ctrl+c) |
| Scroll | cua do scroll <direction> [amount] |
| Drag | cua do drag <x1> <y1> <x2> <y2> |
| Move cursor | cua do move <x> <y> |
| Shell command | cua do shell "command" |
| Open URL/file | `cua do open <url |
| List windows | cua do window ls [app] |
| Focus window | cua do window focus <id> |
| Zoom to window | cua do zoom "App Name" |
| Unzoom | cua do unzoom |
| Share trajectory | cua trajectory share |
| Provider | Example |
|---|---|
cloud | cua do switch cloud my-vm |
cloudv2 | cua do switch cloudv2 my-vm |
docker | cua do switch docker my-container |
lume | cua do switch lume my-vm |
lumier | cua do switch lumier my-vm |
winsandbox | cua do switch winsandbox |
host | cua do switch host |
See references/command-reference.md for full argument syntax.