| name | swift-app-automator |
| description | Automates macOS Swift applications using the Accessibility API. Use when the user needs to interact with, control, or test a native macOS application built with Swift. Triggers include requests like "automate this Swift app", "control that macOS desktop app", or any task requiring programmatic interaction with a Swift-based GUI. |
Swift App Automation
This skill enables the automation of native macOS applications built with Swift (or any native macOS app). It leverages Peekaboo (v3), a CLI tool that uses the macOS Accessibility API to inspect and control UI elements.
Prerequisites
Installation
brew tap steipete/tap && brew install peekaboo
Permissions Check
Always run this first before any automation task:
peekaboo permissions
Required permissions (grant in System Settings > Privacy & Security):
- Screen Recording (Screen & System Audio Recording)
- Accessibility
If permissions are missing, run peekaboo permissions grant for instructions.
Core Workflow
Every automation task follows this pattern:
- Check Permissions: Run
peekaboo permissions to verify access.
- Launch App: Use
open -a "AppName" or peekaboo app launch "AppName".
- Capture UI (See): Run
peekaboo see --app AppName --json to get a snapshot with element IDs.
- Interact: Use element IDs from the snapshot to click, type, scroll, etc.
- Verify: Capture another snapshot to confirm the UI changed as expected.
Example: Interacting with macOS Calculator
# 1. Launch the Calculator app
open -a Calculator
# 2. Capture the UI to find element IDs
peekaboo see --app Calculator --json
# Output includes:
# "snapshot_id": "C7E7D013-...",
# "ui_elements": [
# {"id": "elem_10", "role": "button", "description": "1"},
# {"id": "elem_14", "role": "button", "description": "add"},
# {"id": "elem_18", "role": "button", "description": "2"},
# {"id": "elem_22", "role": "button", "description": "equals"},
# ]
# 3. Extract snapshot ID (use jq or python)
SNAP_ID=$(peekaboo see --app Calculator --json | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
# 4. Perform a calculation: 1 + 2 =
peekaboo click --on "elem_10" --snapshot "$SNAP_ID"
peekaboo click --on "elem_14" --snapshot "$SNAP_ID"
peekaboo click --on "elem_18" --snapshot "$SNAP_ID"
peekaboo click --on "elem_22" --snapshot "$SNAP_ID"
# 5. Verify the result
peekaboo see --app Calculator --json
# Check the output to confirm the result.
Example: Typing into an App
# 1. Capture UI to find the text input element
SNAP_ID=$(peekaboo see --app ChatGPT --json | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
# 2. Click the text input field (use element ID from the snapshot)
peekaboo click --on "elem_142" --snapshot "$SNAP_ID"
# 3. Type text (positional argument, NOT --text)
peekaboo type "Hello, world!" --app ChatGPT
# 4. Press Enter to submit
peekaboo press return --app ChatGPT
Essential Commands
Capture & Inspect
| Command | Description |
|---|
peekaboo see --app <App> --json | Capture UI elements with IDs and screenshot |
peekaboo see --app <App> --annotate | Capture with visual annotations on screenshot |
peekaboo see --mode screen | Capture the full screen |
Click & Interact
| Command | Description |
|---|
peekaboo click --on <elem_ID> --snapshot <SNAP_ID> | Click element by ID |
peekaboo click --on <elem_ID> --snapshot <SNAP_ID> --double | Double-click |
peekaboo click --on <elem_ID> --snapshot <SNAP_ID> --right | Right-click |
peekaboo click --coords 100,200 --app <App> | Click at coordinates |
Type & Keyboard
| Command | Description |
|---|
peekaboo type "text" --app <App> | Type text into focused element |
peekaboo type "text" --return | Type text and press Enter |
peekaboo type "text" --clear | Clear field first, then type |
peekaboo press return | Press Enter/Return key |
peekaboo press tab --count 3 | Press Tab 3 times |
peekaboo press escape | Press Escape key |
peekaboo hotkey "cmd,c" | Keyboard shortcut (Cmd+C) |
peekaboo hotkey "cmd,shift,t" | Multi-modifier shortcut |
Scroll
| Command | Description |
|---|
peekaboo scroll --direction down --amount 5 --app <App> | Scroll down |
peekaboo scroll --direction up --amount 10 --app <App> | Scroll up |
peekaboo scroll --direction down --amount 3 --smooth | Smooth scroll |
Paste (Clipboard)
| Command | Description |
|---|
peekaboo paste "text" --app <App> | Set clipboard, paste, restore |
peekaboo paste --file-path /tmp/image.png --app <App> | Paste a file |
Application Control
| Command | Description |
|---|
peekaboo app launch "AppName" | Launch an app |
peekaboo app launch "AppName" --wait-until-ready | Launch and wait |
peekaboo app quit --app <App> | Quit an app |
peekaboo app relaunch "AppName" | Restart an app |
peekaboo app switch --to <App> | Switch to an app |
peekaboo app list --json | List running apps |
Window Management
| Command | Description |
|---|
peekaboo window list --app <App> --json | List windows |
peekaboo window focus --app <App> | Bring window to front |
peekaboo window move --app <App> --x 100 --y 100 | Move window |
peekaboo window resize --app <App> --width 1200 --height 800 | Resize window |
peekaboo window minimize --app <App> | Minimize window |
peekaboo window maximize --app <App> | Maximize window |
peekaboo window close --app <App> | Close window |
Menu Bar
| Command | Description |
|---|
peekaboo menu list --app <App> | List all menu items |
peekaboo menu click --app <App> --item "New Window" | Click a menu item |
peekaboo menu click --app <App> --path "Format > Font > Show Fonts" | Nested menu |
Important Notes
Snapshot & Element ID Pattern
The most common mistake is trying to click elements without a valid snapshot. Always follow this pattern:
# Step 1: Capture → get snapshot_id and element IDs
SNAP_ID=$(peekaboo see --app <App> --json | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
# Step 2: Use the snapshot_id when clicking
peekaboo click --on "elem_XX" --snapshot "$SNAP_ID"
Extracting Text from UI Elements
To read the text content of a response or UI state:
peekaboo see --app <App> --json | python3 -c "
import json, sys
d = json.load(sys.stdin)
for el in d['data']['ui_elements']:
text = el.get('value', '') or el.get('description', '') or el.get('label', '')
if text and len(str(text)) > 10:
print(f\"[{el['id']}] role={el['role']} | {str(text)[:200]}\")
"
Waiting for Content Generation
When interacting with AI apps or slow-loading content, check if generation is still in progress:
# Poll until generation is complete
peekaboo see --app <App> --json | python3 -c "
import json, sys
d = json.load(sys.stdin)
for el in d['data']['ui_elements']:
desc = el.get('description', '') + el.get('label', '')
if 'Thinking' in desc or 'Stop' in desc or '考えています' in desc:
print('GENERATING')
sys.exit(0)
print('DONE')
"
Global Options
All commands support these flags:
--json / -j: Machine-readable JSON output--app <App>: Target application by name or bundle ID--verbose / -v: Enable verbose logging
For full command reference, see references/peekaboo_command_reference.md.
