computer-use

name

computer-use

description

Control the user's macOS desktop — take screenshots, click, type, scroll, drag, manage apps, and more via the `computer-use` CLI binary. Use this skill whenever the user asks you to interact with a native macOS application (Maps, Notes, Finder, Photos, System Settings, any third-party native app), perform cross-app workflows on their desktop, take a screenshot of their screen, click on something they're showing you, fill in a form in a desktop app, or automate any GUI task. Also trigger when the user says "open [app]", "click on [thing]", "type [text] into [app]", "take a screenshot", "show me what's on screen", "move the mouse", "scroll down", or any similar desktop interaction request. If the user asks to be taught or walked through something on their screen, use teach mode. Do NOT use this for web browsing — prefer browser extension tools for that.

Computer Use (Desktop Control)

You have access to a computer-use CLI binary that controls the user's macOS desktop — screenshots, mouse/keyboard input, app management, session orchestration with safety guards, and teach mode.

First, ensure it's installed by running scripts/ensure-installed.sh from this skill's directory. If not installed, it will auto-install via brew install dnakov/computer-use/computer-use. Once installed, the binary is available as computer-use in PATH.

CU=computer-use

Decision Framework

Pick the right tool tier — speed and precision beat pixel-clicking:

1. Dedicated MCP (Slack, Gmail, Calendar, Linear) — if connected, use it
2. Browser extension tools — for web apps, DOM-aware, faster than clicking pixels
3. Computer use (this skill) — for native desktop apps and cross-app workflows

Computer use IS the right tool for native apps — don't decline a native-app task because there's no MCP for it.

Core Principles

• Look before you assert. Always screenshot before making claims about app state. Don't answer from memory.
• Screenshot first, then click. Coordinates come from the most recent screenshot image. Never guess coordinates.
• Respect tiers. The grant response tells you what's allowed. Don't try to work around restrictions.
• Never click web links. If you see a URL in a native app, don't click it — use browser tools instead.
• Never execute trades or move money. Always ask the user to perform financial actions themselves.

Full Workflow

1. Check permissions

$CU tcc check-accessibility        # → {"granted": true/false}
$CU tcc check-screen-recording     # → {"granted": true/false}

If not granted, use $CU tcc request-accessibility / request-screen-recording to prompt.

2. Start a session

$CU session start --id my-session

3. Request access to apps

$CU session grant --id my-session \
  --apps "Notes" "Finder" \
  --reason "Help organize files" \
  [--clipboard-read] [--clipboard-write] [--system-key-combos]

Response includes tier per app:

• "full" — no restrictions (most apps)
• "click" — left-click + scroll only; NO typing, keys, right-click, modifier-clicks, drag (terminals, IDEs)
• "read" — visible in screenshots only, NO interaction (trading apps)

The --clipboard-read/--clipboard-write flags are separate grants. --system-key-combos enables Cmd+Q, Cmd+Tab, etc. which are blocked by default.

If an app is policy-blocked (media apps: Spotify, Netflix, Kindle, etc.), it will be denied regardless. Check: $CU apps is-policy-blocked --bundle-id com.spotify.client

Call grant again mid-session to add more apps — previously granted apps persist.

4. Take a screenshot

$CU screenshot capture-excluding [--exclude-bundle-ids id1 id2] [--display-id N]

Returns JSON with base64 (JPEG data URL), width, height, displayWidth, displayHeight, displayId, originX, originY. Save the screenshot dimensions — all subsequent coordinates are in image pixels relative to this image.

For higher-resolution inspection of small text or UI details:

$CU screenshot capture-region \
  --region-x X --region-y Y --region-w W --region-h H \
  --output-width W --output-height H

Region coordinates are in logical display points (not image pixels). Useful for reading fine text the downsampled full screenshot can't resolve.

5. Interact

Coordinate system: All coordinates are (x, y) in image pixels from the most recent screenshot. The CLI converts to screen points automatically: screenX = pixelX * (displayWidth / screenshotWidth) + originX

You can also convert explicitly:

$CU display convert-coordinates \
  --pixel-x 500 --pixel-y 300 \
  --screenshot-width 1456 --screenshot-height 819 \
  --display-width 3008 --display-height 1692 \
  --origin-x 0 --origin-y 0

Click

$CU input move-mouse --x 500 --y 300 --relative false
$CU input mouse-button --button left --action click --count 1
# Double click: --count 2, Triple: --count 3
# Buttons: left, right, middle

Type

$CU input type-text --text "Hello world"

Key combos

$CU input keys --key-names command a         # Select all
$CU input keys --key-names command shift z    # Redo
$CU input key --key-name return              # Single key
$CU input key --key-name escape

Check before sending system combos:

$CU input is-system-combo --keys meta+q    # → {"isSystemCombo": true}

System combos (Cmd+Q, Cmd+Tab, Cmd+Space, Ctrl+Cmd+Q, Alt+Cmd+Escape) are blocked unless --system-key-combos was granted.

Scroll

$CU input mouse-scroll --amount 5 --axis vertical    # positive=down
$CU input mouse-scroll --amount -3 --axis horizontal  # negative=left

Drag

$CU input drag --start-x 100 --start-y 200 --end-x 300 --end-y 400
# Start coords optional — omit to drag from current cursor position

Mouse move (without clicking)

$CU input move-mouse --x 500 --y 300 --relative false
# Animated by default (cubic ease-out, 60fps). Use --no-animate to warp instantly.

Hold key for duration

$CU input hold-key --keys shift+down --duration 2.0   # seconds (0-100)

Press and hold mouse (for complex drag sequences)

$CU input mouse-button --button left --action press    # Hold down
$CU input move-mouse --x 300 --y 400 --relative false  # Move while held
$CU input mouse-button --button left --action release   # Release

Wait

$CU wait --duration 1.5   # seconds (0-100)

Get cursor position

$CU input mouse-location   # → {"x": 500, "y": 300}

6. Clipboard

$CU clipboard read                        # → {"text": "..."}
$CU clipboard write --text "hello"
$CU clipboard paste --text "hello world"  # Stash → write → Cmd+V → restore original

Clipboard read/write require --clipboard-read/--clipboard-write grants.

7. App management

$CU apps open --bundle-id com.apple.Notes
$CU apps list-running
$CU apps list-installed
$CU apps classify --bundle-id com.apple.Safari
  # → {"category": "browser", "tier": "full"}
$CU apps app-under-point --x 500 --y 300
  # → {"bundleId": "com.apple.Notes"}
$CU apps unhide --bundle-ids com.apple.Safari

8. Window management

$CU window get-frontmost-app
  # → {"appName": "Notes", "bundleId": "com.apple.Notes", "appIconBase64": "..."}
$CU window get-active-handle
$CU window focus --window-id 12345
$CU window get-app-for-file --path /Users/me/document.pdf
  # → {"appName": "Preview", "appIconBase64": "..."}

9. Multi-display support

$CU display list-all
  # → [{"displayId":1, "width":3008, "height":1692, "scaleFactor":2, "isPrimary":true, ...}]
$CU display get-size [--display-id N]

When multiple monitors are connected, screenshots capture one display. The CLI auto-resolves the best display based on where granted apps have windows.

10. Batch operations

Execute multiple actions in one call — eliminates round-trip latency:

$CU session batch --id my-session --actions '[
  {"action": "left_click", "coordinate": [500, 300]},
  {"action": "type", "text": "Hello"},
  {"action": "key", "text": "return"},
  {"action": "screenshot"}
]'

Allowed batch actions: key, type, mouse_move, left_click, left_click_drag, right_click, middle_click, double_click, triple_click, scroll, hold_key, screenshot, cursor_position, left_mouse_down, left_mouse_up, wait

Actions execute sequentially with 10ms delay between them. Stops on first error, returns partial results.

11. End session

$CU session end --id my-session

Cleanup: unhides any hidden apps, restores stashed clipboard, releases held mouse buttons, releases session lock.

Teach Mode

When the user wants to learn how to do something ("teach me", "walk me through", "show me how"), use teach mode:

# Single tooltip — show explanation, wait for Next/Exit
$CU teach show-step \
  --explanation "This is the search bar." \
  --next-preview "Next: I'll type your search query." \
  [--anchor-x 500 --anchor-y 60]
# → {"action": "next"} or {"action": "exit"}

# Multi-step batch with actions (full walkthrough)
$CU teach batch --session-id my-session --steps '[
  {
    "explanation": "I will click the search field.",
    "next_preview": "Next: Click and type a query.",
    "anchor": [500, 60],
    "actions": [
      {"action": "left_click", "coordinate": [500, 60]},
      {"action": "type", "text": "meeting notes"}
    ]
  },
  {
    "explanation": "Search results are showing. Done!",
    "next_preview": "Click Next to finish.",
    "actions": []
  }
]'

Teach batch flow:

1. All steps validated upfront before any UI shows
2. For each step: show tooltip → user clicks Next → execute actions → next tooltip
3. User can click Exit at any point to stop
4. Trailing screenshot taken after the last step if any step had actions
5. Requires active session with prior screenshot (for coordinate conversion)

The tooltip uses native macOS vibrancy styling, auto-places (below > above > right > left), and supports dark mode.

Tier Enforcement Reference

Tier	Screenshot	Left Click	Scroll	Type/Key	Right Click	Modifier Click	Drag
full	Yes	Yes	Yes	Yes	Yes	Yes	Yes
click	Yes	Yes	Yes	No	No	No	No
read	Yes	No	No	No	No	No	No

App categories:

• Browsers (Safari, Chrome, Firefox, Edge, Arc, Brave, Opera, Vivaldi, etc.) → full
• Terminals/IDEs (Terminal, iTerm, VS Code, Cursor, Xcode, JetBrains, etc.) → click
• Trading/Finance (Webull, TradingView, Binance, etc.) → read
• Everything else → full

If you get a tier error, don't try to work around it with AppleScript, System Events, or shell commands. Use the appropriate alternative:

• Terminal (click tier) → use the Bash tool for shell commands
• Read-only app → ask the user to take actions themselves

Policy-Blocked Apps

These apps are completely blocked — no override possible:

• Media streaming: Apple TV, Spotify, Plex, Amazon Prime Video, Netflix, Disney+, Hulu, Peacock, Paramount+, Tubi, Crunchyroll
• Music: Apple Music, Amazon Music, Tidal, Deezer, Pandora, YouTube Music, Pocket Casts
• Books/Reading: Apple Books, Kindle, Kobo, Calibre, Libby, Audible, Readium

Key Names Reference

Modifiers: Command, Shift, Control, Alt/Option, RCommand, RShift, RControl, ROption, Function, CapsLock Navigation: Return, Tab, Space, Backspace, Delete, Escape Arrows: UpArrow, DownArrow, LeftArrow, RightArrow, Home, End, PageUp, PageDown Function: F1 through F20 Media: VolumeUp, VolumeDown, VolumeMute, BrightnessUp, BrightnessDown, MediaPlayPause, MediaNextTrack, MediaPrevTrack, MediaFast, MediaRewind, Eject, Launchpad, MissionControl Numpad: Numpad0-Numpad9, Add, Subtract, Multiply, Divide, Decimal

System Queries

$CU system read-plist --file /path/to/file.plist --key SomeKey
$CU system read-cf-pref --key AppleInterfaceStyle   # → "Dark" or "Light"
$CU system is-process-running --process-id com.apple.Safari

Session Management Details

$CU session status --id my-session     # Full state: grants, flags, mouse state
$CU session list                        # All active sessions
$CU session lock                        # Check who holds the lock
$CU session revoke --id my-session --bundle-id com.apple.Safari

Only one session can hold the lock at a time. If another session is active, you'll get an error — wait for it to finish or ask the user.

Practical Patterns

Open app, wait, screenshot, click

CU=/Users/sigkitten/dev/cu-swift/.build/debug/computer-use
$CU apps open --bundle-id com.apple.Notes
$CU wait --duration 1.0
$CU screenshot capture-excluding
# Read the screenshot, find the target coordinates, then:
$CU input move-mouse --x 750 --y 400 --relative false
$CU input mouse-button --button left --action click --count 1

Click a field and type

$CU input move-mouse --x 500 --y 300 --relative false
$CU input mouse-button --button left --action click --count 1
$CU input type-text --text "Hello world"
$CU input keys --key-names return

Select all, copy, read clipboard

$CU input keys --key-names command a
$CU input keys --key-names command c
$CU clipboard read

Scroll to find content

$CU input move-mouse --x 500 --y 500 --relative false
$CU input mouse-scroll --amount 10 --axis vertical
$CU wait --duration 0.5
$CU screenshot capture-excluding

Multi-step form fill via batch

$CU session batch --id my-session --actions '[
  {"action": "left_click", "coordinate": [200, 300]},
  {"action": "type", "text": "John Doe"},
  {"action": "key", "text": "tab"},
  {"action": "type", "text": "[e-mail]"},
  {"action": "key", "text": "tab"},
  {"action": "type", "text": "Hello, this is my message."},
  {"action": "key", "text": "return"}
]'