desktop-automation-computer-use

name	Desktop Automation (Computer Use)
description	Control the macOS desktop: take screenshots, move/click the mouse, type text, and read app content via Accessibility API. Uses native macOS APIs (screencapture, CoreGraphics, System Events, Accessibility) with zero external dependencies.
version	2.0.0
category	automation
skillType	claude-skill
assignableRoles	["generalist","designer","qa"]
triggers	["screenshot","click","mouse","type text","desktop automation","computer use","read app","scroll","accessibility","ui elements"]
tags	["automation","desktop","screenshot","mouse","keyboard","macos","accessibility","ui-reading"]
execution	{"type":"script","script":{"file":"execute.sh","interpreter":"bash","timeoutMs":30000}}

Desktop Automation (Computer Use)

Control the macOS desktop by taking screenshots, moving/clicking the mouse, typing text, and reading app content via Accessibility API (no screenshots needed).

Actions

Screenshot

Capture the screen (or a region) to an image file.

bash execute.sh '{"action":"screenshot"}'
bash execute.sh '{"action":"screenshot","output":"/tmp/my-screenshot.png"}'

Move Mouse

Move the mouse cursor to a specific coordinate.

bash execute.sh '{"action":"move","x":500,"y":300}'

Click

Click at a specific coordinate. Supports left click, right click, and double click.

bash execute.sh '{"action":"click","x":500,"y":300}'
bash execute.sh '{"action":"click","x":500,"y":300,"button":"right"}'
bash execute.sh '{"action":"click","x":500,"y":300,"button":"double"}'

Type Text

Type a string of text using simulated keystrokes.

bash execute.sh '{"action":"type","text":"Hello, world!"}'

List Apps

List all running GUI applications with their names and bundle IDs.

bash execute.sh '{"action":"list-apps"}'

Focus App

Bring a specific app to the foreground.

bash execute.sh '{"action":"focus-app","appName":"discover"}'

Read UI (Accessibility API)

Read the UI element tree from a specific app. Returns structured data including element roles, titles, values, and positions — without taking a screenshot.

bash execute.sh '{"action":"read-ui","appName":"discover"}'
bash execute.sh '{"action":"read-ui","appName":"discover","maxDepth":4}'

Output includes: element roles (button, text, image, etc.), titles, values (text content), descriptions, positions, sizes, and child elements.

Get Text (Accessibility API)

Extract all visible text content from an app's UI. Faster than read-ui when you only need text, not the full element structure.