screen-pilot

name	screen-pilot
description	Use ScreenPilot for visual desktop GUI automation. Trigger when an agent needs to select a desktop window, observe labeled UI elements, reason over a screenshot, click/type/scroll/key-press by visual target id, or install/start the local ScreenPilot helper app.
license	MIT
compatibility	macOS with Accessibility permission; ScreenPilot local helper app
allowed-tools	run_command
metadata	{"author":"DDM Team","version":"2.1.0"}

ScreenPilot

ScreenPilot automates desktop apps through an Observe-Think-Act loop:

1. Select a target window.
2. Observe a screenshot annotated with numeric UI element IDs.
3. Choose the intended element ID, optionally with vision reasoning.
4. Execute click, type, scroll, key press, or wait instructions.

The CLI auto-checks the local helper app at http://127.0.0.1:8080/api/health. If ScreenPilot is not running, it starts an installed app. If the app is missing on macOS, it downloads the latest ScreenPilot-macos-arm64.zip release from zhouchangui/bgwndsvr, installs it to ~/Applications/ScreenPilot.app, and starts it.

Commands

Run commands from this skill directory.

# List windows
node scripts/query.mjs --mode select

# Lock a window by title or process name
node scripts/query.mjs --mode select --query "WeChat"

# Capture an annotated screenshot for the locked window
node scripts/query.mjs --mode observe

# Ask a vision model which element to use
node scripts/query.mjs --mode think --query "Find the message input box"

# Click, type, scroll, key press, or run a JSON sequence
node scripts/query.mjs --mode act --target 15 --action click
node scripts/query.mjs --mode act --target 12 --action type_text --text "Hello"
node scripts/query.mjs --mode act --json '[{"type":"wait","duration":1000},{"type":"key_press","keys":["Enter"]}]'

observe writes the latest annotated image to temp/latest_observe.jpg. Read that image before choosing a target_id.

Configuration

• SCREENPILOT_BASE_URL: defaults to http://127.0.0.1:8080
• SCREENPILOT_TOKEN: overrides the local token file
• SCREENPILOT_RELEASE_REPO: defaults to zhouchangui/bgwndsvr
• SCREENPILOT_APP_PATH: overrides the app path used for startup
• OPENROUTER_API_KEY: required only for think mode
• OPENROUTER_MODEL: optional model override for think

Without SCREENPILOT_TOKEN, the CLI reads macOS token candidates under ~/Library/Application Support/.../token.conf, then falls back to the current dev token.

Actions

Action	Parameters	Description
click	target_id	Single left click
double_click	target_id	Double left click
type_text	target_id, text	Click target and type text
scroll	target_id, direction, amount	Scroll up or down
key_press	keys	Key list such as ["Enter"] or ["Meta+v"]
wait	duration	Wait milliseconds

Operating Notes

• On first macOS launch, the unsigned helper app may require manual approval in Privacy & Security.
• Window enumeration and input simulation require ScreenPilot to have Accessibility permission.
• Element IDs change after layout changes; run observe again after navigation, animation, or refresh.
• If an action fails, first re-run observe; if focus is wrong, click a neutral area or re-select the window.