name	macos-gui-skill
description	Use when tasks require operating desktop applications on macOS and need screenshot-guided GUI actions.

macOS GUI Skill

When the user asks for desktop application operations, run the bundled script in this skill for local macOS execution. Use the script path relative to this skill directory so any agent client that loads the skill can call it directly.

Workflow:

Start with node ./scripts/macos-gui-skill.mjs observe to collect visual evidence.
Build one act bundle with 2-5 small GUI actions only when the next steps are obvious from the screenshot.
Run observe again immediately after the bundle to confirm the result.
Use the loop observe -> act -> observe until the task is complete.
Do not start with run-shell or run-applescript; keep them only as escape hatches after repeated visual ambiguity or explicit user instruction.

Runtime:

local macOS execution from the current workspace
the host terminal needs Accessibility and Screen Recording permission
AppleScript-based app activation may also prompt Automation permission
screenshots are stored under ./.codex/artifacts/desktop/

Examples:

node ./scripts/macos-gui-skill.mjs observe --label inbox --show-cursor true
node ./scripts/macos-gui-skill.mjs observe --active-window true
node ./scripts/macos-gui-skill.mjs observe --region 10,20,300,400
node ./scripts/macos-gui-skill.mjs act --steps '[{"type":"activate-app","appName":"Finder"},{"type":"hotkey","keys":["Meta","n"]}]'
node ./scripts/macos-gui-skill.mjs click --x 640 --y 420 --button left
node ./scripts/macos-gui-skill.mjs screenshot --filename desktop-step.png --show-cursor true
node ./scripts/macos-gui-skill.mjs doctor
node ./scripts/macos-gui-skill.mjs list-windows
node ./scripts/macos-gui-skill.mjs window-bounds --app-name Finder
node ./scripts/macos-gui-skill.mjs locate-image --image ./button-template.png --active-window true
node ./scripts/macos-gui-skill.mjs locate-image-center --image ./button-template.png --source-image ./capture.png
node ./scripts/macos-gui-skill.mjs run-applescript --script 'tell application "Finder" to activate'
node ./scripts/macos-gui-skill.mjs run-shell --command 'open -a "System Settings"'

Rules:

Operate the frontmost visible application only.
Treat every observe result as visual evidence. Do not guess the UI state from stale memory.
Prefer observe --active-window true or observe --region ... when full-screen captures include irrelevant context.
Use list-windows or window-bounds to learn the current window layout before narrowing the capture area.
Run doctor before deciding that act is unavailable or before switching to shell or AppleScript fallbacks.
Do not claim the skill "lacks dependencies" unless doctor explicitly reports actReady: false and identifies the blocker.
Do not switch to system-level UI scripts just to probe the environment. Use doctor as the single source of truth for dependency and permission readiness.
Use locate-image or locate-image-center when the next action depends on finding a stable visual template in a PNG screenshot.
Use act for the default path. Each bundle should stay inside one obvious focus chain and contain only GUI actions.
If the host window steals focus between invocations, observe and act restore the remembered target app before continuing.
Do not start with run-shell or run-applescript. If shell or AppleScript is required, use them only after repeated visual ambiguity or explicit user instruction.
If a search box or command palette keeps focus, dismiss it with press-key --key Esc, verify it closed, and only then type into the main surface.
If the next action is ambiguous, stop and ask for user direction.
Before send/delete/submit/payment or other irreversible actions, request confirmation if the user did not state it explicitly.
If an action fails twice, stop and report the current frontmost app and latest screenshot path.

Observation return fields:

path and local_image_path for the captured image
frontmostApp
screenSize
captureRegion
coordinateSpace
permissions

Doctor return fields:

actReady
blockers
fallbackPolicy
permissions
dependencies

Template matching notes:

locate-image returns found, confidence, boundingBox, and center.
locate-image-center returns found, confidence, and center.
Both commands accept --source-image for an existing PNG or will take a fresh screenshot if omitted.
Both commands perform PNG template matching only. They do not read text from the UI.

macOS GUI Skill

Workflow:

Start with node ./scripts/macos-gui-skill.mjs observe to collect visual evidence.

Build one act bundle with 2-5 small GUI actions only when the next steps are obvious from the screenshot.

Run observe again immediately after the bundle to confirm the result.

Use the loop observe -> act -> observe until the task is complete.

Do not start with run-shell or run-applescript; keep them only as escape hatches after repeated visual ambiguity or explicit user instruction.

Runtime:

local macOS execution from the current workspace

the host terminal needs Accessibility and Screen Recording permission

AppleScript-based app activation may also prompt Automation permission

screenshots are stored under ./.codex/artifacts/desktop/

Examples:

node ./scripts/macos-gui-skill.mjs observe --label inbox --show-cursor true

node ./scripts/macos-gui-skill.mjs observe --active-window true

node ./scripts/macos-gui-skill.mjs observe --region 10,20,300,400

node ./scripts/macos-gui-skill.mjs act --steps '[{"type":"activate-app","appName":"Finder"},{"type":"hotkey","keys":["Meta","n"]}]'

node ./scripts/macos-gui-skill.mjs click --x 640 --y 420 --button left

node ./scripts/macos-gui-skill.mjs screenshot --filename desktop-step.png --show-cursor true

node ./scripts/macos-gui-skill.mjs doctor

node ./scripts/macos-gui-skill.mjs list-windows

node ./scripts/macos-gui-skill.mjs window-bounds --app-name Finder

node ./scripts/macos-gui-skill.mjs locate-image --image ./button-template.png --active-window true

node ./scripts/macos-gui-skill.mjs locate-image-center --image ./button-template.png --source-image ./capture.png

node ./scripts/macos-gui-skill.mjs run-applescript --script 'tell application "Finder" to activate'

node ./scripts/macos-gui-skill.mjs run-shell --command 'open -a "System Settings"'

Rules:

Operate the frontmost visible application only.

Treat every observe result as visual evidence. Do not guess the UI state from stale memory.

Prefer observe --active-window true or observe --region ... when full-screen captures include irrelevant context.

Use list-windows or window-bounds to learn the current window layout before narrowing the capture area.

Run doctor before deciding that act is unavailable or before switching to shell or AppleScript fallbacks.

Do not claim the skill "lacks dependencies" unless doctor explicitly reports actReady: false and identifies the blocker.

Do not switch to system-level UI scripts just to probe the environment. Use doctor as the single source of truth for dependency and permission readiness.

Use locate-image or locate-image-center when the next action depends on finding a stable visual template in a PNG screenshot.

Use act for the default path. Each bundle should stay inside one obvious focus chain and contain only GUI actions.

If the host window steals focus between invocations, observe and act restore the remembered target app before continuing.

Do not start with run-shell or run-applescript. If shell or AppleScript is required, use them only after repeated visual ambiguity or explicit user instruction.

If a search box or command palette keeps focus, dismiss it with press-key --key Esc, verify it closed, and only then type into the main surface.

If the next action is ambiguous, stop and ask for user direction.

Before send/delete/submit/payment or other irreversible actions, request confirmation if the user did not state it explicitly.

If an action fails twice, stop and report the current frontmost app and latest screenshot path.

Observation return fields:

path and local_image_path for the captured image

frontmostApp

screenSize

captureRegion

coordinateSpace

permissions

Doctor return fields:

actReady

blockers

fallbackPolicy

permissions

dependencies

Template matching notes:

locate-image returns found, confidence, boundingBox, and center.

locate-image-center returns found, confidence, and center.

Both commands accept --source-image for an existing PNG or will take a fresh screenshot if omitted.

Both commands perform PNG template matching only. They do not read text from the UI.