macos-control

name

macos-control

description

Control macOS UI via the Accessibility API and record short screen captures reviewed frame-by-frame. Use when automating native apps — clicking buttons, filling forms, reading element state, finding elements, getting window bounds — or when the user wants a short screen recording of an animation/transition/flicker reviewed via a contact sheet, optionally pushed to a vitrinka board. Triggers on "click button in app", "fill form in native app", "automate macOS app", "list UI elements", "find button", "read text field value", "get window position", "interact with native app", "record the screen", "capture this transition", "Xs recording", "Nfps", "watch this animation", "why does it jump/flicker", "record this app for a few seconds", "push the recording to a board".

tools control — macOS UI automation + screen recording

Fast native CLI for reading and controlling macOS app UI via the Accessibility API (10-30x faster than osascript: ~66ms get, ~130ms list, ~170ms press vs 2-5s), plus a recording runner (tools control capture) for multi-frame captures with timed actions. tools macos control is an alias.

Resources in this skill: references/capture.md (recording discipline — read before writing capture plans), references/peekaboo.md (peekaboo flag tables), references/vitrinka.md (optional board publishing).

Commands

Discovery (find elements you want to interact with)

tools control preflight --app <name>                 # RUN THIS FIRST — screens, frontmost, windows,
                                                     #   elements by role, browser tab, suggested plan
                                                     #   --wanted screens,windows,elements[,elements:<Role>],browser,frontmost,plan
tools control apps                                   # Running apps — valid --app values
tools control list --app <name> [--depth N]          # Flat list of all elements (max 2000)
tools control tree --app <name> [--depth N]          # Hierarchical nested tree (shows parent-child)
tools control find --app <name> --role button        # Fuzzy: "button" matches AXButton
tools control find --app <name> --title "Save"       # Find by title (substring, case-insensitive)
tools control find --app <name> --text "YouTube"     # Search title+desc+value at once (OR)
tools control find --app <name> --desc "Chat"        # Find by description (many SwiftUI elements use this)
tools control find --app <name> --subrole close      # Fuzzy: "close" matches AXCloseButton
tools control find --app <name> --role button --title "Email" --exact  # --exact forces strict role match

Role/subrole matching is fuzzy by default: button finds AXButton, radio finds AXRadioButton, close finds AXCloseButton. Add --exact when you need strict matching.

Inspection (read element details)

tools control get --app <name> --id <axId>           # Read role/title/value/description
tools control attrs --app <name> --id <axId>         # ALL attributes with decoded values
tools control actions --app <name> --id <axId>       # Available AX actions (AXPress, AXShowMenu, etc.)
tools control window --app <name>                    # Window bounds: x,y,width,height + minimized/fullscreen

Interaction (modify elements)

tools control set --app <name> <target> --value "text"                  # Set text field + HARD VERIFY (reads back, 1 retry)
tools control press --app <name> <target>                               # Press (AXPress — AX action path)
tools control click --app <name> <target>                               # CGEvent click at element center
tools control perform --app <name> <target> --action AXShowMenu         # Any AX action
tools control focus --app <name> [<target>]                             # Activate app + focus element
tools control type --app <name> --text "hello" [<target>]               # Type + HARD VERIFY ([--clear] [--return])
tools control scroll --app <name> --direction down [--amount N]         # Wheel scroll (at <target> center or --coords x,y)
tools control scroll --app <name> <target>                              # No --direction: AXScrollToVisible (bring into view)
tools control hotkey --keys cmd,shift,a [--app <name>]                  # Key combo (--app activates target first)

Verification (wait / assert — replaces sleep-guessing)

tools control wait --app <name> <target> [--timeout 5000] [--interval 200]   # Poll until element exists
tools control wait --app <name> <target> --gone                              # ...until it disappears
tools control wait --app <name> <target> --for enabled|focused               # ...until enabled/focused
tools control wait --app <name> <target> --contains "Saved"                  # ...until AXValue contains text
tools control assert --app <name> <target> [--expect V|--contains T|--gone]  # Single-shot check, exit 1 on fail

Both work as plan steps too: { "do": "wait", "q": "Save", "timeout": 3000 }, { "do": "assert", "id": "status", "contains": "Done" } — plans become UI tests.

Vision (screenshot / annotate / OCR)

tools control screenshot --app <name> --path /tmp/s.png [--window T] [--crop x,y,w,h]
tools control screenshot --app <name> --path /tmp/s.png --annotate [--all]   # numbered boxes on interactable
                                                                             #   elements + legend in --json
tools control ocr --app <name> [--window T] [--crop x,y,w,h]                 # Vision OCR: read text + pixel boxes
tools control ocr --image /tmp/s.png                                         # OCR an existing file

--annotate is the "what can I click?" picture: every interactable element gets a numbered box, the JSON legend maps numbers to id/role/desc. ocr reads rendered text pixels — the check that survives apps lying in their AX tree. Both default to the app's LARGEST window; --window <title> scopes either. (screenshot --annotate = AX element legend; drawing SHAPES on an image is the separate draw command below.)

Draw annotations on ANY image (draw) — capture-agnostic post-processing

tools control draw shot.png --annotate '[{"kind":"highlight","rect":{"x":748,"y":812,"w":1246,"h":430},"label":{"text":"Build pipeline"}}]'
tools control draw shot.png --annotate plan.json --out annotated.png [--preset review-red|callout-amber|redact]

Draws a JSON annotation plan onto an existing image: highlight (rounded-rect outline + translucent wash — the review register), box, ellipse, arrow {from,to}, label {at,text} (chip), blur {rect,strength} (redact secrets), crop {rect} (applied LAST), grid {step,originOffset,labels} (coordinate finder — what clickmap uses). Coordinates are NATURAL IMAGE PIXELS; annotations draw in array order; the input is never mutated without --in-place. --annotate sniffs inline JSON ({/[) vs a plan.json path. MCP twin: annotate_image { input, annotations, output?, preset? } on the genesis-tools server.

Which capture tool feeds it — pick deliberately:

• Web app → playwright-mcp browser_take_screenshot: exact CSS-pixel viewport or full-page DOM capture, no window chrome or personal UI leaking in, fullPage for scrollable pages, works headless.
• Native app / OS surface (menu bar, dialogs, Dock) / cross-app flow → this skill's screenshot/capture: physical-screen capture is the only option there, but it needs the window frontmost/unoccluded and can't capture a full scrollable page.
• Neither can produce an ANNOTATED artifact by itself — that's draw's job, deliberately designed as annotate-an-existing-image so it works on ANY capture source.

Compare two screenshots (compare-screenshot)

tools control compare-screenshot a.png b.png [--threshold 0.1] [--max-mismatch 0.5] [--diff-out diff.png] [--resize-to-match] [--json]

Pixelmatch diff: mismatch count/% + similarity score; --max-mismatch <pct> gates the exit code (0 pass / 1 fail / 2 unusable inputs). Visual-regression checks, migration verification, "did anything change?".

Snapshot / restore (leave the machine how you found it)

tools control snapshot                          # prints JSON: mouse position + frontmost app/window
tools control restore --snapshot '<that json>'  # restores mouse + focus (takes the literal JSON, not a file path)

Wrap disruptive work: SNAP=$(tools control snapshot) → do focus-stealing things → tools control restore --snapshot "$SNAP". Plans do this declaratively with "restore": true.

Record a plan instead of writing one

tools control record-plan start --record all       # commands | activity | all
# ...run tools control commands and/or drive the UI by hand...
tools control record-plan stop --out plan.json     # synthesized, runnable plan
tools control record-plan --record activity --duration 20 --out plan.json   # one-shot

commands logs subsequent tools control action commands (any terminal). activity records real user clicks/keys/scrolls via a CGEvent tap, clicks resolved to AX elements (id > desc > title > coords). all merges both, deduping our own synthetic events. Keystrokes coalesce into type steps, combos into hotkey, wheel bursts into scroll. Review before running.

Multi-session warning: the commands recorder is machine-global. Commands from a different terminal/Claude session get marked "_foreign" in the plan and warned about in status/stop; stop --exclude-foreign drops them. Concurrent subagents of the SAME session are indistinguishable (same env identity) — don't run parallel agents through tools control while recording.

Targeting (<target>)

All interaction/inspection commands accept:

• --q <query> — universal search (checks id + title + desc + value + role + subrole at once). The escape hatch when you don't know WHICH attribute holds the visible text — reach for it first when unsure.
• --id <axId> — exact AXIdentifier.
• Any combination of --role/--title/--desc/--subrole (first match), --window <title> to scope, --exact for strict role matching.
• --depth <n> — search depth (default 15; browser page content may need 40 — a 0-match result at default depth hints this).

Elements without AXIdentifier (browser tabs, toolbar buttons, window close buttons) are fully interactable:

# By AXIdentifier (native apps with .accessibilityIdentifier)
tools control press --app Genesis --id nav-chat

# By role + description (browser elements — no id needed)
tools control click --app "Brave Browser" --desc "Reload" --role AXButton
tools control click --app "Brave Browser" --desc "YouTube" --role AXRadioButton

# By description alone (Genesis settings tabs sharing the same id)
tools control press --app Genesis --desc "Account" --role AXButton

# By subrole (window close/minimize/fullscreen buttons have no id or desc)
tools control click --app Genesis --subrole AXCloseButton --window Settings

# --window scopes search to a specific window (by title substring)
tools control click --app Genesis --subrole AXMinimizeButton --window Genesis

click vs press vs set

Command	Mechanism	When to use
press	AX action (AXPress)	Buttons/toggles in native apps — position-independent, works on obscured or scrolled-away elements
click	CGEvent at element center	Real mouse click — exercises hit testing, works on web content, triggers hover/focus
set	Text fields: CGEvent clear+type, then reads the field back (retry once, fail loud). Other elements: AXValue write	Text fields — verified content
type	CGEvent keystrokes	When you need real typing (autocomplete, validation, non-AX inputs)
focus	NSRunningApplication.activate + AXFocused	Bring app/element to front before typing

click scroll-safety

click checks if the element center is within any visible window bounds. If the element is scrolled out of view (e.g. below the fold in a scroll area), it falls back automatically:

• Buttons: falls back to AXPress (position-independent)
• Text fields: falls back to AXFocus (focusing a text field == clicking it)
• Output includes "fallback": "AXPress"|"AXFocus" and "warning" when this happens

For elements below the fold, prefer press (buttons) or focus + type (text fields) over click.

Gotchas

• One keyboard, one frontmost app — keyboard commands are machine-exclusive. set/type/hotkey activate the target app and stream real CGEvents; a CONCURRENT session doing the same steals frontmost mid-type and keystrokes land in the wrong app (hard-verify catches it as "landed NOWHERE/different element", but the work still fails). Never run two agents that both do keyboard/focus work at the same time — parallelize read-only commands (find/get/ocr/screenshot) freely, serialize interaction.

• App-level screenshots capture the wrong window when an app has multiple windows (e.g. Genesis main + Settings) — apps mark popups/strips as "main". Always pass a window title: tools control screenshot --app X --window "..." (peekaboo equivalent: --window-title).

• Browser elements use AXDescription, not AXTitle — tab text is in desc, not title. Use find --text "YouTube" (searches all attributes) or find --desc "YouTube" specifically.

• Browser tabs are AXRadioButton, not AXButton — find --role AXButton finds bookmark bar items, find --role AXRadioButton finds actual tabs.

• Two instances of the same app (e.g. two Brave profiles): name/bundleId resolution fails loud with a candidates list — target one with --app <pid> (pids from tools control apps). Preflight's browserTab carries pidMatch/warning because AppleScript resolves by name and may answer for the other instance.

Output

Default output is a human-readable summary line (pressed nav-chat, assert ok board-poll-hint). Add --json for the raw machine JSON — {"ok": true, ...} on success, {"ok": false, "error": "..."} on failure (compact; --pretty to indent). In list/find tables, the label column shows title, else desc, else value — static text usually surfaces in value, buttons in desc. Safety semantics: set/type refuse when the target app is not frontmost and hard-verify the field content after typing; screenshot --window and --window scoping fail loud with a candidates list on 0 or 2+ matches; window output flags transient popups ("transient": true).

Plan runner (tools control run)

ONE plan schema covers sequential automation, timed timelines, and recordings:

• no atMs anywhere → sequential (delayMs between steps)
• any step has atMs → timeline (steps fire at their offset from start)
• capture{} present → whole plan handed to the capture runner (records video; steps accepted as alias for its actions)

Top-level result: ok is true only when EVERY step passed; failedSteps carries the count — never trust ok alone without it. Run a JSON plan file:

{
  "app": "Genesis",
  "restore": true,
  "delayMs": 300,
  "steps": [
    { "do": "focus" },
    { "do": "press", "id": "settings-open" },
    { "do": "click", "desc": "Account", "role": "button" },
    { "do": "get", "desc": "Account", "role": "button" },
    { "do": "click", "subrole": "close", "window": "Settings" },
    { "do": "window" }
  ]
}

tools control run plan.json          # human output: ok/FAIL per step + total
tools control run plan.json --json   # machine output: full results array

Plan fields:

• app — default app for all steps (overridable per step with "app": "...")
• restore — snapshot mouse + focus before, restore after
• delayMs — pause between steps (default 200ms, overridable per step with "delay": N)
• exact — force strict role/subrole matching for all steps
• steps[].do — any command name (focus/click/press/set/type/get/find/attrs/actions/perform/window/scroll/hotkey/screenshot/wait/assert)
• Steps take the same fields as CLI flags: id, role, title, desc, subrole, window, value, text, action, keys, direction, amount, path
• wait/assert steps additionally take timeout, interval, gone, for, expect, contains

The runner is the declarative equivalent of the shell snapshot/restore pattern but in one call, with timing and error tracking per step.

Workflow: automate a native app form

# 1. Discover what's in the app
tools control list --app Genesis --depth 5

# 2. Find the text field (by role if no AXIdentifier)
tools control find --app Genesis --role AXTextField

# 3. Check what you can do with it
tools control actions --app Genesis --id auth-email

# 4. Fill the field
tools control set --app Genesis --id auth-email --value "[e-mail]"

# 5. Press the submit button
tools control press --app Genesis --id auth-continue

Workflow: find elements without AXIdentifier

Many apps don't set AXIdentifier on all elements. Use find to locate by role/title/value:

# Find all buttons
tools control find --app Safari --role AXButton

# Find element with specific text
tools control find --app TextEdit --role AXStaticText --value "Untitled"

# Find by title substring
tools control find --app "System Settings" --title "Wi-Fi"

Workflow: get window geometry (for screenshots/crops)

# Get exact window bounds for screenshot cropping
tools control window --app Genesis
# Returns: x, y, width, height, minimized, fullscreen per window

Recording: tools control capture (video + timed actions)

Everything above is single-shot element control. For multi-frame recording — capture a transition/animation while firing timed actions, diff-sampled frames, contact sheets, declarative crops, vitrinka publish — use the capture runner:

tools control capture preflight [--app "<Name>"]   # ALWAYS FIRST when writing a capture plan
tools control capture --help                       # full plan/actions contract
tools control capture plan.json 1>result.json 2>err.log
tools control capture plan.json --annotate draw-plan.json   # one-shot capture+draw: renders the
                                                            #   annotation plan onto every kept frame
                                                            #   -> <sessionDir>/annotated/ (result.annotated)

Read references/capture.md before writing any recording plan — it holds the full recording discipline (duration/fps parsing, capture-target resolution, timing model, crop markers, focus re-assertion, troubleshooting, anti-patterns). references/peekaboo.md has the underlying peekaboo flag tables. Recording requires the external peekaboo binary (homebrew); element control does not.

Capture-plan notes:

• In a capture{} plan run via tools control run, plain verbs are auto-mapped (press→ax-press, set→ax-set, perform→ax-perform) and id→axId; the capture runner also has EXTRA action types (click/hotkey/type/scroll/crop/screenshot markers — see capture --help) that plain plans don't.
• If the result carries capture.failed: true with "peekaboo produced NO output", the peekaboo BINARY is crashing (verify standalone: peekaboo capture live --mode screen --duration 2 --json; 3.0.0-beta3 did this — upgrading to 3.9.4 fixed it). Element control, screenshots, annotate, and OCR don't use that path and keep working.
• Plan focus{} uses ax-tool natively (peekaboo window focus hangs on some versions; the wrapper only falls back to it, then osascript, when ax-tool is unavailable).
• Capture preflight cross-checks its CGWindowList view against the AX API — windows only CGWindowList sees (other Spaces, stale entries) are marked axVisible: false and never picked as the crop basis.

Capture plans support three AX action types (same targeting as the CLI):

Action type	What it does	Needs ax-tool binary?
ax-set	Write a text field value	No (osascript fallback)
ax-press	Press a button/toggle	No (osascript fallback)
ax-perform	Any AX action (AXShowMenu, AXRaise, etc.)	Yes (no fallback)

The binary auto-builds on first tools control run. With it, all three use the fast path (~50-200ms); without it, ax-set/ax-press fall back to osascript (~2-5s) and ax-perform errors.

Plan authoring workflow (element discovery, then recording)

# 1. Discover elements with ax-tool
tools control list --app Genesis --depth 5           # see all elements
tools control find --app Genesis --role AXButton     # find buttons
tools control find --app Genesis --desc "Settings"   # find by description
tools control actions --app Genesis --id nav-chat    # what actions are available?

# 2. Get window bounds (same CG point system as click coords / relativeTo)
tools control window --app Genesis
# Returns: x, y, width, height — use for crop regions or relativeTo offsets

# 3. Write the plan using discovered identifiers
cat > /tmp/plan.json << 'EOF'
{
  "capture": { "mode": "screen", "screenIndex": 0, "duration": 5,
               "activeFps": 8, "noRemote": true, "captureEngine": "cg" },
  "focus": { "app": "Genesis" },
  "actions": [
    { "atMs": 500,  "do": "ax-set", "axId": "auth-email", "value": "[e-mail]", "app": "Genesis" },
    { "atMs": 1500, "do": "ax-press", "axId": "auth-continue", "app": "Genesis" },
    { "atMs": 3000, "do": "ax-perform", "axId": "theme-picker", "action": "AXShowMenu", "app": "Genesis" }
  ]
}
EOF

# 4. Run the capture
tools control capture /tmp/plan.json 1>/tmp/result.json 2>/tmp/capture.err

When to use element commands directly vs the capture runner

Scenario	Use
Fill a form, click a button (no recording needed)	tools control set / tools control press directly
Inspect element attributes, discover identifiers	tools control list / find / attrs / actions directly
Get window bounds for screenshot/crop planning	tools control window directly
Static click-through sequence with screenshots at each step	tools control run plan.json (~100ms/step, native screenshots)
Record UI transitions while interacting with native controls	tools control capture plan with ax-set/ax-press/ax-perform actions
Timed sequence (fill form, wait, press, record the result)	tools control capture — timing precision matters
Open a dropdown and record the menu appearing	tools control capture with ax-perform + AXShowMenu

tools control window as relativeTo source

tools control window returns window bounds in CG points — the exact coordinate system that relativeTo and click coords use. Use it to compute window-relative offsets for capture plans:

# Get Genesis window position
tools control window --app Genesis --json
# {"windows": [{"x": 49, "y": 810, "width": 900, "height": 452, ...}]}

# Use in a plan with relativeTo (offsets from window top-left):
{ "atMs": 500, "do": "click", "coords": {"x": 100, "y": 200},
  "relativeTo": {"app": "Genesis"} }

App name resolution

--app matches in order: exact localizedName, case-insensitive name, bundleIdentifier substring. Examples:

• --app Finder, --app Safari, --app Genesis
• --app "System Settings" (quote names with spaces)
• --app com.apple.finder (bundle ID substring)

Permissions

Requires Accessibility access for the calling terminal process. Grant in System Settings > Privacy & Security > Accessibility.

Common AX actions

Action	Description
AXPress	Click/activate (use press shortcut)
AXRaise	Bring window to front
AXShowMenu	Open context/dropdown menu
AXConfirm	Confirm dialog
AXCancel	Cancel dialog
AXIncrement / AXDecrement	Stepper/slider
AXPick	Select menu item