agent-desktop

# agent-desktop Use this skill when the task involves the host macOS desktop rather than repository files, shell output, or browser-only verification. This project uses `agent-desktop` on the host. Prefer the built-in `Computer` sub-agent for these tasks instead of trying to drive the desktop with plain shell commands. ## When to use it - The user wants to inspect or interact with a native macOS application. - The task needs app launch, window focus, accessibility snapshots, semantic element refs, clicks, typing, keypresses, or scrolling. - A browser-only workflow is not enough, or the target is not a web page. ## Requirements - `agent-desktop` must be installed and its native binary available. - On macOS, the terminal app running `grok` needs Accessibility permission. - Refs from `computer_snapshot` are only valid until the next snapshot. ## Preferred flow 1. Delegate to `task` with `agent: "computer"` unless the current agent already has the `computer_*` tools and the task is tiny. 2. Use `computer_launch`, `computer_list_windows`, or `computer_focus_window` to get the target app/window ready. 3. Start with `computer_snapshot`. 4. Pick one ref-based action. 5. After any UI transition, snapshot again before reusing refs. ## Tool guidance - `computer_snapshot`: primary observation tool; prefer `interactive_only`. - `computer_click`: use `ref` values from the latest snapshot whenever possible. - `computer_type`: requires a target ref and is preferred over coordinate typing. - `computer_press`: use for shortcuts and special keys such as `Enter`, `Tab`, `cmd+space`, or `cmd+k`. - `computer_scroll`: pass an element ref, not raw coordinates. - `computer_get`: read text, value, bounds, role, or states from a ref. - `computer_screenshot`: use for visual confirmation or when the accessibility tree is not enough. ## Reliability rules - Use a snapshot -> act -> snapshot loop. - Prefer refs over coordinates. - Refs go stale after the next snapshot; do not reuse them blindly. - Keep actions reversible and low risk unless the user explicitly asked for something destructive. - If the accessibility tree is poor or ambiguous, use `computer_screenshot` for confirmation and explain the limitation. ## Blockers Stop and report clearly if: - Accessibility permission is missing. - `agent-desktop` is unavailable or its native binary is missing. - The target app/window cannot be found. - Refs are stale or ambiguous after a UI transition. - The requested action could be destructive and the user did not explicitly ask for it.