computer-use

# Computer Use — driving native desktop apps You have MCP tools that read and operate the **user's real applications** through the operating system's accessibility layer. This is not a browser: use it when the work lives in a desktop app (a spreadsheet, a PDF viewer, a native internal tool, a dialog box), and use `playwright-cli` (the `web-browse` skill) for web pages. Two things to internalise before your first call: - **Address elements by index, from a snapshot you were just shown.** That is the path to prefer for everything: it activates the control directly, it is checked against drift, and the mouse pointer does not move. Coordinates exist as a fallback for canvases, maps and custom-drawn UI that expose no element — see [Coordinates and dragging](#coordinates-and-dragging-the-fallback-not-the-default). - **It is off unless the user turned it on** (Settings → Computer Use). The full tool set runs on macOS and Windows alike. They differ in ONE way worth relaying: Windows has no per-process input, so a keystroke takes the user's keyboard focus and a coordinate click moves their real cursor. A "disabled" or "not supported" refusal is a real configuration answer, not a transient error — relay it and stop; do not retry. ## The loop **1. Find the app.** ``` computer_list_apps() ``` Returns the on-screen applications with their bundle ids, pids and window titles. Skip this if the user named an app you can pass straight through — `app` accepts a display name (`"Finder"`, `"Preview"`) or a bundle id (`"com.apple.finder"`), matched case-insensitively. **If the app you need is not running, open it:** ``` computer_launch_app(app="Paint") ``` Give the app's NAME as the OS knows it — the name in the Start menu or the Applications folder. **A filesystem path, a command line and a document are all refused**, because this opens an application and nothing else; if you find yourself wanting to pass a path, the answer is to launch the app and then use the tools below to open the file from inside it. It returns the new window's element tree, so you can act immediately without a separate `computer_get_state`. Two things to remember: - **A cold start can take ten seconds.** The result says how long it took. Do NOT call it twice for one app — a second call is how you end up with two copies and two unsaved documents. - **If the app already has a window it is refused**, naming that window. That is not a failure: call `computer_get_state` on it instead. **2. Snapshot the window. Do this FIRST, every turn — with a screenshot.** ``` computer_get_state(app="Finder", screenshot=True) ``` Request a screenshot on the first snapshot of each turn: do not pass `screenshot=false` while the user is watching. The capture opens the floating live view. You receive a file path, not an image; keep using the outline unless the pixels are needed. Optional: `text_limit` (per-element text cap), `max_tree_nodes`, `max_tree_depth`, `screenshot` (bool). You get a numbered outline: ``` App=com.apple.finder (pid 1041) Window: "Documents", App: Finder. 0 window "Documents" @ x=0,y=0 900x600 1 splitgroup 2 scrollarea 3 button "Back" [AXPress] @ x=18,y=12 28x24 4 textfield "report" (editable) <focused> @ x=120,y=52 300x24 5 row "Q3 numbers.xlsx" (selected) @ x=8,y=90 880x20 7 textfield <secure> Window origin on screen: x=220,y=118 (900x600). Element frames above are relative to it — add the origin for a screen point. Focus: element 4 (AXTextField "report"). Selected text: [Q3] ``` Reading one line: the number at the start is the `element_index` you pass to every action. `[AXPress]` and friends are the actions that element advertises. The indentation is containment, so you can tell a toolbar button from a table cell. Then, in order: - **`(editable)`, `(selected)`, `(expanded)`, `(disabled)`** — state the role cannot carry. `(editable)` is the one to check before typing: a read-only field looks identical to a writable one otherwise, and typing into it succeeds while the text goes nowhere. If a text field has no `(editable)`, it will not accept input — find the one that does instead of retrying. - **`<focused>`** — the caret is here. "Type this in" usually means this element, so pass ITS index — there is no indexless form (see the note under the action table). - **`@ x=…,y=… WxH`** — position and size, **relative to the window**, in pixels. Absent when the element exposes no geometry (ordinary) or the window rect could not be read. The trailing lines: the **window origin** is what converts a frame to a screen point (`computer_click(x=…, y=…)` takes SCREEN coordinates — add the origin, do not pass a frame straight through); **Focus** names the focused element; and **Selected text** is what the user has highlighted, which is what a request like "rewrite what I selected" refers to. A `<secure>` element shows no title, no value, no traits and no frame — only that it exists. **3. Act by index.** | Tool | Use it for | |---|---| | `computer_click(app, element_index)` | press a button, checkbox, menu item, link, row | | `computer_type_text(app, element_index, text)` | type text into that element. `element_index` is **required** — see the note below | | `computer_set_value(app, element_index, value)` | replace a field's whole contents in one step | | `computer_press_key(app, element_index, key)` | a key or chord — `"return"`, `"tab"`, `"escape"`, `"cmd+s"`, `"cmd+shift+a"`. Paste (`cmd+v`) is allowed, but you cannot see the clipboard — when you know the text, prefer `computer_type_text` with the literal text | | `computer_scroll(app, element_index, direction, pages?)` | scroll a scrollable area (`up`/`down`/`left`/`right`) | | `computer_perform_action(app, element_index, action)` | run one of the element's own advertised actions when nothing above fits | | `computer_click(app, x, y)` | click a point when the target has no element — see below | | `computer_drag(app, from_x, from_y, to_x, to_y)` | a slider sweep, a range selection, a reorder — and, with `steps`, a canvas stroke | | `computer_launch_app(app)` | open an app that is not running yet. Name only, never a path | **Every element tool needs an `element_index`, and the keyboard tools are the ones to remember.** There is no "type into whatever is focused" form: an unnamed target has no role or subrole, so the secure-field check cannot inspect it, and an indexless keystroke would land in a focused password box. That applies to `computer_press_key` too — `press_key("tab")` can *move* focus onto a password field, and the next keystroke would go there. So name the field you mean; if you want to tab through a form, address each field by index instead of tabbing blind. `computer_click` is the one element tool that also takes coordinates; `computer_drag` is coordinate-only and `computer_launch_app` takes only a name. Every action returns a **refreshed** tree, so after a click you already have the new indices — do not call `computer_get_state` again just to re-read them. The same goes for window position and size: they are in the snapshot header you were already shown. **Re-probing something the last response already told you is the most common wasted turn.** **Refresh the user's view as you go.** Action results carry the structure but no pixels, so the live-view panel freezes on your last screenshot while you work. After a step that visibly changes the screen — a window opened, a dialog appeared, a file saved, a page navigated — call `computer_get_state(app=…, screenshot=True)` once to push a fresh frame. Use judgement rather than doing it after every keystroke: typing five fields is ONE visible change, not five, and each screenshot costs time and tokens. The test is "would the user see something different now?", not "did I just call a tool?". ### Coordinates and dragging: the fallback, not the default `computer_click` takes **either** `element_index` **or** both `x` and `y` — never both and never neither; supplying both is refused, because the two name different targets and there is no rule for which should win. Reach for coordinates only when the outline has nothing to address: a drawing canvas, a map, a timeline, a chart, a custom-drawn control. An element index is better whenever one exists — it is verified against UI drift, and a coordinate is delivered to whatever happens to be at that point *now*, so a window that reflowed after your snapshot will take the click somewhere you did not mean. Re-snapshot before a coordinate click if anything has changed. Optional on both: `mouse_button` (`left`/`right`/`middle`), `click_count` (1-3 for single/double/triple), and `click_method`: - **`auto`** (the default) — element index → the accessibility press; coordinates → an app-scoped mouse event. Correct almost always; do not override it without a reason. `auto` will **never** pick the pointer-moving path, so leaving it alone is always the pointer-safe choice. - **`app_post`** — send the click to the target app at that point *without moving the user's mouse*. This is what makes clicking a background window safe. - **`accessibility`** — force the press path; requires `element_index`. - **`global`** — **moves the user's real mouse pointer** and clicks there. You have to ask for it by name; nothing resolves to it implicitly — that naming requirement is the only thing between an ordinary click and the user's cursor. Every use is separately recorded in the audit log. Ask for it only when a click has to be physically real (a Dock item, a menu-bar extra, UI that ignores posted events) — say so in your reply *before* you use it, because the user's cursor will jump out from under their hand, and never use it as a first attempt or a retry after an ordinary click failed for some other reason. If it is refused, do not retry: use `app_post` or find another route. - **`sky_click`** — clicks a window that is **behind other windows**, without raising it and without moving the pointer. Ask for it by name when `app_post` reached the app but nothing happened. That is the signature of an app whose renderer does its own hit-testing and so ignores a posted click while another window is in front — browser-based and iPad-ported apps behave this way (Chrome, VS Code, Slack, Freeform are the ones you will meet). It is the one method built on a private Apple API, so it can stop working on a future macOS — when it is unavailable the refusal says so and names `app_post`. Do not reach for it first; reach for it when a covered window is the reason a click did nothing. It is a **left-button recipe only**: pairing it with `mouse_button: right` or `middle` is refused, and the refusal names `app_post` as the route for those, since that reaches the same application through a public API. `computer_drag` is coordinate-only — no accessibility action expresses a sweep between two points — and it takes the same `mouse_button` / `click_method` options. **On Windows a drag always moves the operator's real cursor**, so it must be asked for by name: pass `click_method: "global"`, because the default is the macOS app-scoped route and is refused rather than quietly substituted. ### Drawing: a drag needs `steps` to be a stroke A drag's default shape is a plain two-point sweep. That is correct for a slider, a range selection or a reorder — and it **cannot draw**, because an application samples the pointer as it moves, so a two-point drag can only ever produce a straight line no matter what you meant. To draw, say how many points the path visits: ``` computer_drag(app="Paint", from_x=500, from_y=400, to_x=800, to_y=400, steps=48, path="curved", click_method="global") ``` - **`steps`** (1-512, default 1) — segments the path is divided into. 32-64 draws a smooth curve. More is not better: each point costs real time with the mouse button held down. - **`path`** — `"straight"` (default) interpolates the line; `"curved"` bows it sideways, which is what a hand-drawn stroke looks like. Only meaningful with `steps > 1`. An unknown value is refused rather than straightened, because a straightened stroke is a wrong drawing. Two things that will cost you a turn otherwise: - **Select the tool by `element_index` first, not by clicking pixels.** A drawing app's pencil/brush/shape buttons are ordinary addressable controls, so `computer_click(app=…, element_index=…)` selects them with no pointer movement at all. Only the canvas itself needs coordinates. - **A curved stroke near a window edge can be refused.** The bow travels away from the straight line between your two points, and every point of the path has to be inside the target window — so a stroke that starts and ends inside can still be refused if the curve would leave it. Use `path: "straight"`, or move the endpoints inward. ### Clicking inside a canvas: converting image pixels When a surface has no addressable element — a canvas, a map, a chart — the screenshot note carries the conversion you need: ``` Image-pixel to screen-point conversion: screen_x = 200 + image_x / 0.500, screen_y = 100 + image_y / 0.500 (the image is 0.500x the window's 1600x900 pixels). ``` Read a position off the image, apply that, and you have a screen point for `computer_click(x=…, y=…)` or `computer_drag`. Use it **only** when there is no element to address: an element's own frame is exact, while this is a measurement off a downscaled image and carries its rounding. If the line is absent, the window's geometry could not be read — fall back to element frames rather than guessing a ratio. **4. Release when you are finished with the app.** ``` computer_end_turn() ``` Drops the cached snapshots. Call it when the desktop part of the task is done. It is cheap and it prevents a stale-index refusal later in the conversation. ## When something does not work: change the MECHANISM, not the arguments This is the rule that separates a two-call fix from a twenty-call loop. When an action fails or nothing visibly changed, ask yourself one question before the next call: > **Am I varying arguments on the same mechanism, or switching to a genuinely > different one?** Nudging a coordinate by 20px, retrying the same `element_index`, or re-issuing the same `click_method` are all the SAME mechanism. **Two failures on one mechanism is the signal to switch — not a reason to try a third variation.** The ladder, in order. Go down one rung per failure; never repeat a rung: 1. **`element_index`** on the control itself (the default, and right ~90% of the time) 2. **A different element** — the row or cell that CONTAINS your target, or the control's parent; icon-only buttons often only respond one level up 3. **`computer_perform_action`** with an action the element actually advertises — read the `[AXPress]`-style list in the outline rather than assuming 4. **Keyboard** — `computer_press_key`. Menus, dropdowns, date pickers and scrollbars are frequently keyboard-reachable when they are click-hostile 5. **Coordinates** — `x`/`y` from the element's own position, not from the screenshot 6. **`click_method: "sky_click"`** if the window is covered by another window 7. **Stop and tell the user what you tried.** Two sentences, naming the rungs. That is a better outcome than a twentieth call **Never** re-run a rung that already failed, and never escalate preemptively — do not reason "this is Electron, so clicking will fail, so I will start at coordinates." React to an observed failure, do not predict one. ## Do not report success you have not observed An action that returned without an error is **not** proof it landed. The refreshed tree that comes back with every action is your evidence — read it and name what changed: a new value, a dialog that appeared, a menu that closed, a button that became disabled. - **If nothing in the tree changed, the action probably did nothing.** Say so and go down the ladder. Do not report success. - **A target that VANISHED is usually success, not failure.** A button that is gone after you clicked it, a dialog that closed, a row that disappeared after delete — the element being unfindable is the expected outcome. Do not "retry" it. - Do not go looking for evidence that cannot exist: the Cursor Motion overlay is invisible to screenshots, so its absence proves nothing. The most common failure in desktop automation is reporting success on a silently dropped action. The second most common is retrying an action that already worked. ## Prefer the outline; read the screenshot only if you must The tree is the primary channel and it is usually sufficient. When a screenshot is attached you get a **path**, not an image: ``` Screenshot: /var/folders/.../kirocrew-computer-shots/shot-1769472013411.jpeg (1280x604 jpeg, 24.2 KB) — read it with the fs_read tool only if the tree is insufficient. ``` Read the file only when the outline cannot answer the question: a chart, rendered document, layout problem, or unexposed control. Reading costs roughly 8,000 tokens; capture alone is cheap. If the user asked "show me", give them the path to render. Use `screenshot=false` only for unwatched, structure-only work. ## Reading the refusals correctly These are **answers**, not failures. Relay them and adapt; do not loop. | You see | What it means | What to do | |---|---|---| | a value ending in `…` | the text was cut at `text_limit`, not truncated by the app | re-snapshot with a larger `text_limit`; do NOT read the screenshot to recover it, and do not tell the user the content is missing | | `[tree truncated at N nodes]` | the window has more controls than the budget | raise `max_tree_nodes`, or scroll to bring your target into range — the rest of the window is real, you just have not been shown it | | `Screenshot suppressed: the accessibility tree was truncated …` | a cut-off walk cannot prove the window holds no password field, so no pixels were captured. Routine for a browser or an Electron app at the default budget | if you actually need the image, re-snapshot with a higher `max_tree_nodes` / `max_tree_depth`; otherwise work from the tree and do not re-request the screenshot | | `no state for 'X'. Call computer_get_state first.` | you acted without a snapshot | snapshot, then act | | `state for 'X' is 214s old. Call computer_get_state again.` | the snapshot expired (90s) | snapshot again | | `element_index 7 changed since the last computer_get_state (was 'AXButton "Save"', now 'AXButton "Delete"')` | the UI moved under you — this refusal is what stopped you clicking the wrong thing | snapshot again and re-locate the element by its label, not its old number | | `'…' is a blocked target for computer use (…)` | KiroCrew's own dashboard is permanently refused — driving it would let you change your own security settings | do the task another way; tell the user why | | `refusing to type this text into 'X': …` | the text looked like a sensitive command or credential | do not rephrase to get around it; explain and stop | | `refusing to send input to a secure (password) field …` | the target is a password field | ask the user to type it themselves | | `computer use is disabled …` | the primary switch is off | tell the user to enable it in Settings → Computer Use; do not retry | | `computer use is not supported on this platform (…)` | no driver for this OS (e.g. Linux) | say so once | | `click_method 'app_post' is macOS-only …` / `'sky_click' is macOS-only …` | a macOS-only click method on Windows | use an `element_index` (no pointer moves), or name `click_method: "global"` to accept the cursor move | | `click_method 'sky_click' supports only the left mouse button (…)` | `sky_click` is a left-button-only recipe | use `left`, or switch to `app_post` for a right/middle click — the refusal names it |