computer-use

# Computer Use Use this skill to automate the UI of Microsoft Windows apps. It automates apps via SendInput and UI Automation, and takes screenshots of app windows via Windows.Graphics.Capture that works even if they're occluded. If this plugin is listed as available in the session, treat that as mandatory reading before Windows automation work. Open and follow this skill before saying that Computer is unavailable and before falling back to other Windows automation methods. Before using this skill for the first time in the current conversation context, read the entire `SKILL.md` file in one read. Do not use a partial range such as `Get-Content .\SKILL.md | Select-Object -First 220`; read through the end of the file. Do not mention this internal skill-loading step to the user. ## Bootstrap These setup details are internal. User-facing progress updates should be less technical in nature. Never mention `Node REPL`, `node_repl`, `REPL`, JavaScript sessions, or module exports unless a user is asking for that exact information. If setup or recovery is needed, describe it naturally as connecting to Windows or retrying the Windows connection. The `computer-use-client` module is the core entry point for Computer Use, and is available under `scripts/computer-use-client.mjs` in this plugin's root directory. ALWAYS import it using an absolute path. It loads the bundled `cua_node` `@oai/sky` runtime internally; do not import `@oai/sky` directly from the JavaScript session. Do not spawn `codex-computer-use.exe`, search for the helper executable, or build a custom helper protocol client. App approvals and user interruption handling only work through `scripts/computer-use-client.mjs`. IMPORTANT: If this path cannot be found, stop and report that this plugin is missing `scripts/computer-use-client.mjs`. Then run setup code through the Node REPL `js` tool. In this environment the callable tool id typically appears as `mcp__node_repl__js`; `js_reset` only clears state and is not the execution tool. Run this once per fresh `node_repl` session: ```js if (!globalThis.sky) { const { setupComputerUseRuntime } = await import("<plugin root>/scripts/computer-use-client.mjs"); await setupComputerUseRuntime({ globals: globalThis }); } ``` ## Troubleshooting IMPORTANT: do NOT attempt to dig through source code or control Windows apps through unrelated mechanisms before attempting this workflow. If you run into issues, follow the steps below FIRST. - Do not fall back to PowerShell, shell scripts, SendKeys, or other foreground keyboard/mouse automation just because those tools are visible. Read and attempt this workflow first. - If `js_reset` is visible but `js` is not, do not conclude that `node_repl` is unusable. Use tool discovery for `node_repl js`, then `mcp__node_repl__js`, then `js`, then `node_repl js JavaScript execution`; run the bootstrap cell with the Node REPL `js` tool once it is exposed. - If the Node REPL `js` execution tool is still unavailable after those searches, say that explicitly before choosing any fallback Windows-control path. - If `node_repl` is not available, say that explicitly before choosing any fallback Windows-control path. On the first Computer Use task in a session, try a lightweight call after bootstrap: ```js globalThis.apps = await sky.list_apps(); nodeRepl.write(JSON.stringify(apps, null, 2)); ``` Any non-error response means the Windows helper is reachable. If `list_apps`, `list_windows`, or another lightweight request times out, wait 2 seconds and retry the same lightweight call once. If the retry succeeds, continue from the returned apps. If bootstrap reports `Module not found: @oai/sky`, report the exact setup error and plugin path. Do not add `<plugin root>/node_modules` or try to import `@oai/sky` directly from the JavaScript session. If Computer Use reports that the turn ended, that the user stopped Computer Use, or that it is unavailable for the current turn, stop the task and report that Computer Use was stopped or became unavailable. Do not fall back to foreground keyboard/mouse automation such as PowerShell `SendKeys`. If the same lightweight call times out again, do not keep issuing app input. Reset the JavaScript session if available, rerun the bootstrap cell, and retry `list_apps` once. If it still times out or reports helper communication failure, stop and report that the Windows Computer Use helper may have crashed. If the intended app is present but has no suitable open window, call `await sky.launch_app({ app: targetApp.id })`, then poll `list_apps()` until that app exposes a targetable window. If the intended app is not yet discoverable in `list_apps()` call `await sky.launch_app({ app: "C:\\path\\to\\YourApp.exe" })` or use an equivalent `.exe` process identifier, then poll `list_apps()` or `list_windows()` for the new targetable window. Do not open or navigate the Windows Start menu/Search UI to launch apps. Do not continue while a launcher, splash screen, modal, or permission prompt is blocking the app's workspace. ## Runtime Behavior - Computer Use commands run through the Node REPL `js` tool. Do not look for a separate computer-use-specific JavaScript tool. - Reuse the existing `sky`, `apps`, `targetApp`, `targetWindow`, and `state` bindings across cells. If `targetWindow` already exists, keep using it until a stale handle, activation failure, or missing window error requires recovery. - Store cross-cell values on `globalThis`. The JavaScript session is persistent: top-level `const` and `let` names cannot be redeclared by later retries. Do not declare retry-prone scratch names such as `tree`, `lines`, `state`, or `accessibility` at top level. Use `globalThis` for state you need later, and wrap temporary parsing code in a short `{ ... }` block or use fresh names for one-off retries. - On the first cell, initialize `sky`, list installed apps, and print the returned app objects. Each app includes its currently open targetable windows. - Choose one app from the latest `apps` array. If it has exactly one suitable open window, call `get_window` on that returned window before the first snapshot. This is the Computer Use equivalent of resolving the chosen target into the current canonical object. - For app-control tasks, call `activate_window({ window: targetWindow })` once after selecting the target and before the first snapshot. Activation is idempotent, and restores minimized windows. Skip this only when the task is explicitly passive inspection of multiple windows without stealing focus. - Use `list_windows` as a shortcut only when the task is explicitly about currently open windows or when recovering after you already know the app is running. - After `get_window_state`, replace `targetWindow` with `state.window`; it is the canonical window object that was actually captured. - If bindings still exist after a stale handle error, recover with `sky.get_window({ id: targetWindow.id, app: targetWindow.app })`. If bindings are gone after a reset, call `list_apps` again and choose from the fresh returned objects. Do not reconstruct a window from guessed ids. ### First Computer Use Cell ```js if (!globalThis.sky) { const { setupComputerUseRuntime } = await import("<plugin root>/scripts/computer-use-client.mjs"); await setupComputerUseRuntime({ globals: globalThis }); } globalThis.apps = await sky.list_apps(); nodeRepl.write(JSON.stringify(apps, null, 2)); ``` After that, keep using the existing `targetApp` and `targetWindow` bindings. Do not alternate between `targetWindow`, `window`, `taskWindow`, `targetWindowId`, and `targetWindowApp` across retries. GOOD: choose one returned app, then choose one of its returned windows: ```js globalThis.targetAppMatches = apps.filter((candidate) => /replace-with-app-name-or-id/i.test(`${candidate.id} ${candidate.displayName ?? ""}`), ); if (targetAppMatches.length !== 1) { nodeRepl.write(JSON.stringify(targetAppMatches.length ? targetAppMatches : apps, null, 2)); throw new Error("Expected exactly one matching app; refresh apps or refine the pattern"); } globalThis.targetApp = targetAppMatches[0]; if (targetApp.windows.length !== 1) { nodeRepl.write(JSON.stringify(targetApp.windows, null, 2)); throw new Error( "Expected exactly one target window; call launch_app or refine the window choice", ); } globalThis.targetWindow = await sky.get_window(targetApp.windows[0]); await sky.activate_window({ window: targetWindow }); globalThis.targetWindow = await sky.get_window({ id: targetWindow.id, app: targetWindow.app }); globalThis.state = await sky.get_window_state({ window: targetWindow }); globalThis.targetWindow = state.window; ``` GOOD: if the chosen app is installed but has no returned window yet, launch it by id and poll `list_apps()` for its window: ```js await sky.launch_app({ app: targetApp.id }); for (let attempt = 0; attempt < 10; attempt += 1) { await new Promise((resolve) => setTimeout(resolve, 1000)); globalThis.apps = await sky.list_apps(); globalThis.targetApp = apps.find((candidate) => candidate.id === targetApp.id); if (targetApp?.windows?.length) break; } if (!targetApp?.windows?.length) { const label = targetAppMatches[0].displayName ?? targetAppMatches[0].id; throw new Error(`Launched ${label}, but no targetable window appeared`); } globalThis.targetWindow = await sky.get_window(targetApp.windows[0]); ``` GOOD: if the app is a local `.exe` build and is not returned by `list_apps()` yet, launch it by `.exe` path and poll for the resulting window: ```js await sky.launch_app({ app: String.raw`C:\work\MyApp\bin\Debug\MyApp.exe` }); for (let attempt = 0; attempt < 10; attempt += 1) { await new Promise((resolve) => setTimeout(resolve, 1000)); globalThis.apps = await sky.list_apps(); globalThis.targetAppMatches = apps.filter((candidate) => /MyApp(?:\.exe)?/i.test(`${candidate.id} ${candidate.displayName ?? ""}`), ); if (targetAppMatches.some((candidate) => candidate.windows?.length)) break; } globalThis.targetApp = targetAppMatches.find((candidate) => candidate.windows?.length); if (!targetApp?.windows?.length) { globalThis.windows = await sky.list_windows(); nodeRepl.write(JSON.stringify({ apps: targetAppMatches, windows }, null, 2)); throw new Error("Launched MyApp.exe, but no targetable window appeared"); } globalThis.targetWindow = await sky.get_window(targetApp.windows[0]); ``` GOOD: if the app has multiple windows, choose from that app's returned windows: ```js globalThis.targetWindowMatches = targetApp.windows.filter((candidate) => /replace-with-window-title/i.test(candidate.title ?? ""), ); if (targetWindowMatches.length !== 1) { nodeRepl.write( JSON.stringify(targetWindowMatches.length ? targetWindowMatches : targetApp.windows, null, 2), ); throw new Error("Expected exactly one matching window; refine the title pattern"); } globalThis.targetWindow = await sky.get_window(targetWindowMatches[0]); await sky.activate_window({ window: targetWindow }); globalThis.targetWindow = await sky.get_window({ id: targetWindow.id, app: targetWindow.app }); ``` GOOD: request accessibility text only when it will drive the next action, then narrow it in JS before printing: ```js { const snapshotState = await sky.get_window_state({ window: targetWindow, include_screenshot: false, include_text: true, }); globalThis.state = snapshotState; globalThis.targetWindow = snapshotState.window; } ``` GOOD: when `include_text: true` returns a large tree, print the structured critical fields first, then filter the indexed element tree: ```js { const snapshotAccessibility = state.accessibility; if (!snapshotAccessibility) { throw new Error("No accessibility state returned"); } const pattern = /replace-with-relevant-labels-or-words/i; const treeLines = snapshotAccessibility.tree.split("\n"); const candidates = treeLines.filter((text) => pattern.test(text)).slice(0, 80); const criticalContext = { focused_element: snapshotAccessibility.focused_element, selected_text: snapshotAccessibility.selected_text, selected_elements: snapshotAccessibility.selected_elements, document_text: snapshotAccessibility.document_text, }; nodeRepl.write( [ JSON.stringify(criticalContext, null, 2), "Candidate elements:", ...(candidates.length ? candidates : treeLines.slice(0, 80)), ].join("\n"), ); } ``` BAD: guessing or reconstructing a window instead of using one returned by `list_apps`, `list_windows`, `get_window`, or `get_window_state`: ```js await sky.click({ window: { id: 123456, app: "example.exe" }, x: 400, y: 300 }); ``` GOOD: batch related actions against the selected window, then verify once: ```js await sky.click({ window: targetWindow, x: 400, y: 300 }); // replace with stable window-relative coordinates from the snapshot await sky.type_text({ window: targetWindow, text: "hello" }); await sky.press_key({ window: targetWindow, key: "Return" }); globalThis.state = await sky.get_window_state({ window: targetWindow }); globalThis.targetWindow = state.window; ``` GOOD: after a stale handle error, rehydrate from the current `targetWindow` object: ```js globalThis.targetWindow = await sky.get_window({ id: targetWindow.id, app: targetWindow.app }); ``` GOOD: after a reset or lost binding, list apps again and choose from the fresh returned objects: ```js globalThis.apps = await sky.list_apps(); nodeRepl.write(JSON.stringify(apps, null, 2)); throw new Error("Choose the target app and window from the fresh apps list before acting"); ``` GOOD: for canvas/hotkey apps, focus the work surface, clear modal state, then batch stable coordinate/key actions: ```js await sky.click({ window: targetWindow, x: 400, y: 300 }); // replace with a stable work-surface point from the snapshot await sky.press_key({ window: targetWindow, key: "Escape" }); await sky.press_key({ window: targetWindow, key: "Escape" }); await sky.press_key({ window: targetWindow, key: "KP_0" }); // use numpad keysyms when the app distinguishes them globalThis.state = await sky.get_window_state({ window: targetWindow }); globalThis.targetWindow = state.window; ``` ## Guidelines - Launch apps with `await sky.launch_app({ app: targetApp.id })` when `list_apps` returns the intended app. If the app is not yet discoverable in `list_apps` use an explicit `.exe` path or `.exe` process identifier instead. - Start automating Windows apps by finding the app with `list_apps`, then selecting one of its open windows. - `get_window_state` does not activate or focus the window, so it can be used to inspect multiple windows without stealing focus. Input methods automatically activate their target window first and fail if activation fails. Use `activate_window` only when you explicitly need to bring a window foreground without taking an input action. - Use `list_apps` for default app discovery, app identity, launch candidates, running state, usage metadata, and each app's open windows. Prefer the returned `list_apps` id as the app identifier whenever a suitable candidate is available, even if the app is not currently running. - Use `list_windows` only when the task is explicitly about currently open windows or when you already know the target app is running and need a fresh flat window list. - Occluded windows can be snapshotted without activation. Minimized windows may be listed, but Windows.Graphics.Capture does not capture them reliably while minimized. Input methods activate and restore their target automatically. If a passive snapshot fails after starting from a minimized window, call `activate_window({ window })`, refresh the object with `get_window({ id, app })`, and retry once. - If the intended app is present but has no suitable open window, call `launch_app({ app: targetApp.id })`, then poll `list_apps()` until the app exposes a targetable window. If the app is not yet in `list_apps`, launch it with an explicit `.exe` path or `.exe` process identifier, then poll `list_apps()` or `list_windows()` for the resulting targetable window. If the window never appears, report the exact launch or polling failure. Do not open or navigate the Windows Start menu/Search UI to launch apps, and do not use PowerShell or `Start-Process` as the normal app launch path. - `get_window_state` is an expensive point-in-time snapshot, not a live view. Use it to reason over, then batch related actions without re-snapshotting between every input. - After `get_window_state`, use the returned `state.window` for later actions; it is the canonical window object that was actually captured. - After a kernel reset, stale handle, or lost window binding, recover a current window object with `sky.get_window({ id, app })` using an id and app from an earlier returned `Window`. - By default, `get_window_state({ window })` captures and automatically displays a screenshot, and returns `accessibility: null`. This is the best default for desktop apps with weak accessibility trees. - If you need accessibility text or element indexes, call `get_window_state({ window, include_screenshot: false, include_text: true })`. Request both only when you truly need both the screenshot and accessibility text for the next decision. - Accessibility text is returned as `state.accessibility.tree`. The tree format is: first line `Window: "...", App: ...`, then indexed element tree lines, then at most one critical tail block: `Selected text`, `Selected`, `Document text`, or `The focused UI element is ...`.