<picture>
<source media="(prefers-color-scheme: dark)" srcset="assets/icon-dark.svg">
<img src="assets/icon-light.svg" width="80" height="80" alt="Codewhale whale and pointer icon">
</picture>
# Computer Use
**By Codewhale · macOS beta (notarized app and source) · Windows experimental unsigned preview · Linux experimental: X11 verified on a virtual desktop (Xvfb + openbox) and in the Docker test image, from source; Wayland input on sway (verified on headless sway), other Wayland desktops observation only**
The [0.16.1 release](https://github.com/codewhale-hq/codewhale-cu-plugin/releases/tag/v0.16.1)
provides the notarized universal Mac app and an **unsigned Windows x64 preview**.
The Windows ZIP bundles Node and includes setup instructions; it is intended
for testing, with physical-device, mixed-DPI and signing acceptance still open.
Let Codewhale see and operate your apps — through whichever interface fits
each step: app scripting (AppleScript/JXA) where apps ship a dictionary,
accessible controls, text entry, click, scroll and capture through the same
MCP tools. The macOS helper keeps permissions and human controls in one
menu-bar app.
- **Set up once.** See Accessibility and Screen Recording status, open the
right Settings pane, then run a check in a disposable practice window.
- **Keep working.** macOS selects apps in background mode by default and
never moves your cursor. Accessibility actions and process-directed typing
touch nothing you are using. A pixel click, drag, wheel or modified key is
posted to the target app's process, so your app keeps the front and your
keyboard. Since v0.14.0 a native agent releases a background menu's
keyboard grab after about 50 ms on the measured Mac. There is no per-app prompt
in the default standard approval mode; apps you deny stay off limits, and
human-input yielding and Pause/Stop remain.
- **Stay in control.** See selected apps and their input modes. 0.13.0
adds a compact Codewhale controller while an app is in use. Pause cancels
queued work and releases held input; Stop ends existing sessions. Only the
person using the menu-bar controls can allow input again.
- **Update deliberately.** The app checks for stable releases on request.
Updates verify the digest, Codewhale signature and Apple notarization before
replacing the app, and retain the previous install for rollback.
[Setup page](https://codewhale.net/computer-use) ·
[Setup and troubleshooting](docs/TROUBLESHOOTING.md) ·
[Release notes](CHANGELOG.md) · [Distribution](docs/DISTRIBUTION.md) ·
[Background demo](docs/DEMO.md) · [Contributing](CONTRIBUTING.md) ·
[Security](SECURITY.md)
## Install
Install the helper and connect your host, or build from source:
1. **Mac app (easiest).** Download the disk image from the latest stable
[GitHub release](https://github.com/codewhale-hq/codewhale-cu-plugin/releases),
drag it to Applications, and open it. Open **Computer Use…** from the
whale menu-bar icon, grant what it asks (Accessibility, then Screen
Recording), and run the background check. No terminal, no Node, no
compiler needed.
2. **Inside Codewhale.** Install the Mac app above, then use the
[Codewhale setup](#codewhale-cli-setup) for your installed CLI version.
The published CLI v0.10.2 has an older built-in plugin; installing the
marketplace entry does not replace it with this release. A helper app,
plugin bundle and CLI can have different versions.
3. **From source.** Clone, test, install:
```bash
git clone https://github.com/codewhale-hq/codewhale-cu-plugin
cd codewhale-cu-plugin
npm test # unit + protocol, no GUI input performed
npm run build:app # dist/{macos,linux,windows}
npm run install:app # puts the app in place, registers it, opens it once
```
Needs Node 20+ and, on a Mac, Xcode Command Line Tools. Then wire your
host from the table in [Developer quick start](#developer-quick-start).
**Verify.** Ask your host for `request_access`. It identifies the
computer, helper/version, available tools and missing permissions without
sending input. `via: "app"` means grants belong to the Computer Use app;
`via: "direct"` means they belong to the hosting app or terminal. Follow that
receipt's `appHint`. macOS requires the separately installed helper; direct
mode is an explicit developer override. A missing or disconnected helper is
never permission to bypass its controls. Permission recovery is in
[Setup and troubleshooting](docs/TROUBLESHOOTING.md).
**Standard approval (since 0.13.0).** On a Mac, the default is
`standard`, modelled on cua-driver's promptless default: once you have granted
macOS Accessibility and Screen Recording to the helper, your agent can read and
operate any app you have not denied. There is no trust panel, no Touch ID step
and no per-app question. What stays yours: those two macOS grants, **Pause**
and **Stop**, an app you deny, the exact-script approval for `app_script`, and
the confirmation before an action that pays, buys, sends, transfers or deletes.
`request_access` and `consent {action:"status"}` report `approval:"standard"`.
Write `{"mode":"strict"}` to `~/.codewhale-cu/approval.json` to restore the
earlier prompts: the first request opens the helper's native panel (**Trust
until revoked…**, confirmed with Touch ID or your Mac password), and unapproved
apps refuse `consent_required`. That file is the switch both the helper app and
the MCP server read; `CODEWHALE_CU_APPROVAL=strict` also tightens whichever
process it reaches, but a helper opened from the Dock or by `open` never sees
your shell's environment. Strict is the only behavior on Windows, Linux and
non-local computers, and a file that cannot be parsed selects strict. The file
is an ordinary preference: any program running as you can rewrite it, so strict
restores the questions, not protection from a hostile local program.
Both an updated MCP plugin and an updated helper are needed for standard mode,
and Codewhale must load the 0.13 plugin rather than an older built-in copy.
Older clients can retain their own consent cards or per-app gates.
**Choose your surface.** Apps on this Mac use Computer Use; your signed-in Chrome
uses [Codewhale for Chrome](https://github.com/codewhale-hq/codewhale-plugin-marketplace/tree/main/plugins/chromewhale);
a clean browser uses Computer Use's `browser` tool. `/computer status` checks
the first route; Chrome has its own connector and per-site grants.
**Operating guidance.** The server sends its concise packaged skill in standard
MCP `initialize.instructions`. Hosts that consume that field can supply it
before the first action. `resources/list` / `resources/read` expose the exact
same guide plus detailed operations, refusal recovery and recording references.
This supplies guidance; host-side delivery/logging and approvals remain the
host's responsibility. Engine support is maintained separately.
**Capture identity.** Current source returns `raster_id` on screenshot, zoom
and OCR receipts. Repeat it in pixel targets and parent zooms; newer captures
refuse older pins before input. Nested zooms crop the bound parent file. The
packaged guide teaches this through the existing MCP instructions/resources.
This does not establish that the UI stayed unchanged or that the current
source shipped in the installed helper. [Cua comparison and remaining
parity](docs/CUA_PARITY.md) records the upstream reference and proof gaps.
**Since 0.15.0.** Electron/Chromium hover and drag can use an existing,
explicitly enabled renderer endpoint; otherwise they refuse before input.
Native drag-session detection, runtime Tk refusal, web keyboard-scroll fallback
and deeper menus are also implemented. Native semantic targets retain the
observed accessibility object and application launch identity; replacements
and expired observations refuse before input, even when labels match. See the
[agent-desktop comparison](docs/AGENT_DESKTOP.md) and
see [qualification and restrictions](docs/CUA_PARITY.md#october-31-macos-phase-3-source-candidate).
**Status.** Release status is recorded in [CHANGELOG.md](CHANGELOG.md). The
notarized Mac app is published on this repository's
[GitHub releases](https://github.com/codewhale-hq/codewhale-cu-plugin/releases)
(stable release: v0.16.1); remaining
gates are tracked in [the release checklist](docs/RELEASE_CHECKLIST.md). The
[setup page](https://codewhale.net/computer-use) offers the notarized disk
image directly.
The native setup panel, background check and updater require **macOS 13.5+**
for the self-contained bundle. The source MCP server includes experimental
Windows and Linux backends, with HarmonyOS devices over hdc. Windows ships
only as an unsigned x64 preview ZIP and Linux remains source and Docker.
The Linux source candidate adds an approval and Pause/Stop window; neither
platform has qualified exact-window targeting or a signed installer, and both
are excluded from the plugin's public host eligibility until those gates pass.
Their raw input uses the shared desktop
and must not be treated as background control. Windows semantic mutations
bind the observed window and element runtime identities and refuse stale
paths. See the
[publication review](docs/PUBLICATION_REVIEW.md) and
[porting plan](docs/PORTING.md).
The SSH route remains experimental. See the platform-specific
[limitations](docs/LIMITATIONS.md).
**Linux approval setup (source candidate).** The Linux helper uses a GTK 3
control window and a system authentication prompt for each app approval. It
requires Python 3 with GTK 3 bindings, polkit, and a desktop authentication
agent. After building and installing the app, an administrator can install
its bundled policy using the default installation path:
```bash
sudo install -o root -g root -m 0644 "$HOME/.local/share/codewhale-computer-use/plugin/app/linux/net.codewhale.computer-use.policy" /usr/share/polkit-1/actions/net.codewhale.computer-use.policy
```
If `XDG_DATA_HOME` was changed, use the installed app's policy path instead.
Quit and reopen Computer Use after installing the policy. If the policy, GTK,
or an authentication agent is unavailable, approvals fail closed. The session
must be local and active. After authentication, the controller minimizes before
granting access, and remains available from the desktop's window controls. If
the desktop cannot confirm it minimized, the request is refused. The helper
does not offer macOS-style remembered host trust on Linux. This controller has
only been exercised on a disposable Linux desktop; real distributions and
desktop environments still need qualification.
The server works with MCP hosts including Codewhale, Kimi Code, Claude Code,
Codex CLI and Cursor. It has zero runtime npm dependencies. Developer
checkouts use Node 20+; the macOS distribution includes a pinned Node 24 LTS
runtime. Codewhale still reviews, trusts and enables a plugin through its
existing Engine authority before the model can use it.
## The Mac app
The notarized universal app ships as a drag-to-Applications disk image on
this repository's
[GitHub releases](https://github.com/codewhale-hq/codewhale-cu-plugin/releases),
only from a published stable release whose `release.json` receipt qualifies;
the ZIP archive on the same release is what the in-app updater installs.
The plugin marketplace links there, and the
[setup page](https://codewhale.net/computer-use) offers the image directly.
Release status is recorded in [CHANGELOG.md](CHANGELOG.md). To build and install
from source instead, use door 3 in [Install](#install). See
[docs/DISTRIBUTION.md](docs/DISTRIBUTION.md) for the packaging,
notarization and release procedure.
## Codewhale CLI setup
For the published CLI v0.10.2, connect the current app's MCP server directly.
The Mac app includes Node, so this route needs no separately installed runtime:
```bash
APP="/Applications/Codewhale Computer Use.app"
codewhale mcp add computer --command "$APP/Contents/MacOS/node" --arg "$APP/Contents/Resources/plugin/mcp/server.mjs"
codewhale mcp tools computer
```
If the app is installed elsewhere, set `APP` to its actual bundle path.
If the older built-in plugin is already enabled, run
`/plugin disable computer-use` in Codewhale before using this connection.
Direct MCP supplies tools without installing the plugin's commands and skills.
In an existing session run `/mcp reload`, select Operate mode, and ask:
“Check Computer Use permissions with request_access. Do not send input.”
The receipt should report `via: "app"`, the running helper version and any
missing permissions. Follow its setup hint if the helper is absent or
unreachable; do not bypass it. A permission probe does not establish that
arbitrary app actions work.
The November 7 check used published CLI0.10.2 with the current installed
CU0.16.1 server/helper: discovery and a real `request_access` call passed.
A local synthetic model selected the call; no external model service or
computer input was used. This was an existing-helper check, not a clean-Mac
installation or an app-task qualification.
On the unreleased CLI source candidate that removes the built-in copy, the
ordinary marketplace route is available: install `computer-use`, inspect its
capabilities, run the exact hash-bound trust confirmation, then enable it.
See [catalog version and trust details](https://github.com/codewhale-hq/Codewhale/blob/main/docs/PLUGIN_MARKETPLACE.md#computer-use-across-versions).
That source transition is not a published CLI migration.
## Developer quick start
Choose the appropriate setup route ([Install](#install)), then point
your host at the server:
| Host | Configuration |
|---|---|
| Codewhale | Use the [version-specific CLI setup](#codewhale-cli-setup); configure the current app directly over MCP on v0.10.2. |
| Kimi Code | Run `/plugins install /path/to/codewhale-cu-plugin`, then `/reload`. The native `kimi.plugin.json` registers the server and skills. |
| DeepSeek Harness (`dsh`) | `dsh plugin --profile web add @codewhale/computer-use-dsh`, then link `skills/` into `~/.dsh/skills`. See [DeepSeek Harness](#deepseek-harness-dsh). |
| Claude Code | `claude mcp add computer -- node /path/to/mcp/server.mjs` |
| Codex CLI | `~/.codex/config.toml`: `[mcp_servers.computer]` `command = "node"` `args = ["/path/to/mcp/server.mjs"]` |
| Cursor / Windsurf / VS Code | `{"mcpServers":{"computer":{"command":"node","args":["/path/to/mcp/server.mjs"]}}}` |
| Gemini CLI | same JSON in `~/.gemini/settings.json` |
| opencode | `{"mcp":{"computer":{"type":"local","command":["node","/path/to/mcp/server.mjs"]}}}` |
`/path/to/mcp/server.mjs` can be this checkout or the stable copy inside the
installed app (`install:app` prints it — on macOS
`/Applications/Codewhale Computer Use.app/Contents/Resources/plugin/mcp/server.mjs`
after dragging the release app to Applications, or the corresponding
`~/Applications/` path with the source installer).
`npm link` also gives you a `codewhale-cu` command for hosts that want one.
For the self-contained Mac release, use its bundled
`<installed app>/Contents/MacOS/node` as the host's
command to avoid depending on a Homebrew Node installation. Quit Claude
Desktop before editing its MCP configuration, then reopen it and verify the
saved command; Desktop may rewrite its configuration on restart.
### Kimi Code
Install the desktop helper above first, then use Kimi's `/plugins install`
command with this checkout's absolute path. Review the local plugin and choose
**Trust and install**. Kimi copies it into its managed plugin directory and
enables its MCP server. Run `/reload` in an existing session, then `/mcp`:
`plugin-codewhale-computer-use:computer` should show **connected** and the
tool list (37 advertised in 0.9.0; counts vary by host).
`/plugins info codewhale-computer-use` shows the installed version and status.
This flow was verified with Kimi Code 0.41.0. See the
[Kimi plugin documentation](https://moonshotai.github.io/kimi-code/en/customization/plugins.html)
for managed updates and removal.
The server needs Node 20 or newer on Kimi's PATH and uses the installed
Codewhale permission helper. Existing MCP servers and model settings are
preserved. Models without vision can read the default text observations and
request local macOS OCR; models with vision can also request screenshots.
### DeepSeek Harness (dsh)
Install the desktop helper above first, then add the bundle in
[`integrations/dsh`](integrations/dsh). `dsh` manages a profile's plugins as
npm packages: `dsh plugin` forwards to pnpm in the profile directory and then
reconciles `dsh.profile.bundles`, so one command installs the package *and*
adds its patch layer to the profile.
```bash
dsh plugin --profile web add @codewhale/computer-use-dsh
```
Use the path to this checkout's `integrations/dsh` instead of the package name
to install from source. The tools arrive as `mcp__computer-use__<name>`; the
bundle runs the server from the installed app with the Node already running
`dsh`, so nothing else needs configuring. Set `CODEWHALE_CU_SERVER` to point at
a `/Applications` install or a source checkout's `mcp/server.mjs`.
The two skills install separately. A profile whose agent presets own skill
discovery — the `web` profile is one — disables the host-level
`skill-filesystem` row, so a bundle cannot reach it. The user skill root is
read in every composition:
```bash
PLUGIN="$HOME/Applications/Codewhale Computer Use.app/Contents/Resources/plugin"
mkdir -p ~/.dsh/skills
ln -sfn "$PLUGIN/skills/computer-use" ~/.dsh/skills/computer-use
ln -sfn "$PLUGIN/skills/recording" ~/.dsh/skills/recording
```
To scope desktop control to sessions that ask for it rather than every session
on the profile, put the same `mcp-computer-use` row in an agent preset instead
of installing the bundle: duplicate `standard` in the `dsh` UI, then add the
row from `integrations/dsh/cordis.patch.yml` to the copy's `agent.cordis.yml`
and set `customSkillDirs` on its `skill-filesystem`. A preset whose rows cannot
be resolved is listed as broken with the reason, so a typo shows up in the
picker rather than at session start.
Verified with `dsh` 0.1.5-rc.1 and plugin 0.6.0 on macOS: `dsh plugin add`
into a profile, 44 tools bridged, both skills in the catalog, and a live
`computer_list` call returning from the helper. Remove it with `dsh plugin
--profile web remove @codewhale/computer-use-dsh`. Models without vision read
the default text observations and can request local macOS OCR; models with
vision can also request screenshots, which `dsh` bridges as attachments.
## The desktop app
macOS grants Accessibility and Screen Recording to the *responsible process*
of a permission check. A bare `node mcp/server.mjs` inherits its host's
identity, so the grant lands on Terminal, Cursor, or Claude, and every host
needs its own. The app fixes that:
- **What it is** — a tiny native launcher (`app/macos/launcher.c` on macOS)
that runs `app/daemon.mjs`, a long-lived process that executes the platform
backend and answers one-line JSON requests over a per-user socket
(`~/.codewhale-cu/app.sock`, a named pipe on Windows). Only the same
allow-listed tool set the ssh agent accepts will execute; the socket is
never a shell.
- **How the server uses it** — every call on the local computer goes through
the app when it is running. If it is installed but not running, the server
launches it (via LaunchServices on macOS, so it is its own responsible
process) and waits for it. On macOS, a missing helper returns
`app_setup_required` with instructions to install and open the separate app;
a registered helper that cannot start fails closed. `request_access` reports
which mode is active (`via: "app"` or `"direct"`). Linux and Windows can run
directly in the server process when no helper is installed.
`CODEWHALE_CU_APP=off` is an explicit developer override for direct mode,
never an agent workaround for the person's Pause or Stop choice.
- **First launch** — open the whale menu to review permission status. Only the
setup buttons request grants; starting the helper does not prompt automatically.
Where `install:app` puts things:
| OS | App | Also |
|---|---|---|
| macOS | `~/Applications/Codewhale Computer Use.app` (registered with LaunchServices) | `--login`: `~/Library/LaunchAgents/net.codewhale.computer-use.plist` |
| Linux | `~/.local/share/codewhale-computer-use` | `~/.local/bin/codewhale-computer-use`, `~/.local/share/applications/net.codewhale.computer-use.desktop`, hicolor icons; `--login`: `~/.config/autostart/` |
| Windows | `%LOCALAPPDATA%\Programs\Codewhale Computer Use` | Start Menu shortcut with icon; `--login`: Startup-folder shortcut |
### macOS permission recovery
The build signs the complete app bundle, and installation signs again after
pinning Node, then verifies the installed signature. Signing only the launcher
leaves an invalid app identity: macOS can show an enabled switch while refusing
the grant and repeatedly prompting for Accessibility.
If upgrading from that broken package, remove the old Codewhale Computer Use
entry from Accessibility, add `~/Applications/Codewhale Computer Use.app`, and
enable it. Refresh its Screen & System Audio Recording grant and restart the
helper. The probe must report Accessibility `granted` and screen capture `ok`;
`via: "app"` alone is not a pass. Authentication in System Settings is handled
by the user. Never modify the TCC database.
Builds use `CODEWHALE_CU_SIGN_IDENTITY` when configured, otherwise an available
Developer ID or Apple Development identity, and fall back to ad-hoc signing.
Ad-hoc updates can require a new grant. Installation preserves the selected
identity and verifies the finished bundle. Building the native macOS helper
requires Xcode Command Line Tools on a Mac; the separate Computer Use app
download includes the compiled helper. Do not edit installed resources without
re-signing.
### Background control and visible cursor on macOS
Use `open_application` with `activate: false` to select the input destination
without bringing it forward. Keyboard events go to that process and semantic
actions use Accessibility — both are quiet: no cursor movement, no activation.
Unqualified `get_app_state`, `list_windows`, and `screenshot` follow that app,
including behind other windows. Under the 0.13.0 helper, explicit display/region
captures require host approval and are refused when an app exclusion could expose
private content. Prefer an app-scoped capture.
Text insertion prefers writable accessibility selection over process key events.
For an authorized workflow that requires keyboard focus, explicitly select
`activate: true`; receipts then say `keyboard_delivery: "foreground-guarded"`.
Keystrokes and raw mouse gestures stop if another app takes focus; gestures never
reactivate it. Select `activate: false` again to
return to process-bound delivery. In either mode, verify the application
result: a successful dispatch alone does not prove the app handled it.
Coordinate `left_click` first resolves the bound application's accessibility
control and uses accessibility (`strategy: "a11y"`). Text fields focus directly,
rows can select, and menu items can use their advertised pick action. Right-click
uses an advertised context-menu action. Background scrolling uses the selected
scrollbar: native increments when advertised, otherwise normalized 5% steps,
with the actual unit and value change in the receipt. With v0.13.1 or newer,
standard approval mode also permits raw clicks, drag, hover and wheel events
posted to the selected window's process. A coordinate scroll without an
accessibility scrollbar uses that wheel route. Strict approval mode requires
`activate: true` for raw gestures. There is no automatic foreground fallback.
When the authorized task needs focus in an allowed app, select `activate: true`
for foreground control without a second approval prompt. In either mode,
raw clicks, hover, drag and wheel go to the bound
app's window as process-routed events (`strategy: "pid-routed"`,
`pointer_moved: false`) — the agent has its own pointer, drawn in the preview,
and **the user's cursor is never moved, warped or held on macOS**. There is no
real-cursor fallback: a helper that cannot resolve the window route refuses
`bg_dispatch_unavailable`. `pointer` down/move/up buffer a drag and deliver it
to the window on `up`. Return to `activate: false` when the foreground step
ends.
`activate: false` is the default on every platform, not just macOS: Windows
launches the app minimized and Linux hands focus back to the previous window
after launch. Raw input on Windows and Linux is still shared-surface by
nature — background there means the launch does not steal focus, not that
input becomes background-safe.
The binding receipt exposes `input_scope`, `shared_pointer: false`,
`pointer_route: "pid-routed"` and `isolated_desktop: false`. The preview title distinguishes background app
control from shared-desktop control. It is a view of the app, not a sandbox.
Since 0.13.0 the helper uses a compact Codewhale controller naming the app, with
a direct Stop computer use button and Pause in its menu. Stop ends all current
computer sessions. Idle shows Ready; Settings holds remembered access, previous
connections and the optional background diagnostic. It is Codewhale's own UI, not an assertion that macOS shows an
identical system indicator. It does not capture the screen. The screenshot
preview is off by default.
Only when the user asks to watch, enable `preview` with `enabled: true` to show a small, nonactivating window
containing the controlled app and a cyan cursor labeled Codewhale. The preview
refreshes on a timer while a session is bound (`CODEWHALE_CU_PREVIEW_REFRESH_MS`,
default 1000; 0 disables), so it behaves like a live view of the app instead of
a frozen still; its cursor is separate from the hardware pointer. Use the
Close button, **Close preview** in the Computer Use menu bar, or `enabled: false`
to stop watching. The panel also hides after
60 seconds without session activity or when that session closes; a session
that disappears stops renewing the panel, which then expires.
### Consent and shared-machine manners on the local computer
The Mac helper owns local admission. Default standard mode admits local hosts
without a trust panel or per-app prompt, subject to denied apps and the human
controls above. In strict mode, a native decision grants this Codewhale
installation permission until the person revokes it in Computer Use; other
hosts without a persistent key receive connection-scoped approval.
Discovery and `consent {action:"status"}`
do not themselves ask for access. The latter reports the helper's actual trust
and exclusions. An installed older plugin may still use the legacy per-app
ledger until it is updated.
Codewhale supplies its stable secret on the plugin's private stdin channel.
The plugin derives an Ed25519 identity and signs a fresh helper lease challenge;
the native helper stores only the approved public key and exclusions in Keychain.
This proves possession of the same installation key, not the vendor identity or
code signature of a running app. It is not an OS sandbox: same-user code able
to read Codewhale's secret store can impersonate that installation. Details and
qualification limits are in [Limitations](docs/LIMITATIONS.md).
App and foreground exclusions still win. On the current helper, app
`consent deny` or `revoke` narrows access; it cannot clear a native exclusion.
Whole-host revocation is a human control in Computer Use. The agent must not
retry a denial under another name, session or tool. macOS Accessibility, Screen
Recording and Automation permissions remain separate OS decisions.
Whole-screen captures can show unrelated apps; use scoped observations and
captures for the task. A host with app exclusions cannot take an unscoped
whole-screen capture or use a browser route that cannot enforce those exclusions.
Trust does not start a recording or preview. Spawned task computers and remote
computers retain their existing transport rules.
`app_script` needs a fresh approval for the exact script; the helper enforces
its single-use token and app exclusions.
Irreversible-action confirmations and computer registration or spawning still
need a host-attested decision or MCP elicitation. Without a supported decision
path the call refuses `consent_needs_user`.
For macOS hosts without the helper, an interim `osascript` dialog is available
only with `CODEWHALE_CU_NATIVE_PROMPT=1`; it is off by default. An installed but
disconnected helper is never bypassed. While a helper prompt is open, other
input and observation are refused with `consent_pending`; Stop cancels it.
Prompts are rate-limited, and declining or timing out does not authorize a retry.
See [docs/LIMITATIONS.md](docs/LIMITATIONS.md) for the decision paths and their
verification limits.
In explicitly selected foreground mode for an approved app, whenever a shared surface is taken —
a real-pointer gesture, foreground keys, or an activation — and before a
press that opens a menu in a background app, the helper waits
for a gap in the user's hardware input first (bounded, ~450 ms gap within a
2.5 s window by default; `CODEWHALE_CU_YIELD_GAP_MS` /
`CODEWHALE_CU_YIELD_WAIT_MS`). If no quiet window arrives, the call refuses
`user_busy` before sending input. Successful receipts report the wait as
`yield_ms`. Turn-taking, not a lock: `user_input_during_lease` still reports
input that arrived mid-action.
### Browser control (CDP)
`browser` drives a Chromium-family browser over the Chrome DevTools protocol
in a self-owned profile under the state dir — the person's own browser is
never attached to, typed into, or closed. `start {url?}` launches (or reuses)
the instance and binds this session's own tab; `navigate`, `click` (a CSS
selector's box center, or a page-viewport point), `type` (optional selector +
`enter`), `screenshot`, `status`, and `stop` follow. Page screenshots return
as inline images like screen captures do. `stop` closes this session's tab;
the shared browser goes down when no tabs remain. Coordinates in this family
are page-viewport pixels (`space: "page-viewport"`) — never screen points.
Node 22+ is required for the WebSocket transport; older runtimes refuse with
`unsupported_runtime`.
This is background control of a local app, not an isolated desktop. Some apps,
system dialogs, and workflows may still require foreground interaction. The
opt-in `node scripts/verify-macos.mjs` test exercises a uniquely named TextEdit fixture (closed automatically afterward; add `--preview` to test the overlay),
Unicode input, selection, screenshots and foreground preservation through a
fresh MCP connection. Recording is separately opt-in with `--recording`. It creates local receipts and is
separate from the automated unit suite.
Logs: `~/Library/Logs/Codewhale Computer Use/app.log` (macOS),
`~/.local/state/codewhale-computer-use/app.log` (Linux),
`%LOCALAPPDATA%\Codewhale Computer Use\app.log` (Windows).
`npm run remove:app` reverses the install. Every bundle carries a complete
runtime copy of the plugin, so hosts can target the installed path and survive
deleting this checkout.
## Session ownership
Each MCP connection has independent computer selection, application binding,
rasters, accessibility observations and held-input ownership. Local actions
are serialized by the permission-owning app; stopping or cancelling one
session releases its own held input. Normal client shutdown closes the app
session; losing the client connection also cancels and releases its input.
On macOS, the native input helper also releases its press when its owning
process disappears, including in direct mode. On macOS, session exit stops a
recording started by that session; cancelling an ordinary request leaves an explicitly
started recording running until stopped or the session exits.
Session protocol 2 (introduced in 0.2.1) is required: upgrade the helper and
restart existing MCP connections together. An old client or helper is refused
with an upgrade error instead of sharing another client's input state.
## Frontier ability set
The advertised surface is merged for context economy — `click`, `pointer`,
`clipboard`, `recording`, `computer`, `browser`, with `key {duration}` covering
holds; `list_sessions`, `kill_app`, `set_window_frame` and `trajectory` ride
alongside as themselves, and `list_apps {installed:true}` returns the
installed catalog rather than running processes.
### Capability grants
`CODEWHALE_CU_GRANT` narrows a server process at launch: `read-only`, or a
comma list of tool names (a merged name expands to its whole action set).
`tools/list` advertises only granted tools, ungranted calls fail as
`not_granted` before validation, and the app daemon enforces the same set on
the session lease so a narrowed server cannot smuggle one through. Cleanup and
`stop_computer_control` are never blocked; `request_access` reports the grant.
The per-action wire names (`left_click`, `read_clipboard`, `recording_start`,
`computer_list`, `hold_key`, …) remain callable as aliases, so pinned hosts and
receipts keep working; `tools/list` advertises the merged set only.
- **Observe & resolve** — `list_apps` (regular running apps by default;
`all:true` for helpers, `installed:true` for the launchable app catalog), `list_windows`, `list_displays`,
`switch_display`, `get_app_state` (accessibility/UIA/uitest tree with
element indices + `state_id`), `screenshot` (display/region, raster-bound
coordinates), `zoom` (close-up crop of the last raster), `cursor_position`,
`open_application` (exact-name rule), `request_access` (fail-closed
permission/capability probe), `wait_for` (poll the accessibility tree
until a query/role appears or disappears, then act on the fresh
`state_id`).
- **Pointer** — `click` (left/double/triple/right/middle through
`button`/`clicks`), `pointer` (move/down/up), `drag`, `scroll`
(4 directions).
- **Keyboard & text** — `type` (unicode), `key` (chords, `repeat`, or
`duration` to hold), `set_value` (semantic, background-safe), `select_text`,
`perform_action` (element's own actions: AXPress / UIA Invoke / AT-SPI / uitest),
`invoke_menu` (menu items by title path; accessibility only — no key events
or focus lease; window-targeted items may need a key window).
- **Scripting** — `app_script` (macOS, local computer): AppleScript or JXA
through osascript into apps that ship a scripting dictionary — the
programmatic interface to prefer over clicking wherever one exists.
Returns stdout as `result`; refusals are typed (`script_error`,
`script_timeout`, `automation_denied` for a declined/missing Automation
consent). Refused on ssh/docker/hdc computers: a remote channel stays
computer-use only, never a shell.
- **Guidance & policy** — the operating skill ships as MCP resources
(`skills/list`, `skills/get`, `resources/read` of `skill://codewhale-cu/…`,
sha256 manifest) and as the `skills/computer-use/` pack in this repo; every
tool advertises MCP annotations (readOnly / destructive / idempotent /
openWorld) for host approval and sandbox policy.
- **Recording** — `recording` (start/stop/status/list; see below).
- **Computers** — `computer` (list/switch/register with ssh agent
auto-push/spawn/remove). `spawn {id, transport:"docker"}` provisions a
task-owned disposable Linux desktop container: it is registered
`owned:true`, becomes active, takes every tool unchanged through the
same agent protocol as ssh, and is destroyed by `remove` or when the
MCP session ends, and stops after its lifetime (six hours by default). A computer is an execution environment — prefer a
spawned desktop for work that does not need the user's own session,
and keep `local` for the tasks that do.
- **Safety** — `stop_computer_control` kill switch; permission probes that
name the missing grant; receipts on every call naming the computer it
happened on.
## Requirements
Tools and permissions are probed at call time; `request_access` reports what is
missing and every capability **fails closed naming the missing tool or
permission** — it never guesses and never half-acts.
- **macOS** — Accessibility + Screen Recording for the app (or, in direct
mode, for the terminal that hosts the server). python3+pyobjc or cliclick
improves cursor reads.
- **Windows** — PowerShell (built in). Recording is unavailable pending
session-owned recorder cleanup.
- **Linux** — X11: xdotool, wmctrl, scrot or imagemagick, xclip, and ffmpeg
for recording (`x11grab`); Python 3 and xmodmap for non-ASCII typing;
python3-pyatspi for the accessibility tree.
Wayland: grim screenshots, wl-clipboard and AT-SPI. Keyboard and pointer
input work on sway and Hyprland (python3 and libxkbcommon, both present
on those desktops), where the compositor names the app that would
receive the input. GNOME, KDE and other compositors do not, so input is
refused there. Recording is unavailable on Wayland.
- **HarmonyOS** — `hdc` on PATH with the device connected
(`hdc list targets`); ffmpeg on the host for snapshot-series recordings.
- **Spawned computers** — a reachable docker daemon (`docker version`).
`computer spawn` builds the plugin's Linux desktop image from
`docker/Dockerfile` on first use; no sshd, keys or host mounts are
involved — the agent rides `docker exec` and the container boundary is
the isolation.
## How the four platforms map
| Ability | macOS | Windows | Linux | HarmonyOS |
|---|---|---|---|---|
| Accessibility tree | Native Accessibility API | UIAutomation | AT-SPI (pyatspi) | `uitest dumpLayout` |
| Raw input | Native CGEvent to the selected process | user32 SendInput/mouse_event (PowerShell) | xdotool (X11); virtual keyboard and pointer on sway/Hyprland; refused on other Wayland compositors | `uitest uiInput` |
| Screenshots | `screencapture` | .NET CopyFromScreen | scrot/import (X11), grim (Wayland) | `snapshot_display` |
| Recording | ScreenCaptureKit → .mov (no recorder overlay) | unavailable pending owned cleanup | ffmpeg `x11grab` → .mp4 (X11); unavailable on Wayland | snapshot-series + ffmpeg mux |
| Clipboard | pbcopy/pbpaste | Get/Set-Clipboard | xclip/xsel, wl-clipboard | fail-closed (not exposed by hdc) |
## Remote computers (ssh)
```json
computer_register { "computer": "winbox", "transport": "ssh", "host": "winbox.lan", "user": "me" }
```
Registration pushes the self-contained agent (`agent.mjs` + `src/`) to
`~/.codewhale-cu/agent/` on the remote over scp, probes the remote platform
through it, and pins the result. Remote calls run
`node agent.mjs <base64 json>` — one JSON receipt line back. Only an
allow-listed tool set executes remotely; arguments travel as data, never as
shell. Requires publickey ssh (BatchMode) and Node ≥ 20 on the remote.
## HarmonyOS computers
```json
computer_register { "computer": "pad", "transport": "hdc" }
```
Drives the device over `hdc shell uitest ...` and `snapshot_display`. Element
targets come from `dumpLayout`; input is touch-synthesis (click / swipe /
inputText / keyEvent). Recording is honestly labeled `snapshot-series`
(frame captures muxed on stop) because HarmonyOS exposes no CLI screen
recorder.
## Verification status
**0.13.0 candidate.** A signed build passed native human approval, Unicode
input/click/scroll/app capture, reconnect through a new MCP process after helper
restart, and human revocation on one maintainer Mac. All 115 observer samples
had no pointer or foreground changes. The subsequently simplified native UI
and release packaging still need qualification. Focused source checks alone
do not qualify an installed app. Final
counts and release receipts belong in the [release notes](CHANGELOG.md) and
[release checklist](docs/RELEASE_CHECKLIST.md).
Earlier source-suite and live receipts below cover their named versions.
Protocol, routing, session and injected-runner tests perform no native input
and do not qualify a distributed app.
**Live-verified: macOS 0.8.0 acceptance (one maintainer Mac, arm64,
2026-09-17).** A muse-driven pass over the installed 0.8.0 plugin plus
fixes: 36 advertised tools with hidden aliases callable; TextEdit
invoke_menu open/close with `front_lease`/`front_restored` honesty on every
key receipt; read-only grant narrowing (19 tools, `not_granted`, grant on
refusal receipts too); a 3-turn trajectory with a refusal that replay
stopped at; 143 installed apps with running flags; TextEdit frames with
verified geometry and Calculator's fixed-size refusal via `ax_errors`;
browser start/navigate/click/type/screenshot/stop in a self-owned profile
with the user's Chrome untouched; session list, live preview refresh/hide,
and verified `kill_app`; malformed calls failing as `bad_args`/`bad_target`
(never TypeError); `element_stale` naming state and app; the kill switch
failing closed; and the real pointer provably unmoved across background
clicks. Tracked as Codewhale CU SHA-6642.
**Native macOS 0.4.0 (one maintainer Mac, arm64).** The 0.4.0 universal
app was Developer ID signed and Apple-notarized from commit `249ae77`, then
installed over the notarized 0.3.1 app on the maintainer Mac; the packaging
facts are in [docs/releases/0.4.0.json](docs/releases/0.4.0.json). The AX
primitives it adds (Return from `type`, filtered `get_app_state`, `focus`,
`get_value`, `strategy:"app"`) are covered by the source suite; their native
qualification on an installed build is still open.
**Native macOS 0.3.1 (one maintainer Mac, arm64).** An earlier signed
0.3.1 candidate passed the menu-bar owner crash and reopen check with
isolated state. The updater's apply step replaced an installed notarized
0.3.0 app with that candidate: the previous bundle was retained for rollback,
the helper restarted with controls stopped, and all 33 runtime files plus the
3 native executables in the installed bundle matched the build. The open
qualification gates, and the build, signing and notarization facts for the
public 0.3.1 archive, are in [CHANGELOG.md](CHANGELOG.md) and
[docs/releases/0.3.1.json](docs/releases/0.3.1.json).
**Live-verified during development: macOS (arm64, single Retina display,
macOS 26.1).** Each of the 27 fixture workflows has a five-trial passing run.
The broad 26-task run passed 129/130 trials; its dynamic-page failure was a
fixture clock race, corrected and repeated 5/5. The repaired file-picker flow
also passed 5/5 separately. The matrix retains the broad failure and both
focused runs; these are 0.2.1 development receipts, not full Codex parity or
final release qualification. Shared-desktop pointer displacement is measured
and sometimes nonzero. OS permissions survived the signed 0.2.1 update. Commit
identifiers inside the matrix and result files refer to the private
pre-publication history, not to commits in this repository.
**Text and vision use the same actions.** App observations default to a text
summary containing controls, values, actions and layout. `detail:"full"`
exposes nested menus and tree structure. On macOS, `include_ocr:true` adds
on-device recognition of visible text with confidence and coordinate targets,
without a vision model or remote service. The default does not capture an
image. OCR was verified on a generated image and the actual Codewhale app;
it does not interpret unlabeled icons, charts or other graphical meaning.
Screenshots and zoom remain available to models that support images.
**Linux X11 (experimental, from source).** Verified on 2026-10-11 on virtual
X11 desktops (Xvfb with openbox, and a full XFCE session) and in the Docker
test image, at repeats 5 over the 27-task suite. GNOME and KDE sessions have
not been tested.
These 2026-10-11 results are historical and do not establish the current
helper route. The 0.13.0 source candidate has a GTK/polkit approval controller;
its isolated Linux checks do not qualify other desktops. Direct mode remains
available for development when explicitly selected by the user and does not
support held gestures.
| route | demonstrated | not demonstrated |
|---|---|---|
| isolated Xvfb, direct | 24/27 | the two drag tasks (need the desktop helper); `native.modal_dialog` 0/5 |
| isolated Xvfb, desktop helper | 25/27 | `control.permission_denied` (a direct-route check); `native.modal_dialog` 0/5 |
| shared display, direct | 23/27 | the two drag tasks; `browser.unicode_type` 4/5; `native.modal_dialog` 0/5 |
| shared display, desktop helper | 25/27 | `control.permission_denied`; `native.modal_dialog` 0/5 |
| Docker image, direct | 23/27 | the two drag tasks; `browser.unicode_type` 2/5; `native.modal_dialog` 0/5 |
| Docker image, desktop helper | 23/27 | `browser.unicode_type` 4/5; `browser.modifiers` 4/5; `control.permission_denied`; `native.modal_dialog` 0/5 |
| XFCE session (xfce4-session, xfwm4, panel) on Xvfb, direct | 23/27 | the two drag tasks; `browser.unicode_type` 4/5; `native.modal_dialog` 0/5 |
| XFCE session, desktop helper | 24/27 | `browser.unicode_type` 4/5; `control.permission_denied`; `native.modal_dialog` 0/5 |
| Wayland: headless sway 1.9 (XWayland for Tk), direct | 21/27 | the two drag tasks; `browser.upload` 0/5 (GTK's header-bar file chooser puts Open where the task's X11-layout click does not reach); `browser.unicode_type` 0/5 (Chromium on Wayland drops 🐳); `native.unicode_type` 4/5; `native.modal_dialog` 0/5 |
Known limits: on X11, input to a window is refused while a visible dialog
explicitly identifies that window as its parent. Interact with the dialog or
close it before returning to the parent. This conservative rule also blocks
some non-modal dialogs, and cannot identify every toolkit dialog or eliminate
window changes between the check and input. On Wayland a toolkit-modal dialog
can still let typed text reach its parent, so click into the dialog first.
X11 non-ASCII typing holds a temporary key mapping while sending
input, but delivery is not guaranteed; read it back with `get_value`. On
Wayland, input goes only to the app that sway or
Hyprland names as its receiver: the focused window's process for keys, and the
window under the point for the pointer. It is refused where that cannot be
told: overlapping floating windows of different apps, overlapping XWayland
windows of different apps, and pointer input at the current pointer position
on sway, which does not report it. GNOME, KDE and other Wayland compositors
name no receiver, so input there is refused, leaving screenshots (wlroots
only), the clipboard and the accessibility tree. Hyprland is unit-tested only.
Not tested: GNOME or KDE sessions, a real sway login session, real XWayland
on a desktop, multiple monitors, HiDPI, SSH remote. Receipts are in
`parity/results/`; [docs/PARITY_MATRIX.md](docs/PARITY_MATRIX.md) and
[docs/LIMITATIONS.md](docs/LIMITATIONS.md) have the detail. The Linux helper
requires the system approval setup above before it can perform held gestures.
**Background input has a native AppKit verification harness** with an independent
foreground/cursor observer: `node scripts/verify-background-macos.mjs`.
Older per-family receipts do not establish background isolation — details in
[docs/LIMITATIONS.md](docs/LIMITATIONS.md).
**Not live-verified:** macOS non-Retina and mixed-DPI, Windows, Wayland beyond headless sway,
HarmonyOS, SSH remote — implemented, no device receipts. A
[same-document native editing comparison](parity/results/native-text-comparison-darwin-2026-09-07.json)
completed 5/5 trials with each of Codewhale and Codex; it covers one workflow,
not the full task suite. A separate
[public-browser navigation comparison](parity/results/public-browser-comparison-darwin-2026-09-07.json)
also completed 5/5 trials on each surface, using Codewhale 0.2.1. Known behavioral limitations and the
release gating checklist are in
[docs/LIMITATIONS.md](docs/LIMITATIONS.md) and
[docs/RELEASE_CHECKLIST.md](docs/RELEASE_CHECKLIST.md); how to run or extend
the suite is in [docs/PARITY.md](docs/PARITY.md).
## Layout
```
mcp/server.mjs MCP stdio server (JSON-RPC 2.0), tool dispatch, receipts
src/tools.mjs tool schemas — single source of truth for tools/list
src/backends/ darwin / win32 / linux / harmonyos
src/transport.mjs app socket · local · ssh · hdc executors
src/app-socket.mjs app naming, socket protocol client, launch-on-demand
src/app-handler.mjs allow-listed request handler shared by app + ssh agent
app/daemon.mjs the desktop app process
app/macos/launcher.c native bundle executable (keeps TCC attribution on the app)
agent.mjs ssh remote agent (one-shot, or `--serve` persistent session)
assets/ icon source + generated .png/.icns/.ico/hicolor, prebuilt mac launcher
scripts/ build-icons · build-app · install-app · smoke
commands/, skills/ Agent Plugins v1 command + skills for hosts that read them
```
## Development
```bash
npm test # unit + protocol + app tests (no GUI input performed)
npm run smoke # live end-to-end against this machine (isolated state dirs)
npm run build:icons # regenerate assets/ from assets/icon-source.png
npm run build:app # bundles into dist/ (compiles the mac launcher when clang is present)
```
Proven levels are separated: local live (this Mac: darwin) > mocked transport
(ssh protocol, harmony backend logic) > code-complete (win32/linux paths,
implemented to their documented tool interfaces but only verifiable on those
platforms).
This repository supplies the independently installed Computer Use plugin and
native helper. Codewhale installs and updates the plugin through Connectors
from a stable published release; the helper has its own verified app updater.
Each release carries a portable `computer-use.tar.gz` asset for the existing
plugin installer. Changing this checkout updates neither installed
component. Plugin and helper updates do not require rebuilding Engine.
## Support and contributing
Use [GitHub issues](https://github.com/codewhale-hq/codewhale-cu-plugin/issues) for
bugs and questions. Include the plugin version, macOS version, the failed
action and the error text, with private app contents and credentials removed
from any log excerpt. [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)
covers the common setup problems first. Contribution expectations are in
[CONTRIBUTING.md](CONTRIBUTING.md); vulnerability reporting and the security
model are in [SECURITY.md](SECURITY.md). This is a beta maintained on a
best-effort basis; there is no support commitment or response-time promise.
## License
MIT — see `LICENSE`.