# Computer Use MCP
Give an AI agent the ability to discover apps, understand their controls, and work
on a desktop. Computer Use MCP connects MCP-compatible agents to native tools on
**macOS, Windows and Linux**.
Use app scripting and accessibility controls when available, and screenshots,
mouse and keyboard input when the interface requires them.
## See it paint
**Astra painted this through real mouse strokes.** The agent used the OpenAI
Responses API to operate the bundled paint studio through this MCP: 105 strokes,
seven colors and 15 model calls, followed by a verified PNG export. The result is
an original Van Gogh-inspired landscape made through native pointer events.
```sh
# From a clone of this repository, after the setup below:
npm install --prefix agents/openai-agent
npx playwright install chromium
export OPENAI_API_KEY=your-key
node agents/openai-agent/showcase.mjs paint --studio --turns 24 --tokens 200000
```
This makes paid API calls. The recorded run used 191,734 input and 3,274 output
tokens. [Run the demos and read their limits](agents/openai-agent/README.md).
## See it read
**Thirty-eight seconds, ten fields, no answer key.** The invoice on the left is
drawn on a canvas, so its text never reaches the accessibility tree — the only way
to read it is to look. DeepSeek Flash reads it from screenshots, types each value
through the accessibility API, and presses Verify; the local host scores the
entries and tells it only which ones are wrong. This run scored 10 of 10 on the
first Verify, over 12 model calls and 23 tool calls, at a 94% prompt-cache hit
rate. Every entry is recorded with its `isTrusted` flag, so a submission that was
not typed through the UI is refused.
```sh
npm install --prefix agents/deepseek-agent
npx playwright install chromium
export DEEPSEEK_API_KEY=your-key
node agents/deepseek-agent/showcase.mjs --seed 7 # the run above
node agents/deepseek-agent/showcase.mjs --receipt random # a real receipt from the web
```
[Full-resolution capture](docs/assets/deepseek-ledger-run.mp4) ·
[how it works, and what it cannot fake](agents/deepseek-agent/README.md)
## See it plan
**Watch the work instead of reading tool calls.** The run console is an MCP App the
agent reports into. It declares a plan, marks one task active at a time, and attaches
real screen captures as it goes — the frame above is a live capture of Blender's own
window.
Underneath the conversation is everything it is doing: each thought, each tool call
and each result, with timings. That feed is reported by the host driving the run
rather than by the model, so it is complete — Blender's own calls appear in it too.
You can type while it works. The message lands in the transcript that every run tool
returns, so the agent reads it on its next call and can change course mid-task. Attach
an image and it gets both the picture and the path the file was saved at, which is how
Blender can open your reference as a texture rather than the agent working from its own
description of it. `web_search` and `scrape` let it check a fact it does not know —
both run locally, so they work with any model.
Look at it right now, with no API key:
```sh
npx -y -p @zavora-ai/computer-use-mcp computer-use-mcp-console
```
That opens the console and drives a **scripted** walkthrough — the words are written
into the host, but everything it looks at is real, so the frame on the right is a
capture of your actual screen. For a live agent instead, start the host with
`--no-demo` and point one at `http://127.0.0.1:4517/mcp`:
```sh
computer-use-mcp-console --no-demo
```
That endpoint serves the desktop tools and the run tools from the same server, so
whatever the agent plans and captures appears in the browser. The screenshot above
is [an ADK-Rust agent on DeepSeek Flash](https://github.com/zavora-ai/adk-rust/tree/main/examples/blender_console)
refining a watch it modelled earlier in the same conversation.
The console is loopback-only and unauthenticated: `/mcp` grants desktop control to
anything that can reach the port. Fine on your own machine, never on a shared host.
## Set up your agent
Requires **Node.js 20+** and an interactive desktop. npm installs the native module
for your platform (macOS, Windows or Linux, on x64 or arm64) from its own package,
`@zavora-ai/computer-use-mcp-<platform>-<arch>`, chosen by OS and CPU, so an install
downloads one binary. Installing with optional dependencies turned off
(`--omit=optional`) leaves it out, and the server says how to fix that.
Add this to your agent's MCP server configuration:
```json
{
"mcpServers": {
"computer-use": {
"command": "npx",
"args": ["-y", "@zavora-ai/computer-use-mcp"]
}
}
}
```
For a client with a different configuration format, use `npx` as the command and
`-y @zavora-ai/computer-use-mcp` as its arguments. See [agent configuration](AGENTS.md)
for Cursor, Claude Desktop, Codex and programmatic examples. No model API key is
needed by the MCP server itself.
Before using desktop tools:
| Platform | Setup |
|---|---|
| macOS 14+ | Grant the host app Accessibility (called Device Control and Data Access from macOS 27) and Screen Recording access. The host is the app that launches the server (your terminal, Cursor or Claude Desktop), not `node`. Input Monitoring lets the user-active guard see when you are typing or moving the mouse. App scripting may also request Automation access. Until Accessibility is granted, input tools refuse with `accessibility_permission_denied` rather than silently doing nothing. The capture helper (fast captures and OCR) and the status card are compiled on first use, so install the Xcode Command Line Tools (`xcode-select --install`). |
| Windows | Run in a signed-in desktop session. Protected or elevated windows may need matching privileges. Coordinates are physical pixels at any display scale. |
| Linux | Use an X11 or Wayland session. On GNOME 46+ and KDE Plasma 6.1+, Wayland input goes through the desktop's RemoteDesktop portal: you approve once and the choice is remembered. Elsewhere on Wayland install `ydotool`. Accessibility needs AT-SPI (`python3-gi`, `gir1.2-atspi-2.0`). See [testing on Linux](docs/TESTING-LINUX.md) and [platform details](docs/ARCHITECTURE.md). |
Ask your agent to run `doctor` to check capabilities. The `computer-use-mcp` command checks
its own install first (dependencies, build, native module) and names the fix on stderr if
something is missing. Office must be installed and activated separately; its folder-access
prompts are normal setup requirements. Downloads is not guaranteed to bypass those prompts.
## What it can do
| Capability | Examples |
|---|---|
| Discover applications | Find installed and running apps, identify their targets, and choose an automation approach. |
| Read the interface | Capture a window, zoom into a region, inspect accessibility controls and find a button or field; read a window's text by on-device OCR when an app draws its own UI (macOS); wait for a window, for text, or for the screen to settle instead of polling screenshots. |
| Operate applications | Click controls (`click` with button and count), fill forms, select menus, type text, drag paths and use keyboard shortcuts; set a session target once instead of naming the window on every call; run a known sequence in one call (`run_steps`). Every action says whether it was verified, and every refusal names a safe next call. |
| Use app scripting | AppleScript/JXA on macOS, PowerShell on Windows, and bash or PowerShell on Linux. |
| Manage desktop work | Target individual windows, switch focus, read/write the clipboard and inspect displays. |
| Build agent hosts | Add persistent desktop sessions, verified workflows, an MCP App console, isolated browser contexts and supervised runtimes. |
| Stay in control | A desktop status card (on by default on macOS and Windows; `COMPUTER_USE_STATUS_CARD=true` on Linux) shows what the agent is doing with Pause and Stop, and `ctrl+alt+escape` stops it. Moving the mouse or typing pauses the agent until you stop. A locked screen is refused by name, and on macOS and Windows the display stays awake while the agent works. Neither the card nor the agent's pointer appears in its screenshots (on Linux the card is painted out of them, and it reserves a strip of the desktop instead of floating on top). |
The default profile, **`desktop`**, exposes 37 tools on macOS: observation,
coordinates, OCR for apps that draw their own UI, waits and the session target.
Set `COMPUTER_USE_PROFILE=full` for every tool (78 on macOS), or `core`, `ax`,
`scripting` and `windows-admin` for other bounds. Tools that don't exist on your
platform are not listed. Optional host services add their own tools. The server's
instructions to the agent are generated for your platform and profile, so they only
name tools that are there.
Try asking your agent:
- “Find my installed Office apps and explain how you can automate them.”
- “Open a new text document, write a short meeting agenda and save it.”
- “Inspect this application's controls and fill out the form I describe.”
- “Create a new painting with a swirling night sky and export it.”
Always specify the intended app or window. Prefer connectors, APIs or filesystem
operations for work that does not need a desktop. Platform and application
support varies; an available tool is not a promise that every app exposes usable
controls.
## Use it from code
```js
import { createComputerUseServer } from '@zavora-ai/computer-use-mcp'
import { connectInProcess } from '@zavora-ai/computer-use-mcp/client'
// The default profile is `desktop`; pass { profile: 'full' } for every tool.
const client = await connectInProcess(createComputerUseServer())
try {
const apps = await client.discoverApplications({
query: 'office',
include_capabilities: true,
})
console.log(apps.content)
} finally {
await client.close()
}
```
[Application examples](examples/README.md) · [AI agent examples](agents/README.md)
· [Responses showcases](agents/openai-agent/README.md)
## Architecture
```mermaid
flowchart TD
Agent[AI agent or MCP client] --> Server[MCP server]
Server --> Policy[Tool permissions and target checks]
Policy --> Session[Desktop session]
Session --> Native[Native mouse, keyboard and screen tools]
Session --> AX[Accessibility controls]
Session --> Scripts[App scripting]
Native --> Apps[Desktop applications]
AX --> Apps
Scripts --> Apps
Server --> Hosts[Optional host services]
Hosts --> Browser[Isolated browser]
Hosts --> Tasks[Persistent tasks and session console]
```
The TypeScript server handles tool requests, permissions and cancellation. A Rust
native module connects to the operating system. Optional host services provide
longer-lived sessions and isolated execution. Results return to the agent so it
can observe, act and verify.
For protocol details, deployment controls and the full component diagram, see
[Architecture](docs/ARCHITECTURE.md) and the [host integration guide](docs/STRATEGY.md).
Legacy MCP clients remain supported; newer protocol features are opt-in.
## Develop locally
```sh
git clone https://github.com/zavora-ai/computer-use-mcp.git
cd computer-use-mcp
npm ci
npm run build:ts
npm test
```
For live desktop use from a clone, install Rust and build your platform's native
module: `npm run build:native` on macOS, `npm run build:native:win` on Windows x64,
or `npm run build:native:linux` on Linux. Windows arm64 has `build:native:win:arm64`.
The packaged npm install does not require a local Rust build.
`npm run smoke` checks the local desktop; `npm run test:browser` tests a disposable
browser fixture. These live checks need their documented platform prerequisites.
[Development scripts](scripts/README.md) lists the maintained commands.
## Documentation and support
- [v7.7.0 release notes](docs/releases/v7.7.0.md): honest results, `run_steps`, a person in control by default, faster captures, Linux on Wayland, a native package per platform
- [v7.6.1 release notes](docs/releases/v7.6.1.md): the review fixes and the desktop status card
- [v7.6.0 release notes](docs/releases/v7.6.0.md) (includes 7.5.0: [its notes](docs/releases/v7.5.0.md))
- [v7.4.0 release notes and rollout checklist](docs/releases/v7.4.0.md)
- [Reviews](docs/reviews/): the 2026-10-09 review and design pass behind 7.6.0, the 2026-10-11 three-pass review behind 7.6.1, and the 2026-10-23 competitor study behind 7.7.0
- [v8 roadmap](docs/specs/v8/roadmap.md): what comes next
- [Tool usage and agent setup](AGENTS.md)
- [Testing on Linux](docs/TESTING-LINUX.md) and [in UTM virtual machines](docs/TESTING-IN-UTM.md)
- [Architecture and platform support](docs/ARCHITECTURE.md)
- [Sessions, Tasks, browser isolation and supervision](docs/STRATEGY.md)
- [Token and interaction efficiency](docs/EFFICIENCY.md)
- [Security and reporting vulnerabilities](SECURITY.md)
- [Report a bug or request a feature](https://github.com/zavora-ai/computer-use-mcp/issues)
MIT licensed. Desktop access can change real applications and files; configure
permissions and target applications deliberately. The bundled HTTP server is
loopback-only. Exposing a remote endpoint requires host-owned authentication.