OS-level desktop automation tool server. 42 tools for controlling any application on Windows, macOS, and Linux. Model-agnostic — works with any AI that can do function calling via REST or MCP (Claude, GPT, Gemini, Llama, Mistral, or plain HTTP). No built-in LLM in serve/mcp mode. You are the brain. ClawdCursor is the hands.
Instrucciones de origen · Vista previa de solo lectura
name
clawdcursor
version
0.7.5
description
OS-level desktop automation tool server. 42 tools for controlling any application on Windows, macOS, and Linux. Model-agnostic — works with any AI that can do function calling via REST or MCP (Claude, GPT, Gemini, Llama, Mistral, or plain HTTP). No built-in LLM in serve/mcp mode. You are the brain. ClawdCursor is the hands.
homepage
https://clawdcursor.com
source
https://github.com/AmrDab/clawdcursor
privacy
All processing runs locally. Server binds to 127.0.0.1 only — not network-accessible. No telemetry, no analytics. Screenshots stay in memory. In agent mode (start), screenshots/text are sent only to the user's configured AI provider.
If a human can do it on a screen, you can too. No API? No integration? No problem.
USE AS A FALLBACK — NOT FIRST CHOICE
Before reaching for any ClawdCursor tool, ask:
Is there a native API? (Gmail API, GitHub API, Slack API) → use the API
Is there a CLI? (git, npm, curl) → use the CLI
Can you edit the file directly? → do that
Is there a browser automation layer? (Playwright, Puppeteer) → use that
None of the above work? Now use ClawdCursor. It's for the last mile.
Modes at a Glance
Mode
Command
Brain
Tools available
serve
clawdcursor serve
You (REST client)
All 42 tools via HTTP
mcp
clawdcursor mcp
You (MCP client)
All 42 tools via MCP stdio
start
clawdcursor start
Built-in LLM pipeline
All 42 tools + autonomous agent
In serve and mcp modes: you reason, ClawdCursor acts. There is no built-in LLM. You call tools, interpret results, decide next steps.
Connecting
Option A — REST (clawdcursor serve)
clawdcursor serve # starts on http://127.0.0.1:3847
All POST endpoints require: Authorization: Bearer <token> (token saved to ~/.clawdcursor/token)
GET /tools → all tool schemas (OpenAI function-calling format)
POST /execute/{name} → run a tool: {"param": "value"}
GET /health → {"status":"ok","version":"0.7.5"}
GET /docs → full documentation
Example:
POST /execute/get_windows {}
POST /execute/mouse_click {"x": 640, "y": 400}
POST /execute/type_text {"text": "hello world"}
If the server isn't running, start it yourself — don't ask the user:
clawdcursor serve
# wait 2 seconds, then verify: GET /health
Works with Claude Code, Cursor, Windsurf, Zed, or any MCP-compatible client. All 42 tools are exposed identically.
Option C — Autonomous agent (clawdcursor start)
POST /task {"task": "Open Notepad and write Hello"} → submit task
GET /status → {"status": "acting"} | "idle" | "waiting_confirm"
POST /confirm {"approved": true} → approve safety-gated action
POST /abort → stop current task
Use delegate_to_agent tool to submit tasks from within MCP/REST sessions. Requires clawdcursor start running on port 3847.
Polling pattern:
POST /task {"task": "...", "returnPartial": true}
→ poll GET /status every 2s:
"acting" → still running, keep polling
"waiting_confirm" → STOP. Ask user → POST /confirm {"approved": true}
"idle" → done, check GET /task-logs for result
→ if 60s+ with no progress: POST /abort, retry with simpler phrasing
returnPartial mode — send {"returnPartial": true} with POST /task:
ClawdCursor skips Stage 3 (expensive vision) and returns control to you if Stage 2 fails:
{"partial":true,"stepsCompleted":[...],"context":"got stuck on dialog"}
You finish the task with MCP tools, then call POST /learn to save what worked.
POST /learn — adaptive learning:
After completing a task with your own tool calls, teach ClawdCursor for next time:
POST /learn
{"processName":"EXCEL","task":"create table with headers","actions":[{"action":"key","description":"Ctrl+Home to go to A1"},{"action":"type","description":"Type header name"},{"action":"key","description":"Tab to next column"}],"shortcuts":{"next_cell":"Tab","next_row":"Enter"},"tips":["Use Tab between columns, Enter between rows"]}
This enriches the app's guide JSON. Stage 2 reads it on the next run — no vision fallback needed.
The Universal Loop
Every GUI task follows the same pattern regardless of transport:
1. ORIENT → read_screen() or get_windows() see what's open and focused
2. ACT → smart_click() / smart_type() / key_press() do the thing
3. VERIFY → check return value → window state → text check → screenshot
4. REPEAT → until done
Verification (cheapest to most expensive)
Tool return value — every tool reports success/failure. Check it first.
Window state — get_active_window(), get_windows() — did a dialog appear? Did the title change?
Text check — read_screen() or smart_read() — is the expected text visible?
Screenshot — desktop_screenshot() — only when text methods fail. Costs the most.
read_screen() → FIRST. Accessibility tree: buttons, inputs, text, with coords.
Fast, structured, works on native apps.
ocr_read_screen() → When a11y tree is empty (canvas UIs, image-based apps).
smart_read() → Combines OCR + a11y. Good first call when unsure.
desktop_screenshot() → LAST RESORT. Only when you need pixel-level visual detail.
desktop_screenshot_region(x,y,w,h) → Zoomed crop when you need detail in one area.
Clicking
smart_click("Save") → FIRST. Finds by label/text via OCR + a11y, clicks.
Pass processId to target the right window.
invoke_element(name="Save") → When you know the exact automation ID from read_screen.
cdp_click(text="Submit") → Browser elements. Requires cdp_connect() first.
mouse_click(x, y) → LAST RESORT. Raw coordinates from a screenshot.
Typing
smart_type("Email", "user@x.com") → FIRST. Finds field by label, focuses, types.
cdp_type(label="Email", text="…") → Browser inputs. Requires cdp_connect() first.
type_text("hello") → Clipboard paste into whatever is focused.
Use after manually focusing with smart_click.
Browser / CDP
1. navigate_browser(url) → opens URL, auto-enables CDP
2. cdp_connect() → connect to browser DevTools Protocol
3. cdp_page_context() → list interactive elements on page
4. cdp_read_text() → extract DOM text (returns empty on canvas apps → use OCR)
5. cdp_click(text="…") → click by visible text
6. cdp_type(label, text) → fill input by label
7. cdp_evaluate(script) → run JavaScript in page context
8. cdp_scroll(direction, px) → scroll page via DOM (not mouse wheel)
9. cdp_list_tabs() → list all open tabs
10. cdp_switch_tab(target) → switch to a specific tab
If CDP isn't connected, switch tabs with keyboard:
get_windows() → list all open windows (use to find PIDs)
get_active_window() → what's in the foreground right now
focus_window(processName="Discord") → bring to front (auto-minimizes phantom off-screen windows)
minimize_window(processName="calc") → minimize a window — 1 call, cross-platform
also accepts: processId, title
Rule: Always focus_window() before key_press() or type_text(). Keystrokes go to whatever has focus — if that's your terminal, not the target app.
Canvas apps (Google Docs, Figma, Notion)
DOM has no readable text. Pattern:
ocr_read_screen() → read content (DOM extraction fails)
mouse_click(x, y) → click into the canvas area
type_text("your text") → clipboard paste works even on canvas
delegate_to_agent("Open Gmail, find latest email from Stripe, forward to billing@x.com")
→ poll GET /status every 2s
→ if waiting_confirm: ask user → POST /confirm {"approved": true}
→ if idle: task done
Set env var: ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, MOONSHOT_API_KEY, etc.
Varies
OpenClaw users
Auto-detected from ~/.openclaw/agents/main/auth-profiles.json
No extra setup
Run clawdcursor doctor to auto-detect and validate providers.
Security
Network isolation: Binds to 127.0.0.1 only. Verify: netstat -an | findstr 3847 — should show 127.0.0.1:3847, never 0.0.0.0:3847
Ollama: 100% offline. Screenshots stay in RAM, never leave the machine.
Cloud providers: Screenshots/text sent only to your configured provider. No telemetry, no analytics, no third-party logging.
Token auth: All mutating POST endpoints require Authorization: Bearer <token>. Token at ~/.clawdcursor/token.
Safety tiers: Auto / Preview / Confirm. Agents must never self-approve Confirm actions.
Coordinate System
All mouse tools use image-space coordinates from a 1280px-wide viewport — matching screenshots from desktop_screenshot. DPI scaling is handled automatically. Do not pre-scale coordinates.
Safety
Tier
Actions
Behavior
🟢 Auto
Navigation, reading, opening apps
Runs immediately
🟡 Preview
Typing, form filling
Logged
🔴 Confirm
Send, delete, purchase
Pauses — always ask user first
Never self-approve Confirm actions.
Alt+F4 and Ctrl+Alt+Delete are blocked.
Server binds to 127.0.0.1 only.
First run requires explicit user consent for desktop control.
Error Recovery
Problem
Fix
Port 3847 not responding
clawdcursor serve — wait 2s — GET /health
401 Unauthorized
Token changed — read ~/.clawdcursor/token and use fresh value
CDP not available
Chrome must be open. navigate_browser(url) auto-enables it.
CDP on wrong tab
cdp_list_tabs() → cdp_switch_tab(target)
focus_window fails
get_windows() to confirm title/processName, then retry
smart_click can't find element
read_screen() for coords → mouse_click(x, y)
key_press goes to wrong window
You skipped focus_window — always focus first
cdp_read_text returns empty
Canvas app — use ocr_read_screen() instead
Same action fails 3+ times
Try a completely different approach
Platform Support
Platform
A11y
OCR
CDP
Windows (x64/ARM64)
PowerShell + .NET UIA
Windows.Media.Ocr
Chrome/Edge
macOS (Intel/Apple Silicon)
JXA + System Events
Apple Vision
Chrome/Edge
Linux (x64/ARM64)
AT-SPI
Tesseract
Chrome/Edge
macOS: Grant Accessibility in System Settings → Privacy → Accessibility.
Linux:sudo apt install tesseract-ocr for OCR support.
