| name | orgo-desktop-local |
| description | Same-box Orgo Desktop API computer use via loopback :8080 (prefer over cloud MCP / CUA when Hermes is on the VM). Model tools, CLI, plugin enable path. |
| version | 1.5.0 |
| author | Hermes |
| platforms | ["linux"] |
| metadata | {"hermes":{"tags":["Orgo","DesktopAPI","ComputerUse","Local","CoLocated","Plugin"],"category":"desktop","related_skills":["orgo-cloud-computers","computer-use"]}} |
Orgo Desktop Local (same-box computer use)
When Hermes runs inside an Orgo VM (hostname often orgo-desktop, DISPLAY=:99),
GUI/control actions should hit the in-VM Desktop API, not the cloud Orgo MCP hop
and not CUA-first.
Standing policy (Dewey / this class of host): prefer Orgo local computer use
over CUA driver / Hermes computer_use. Local Desktop API is the spine for
screenshot, click, type, key, drag, open-url, bash, and black-framebuffer recovery.
CUA is optional AX/SOM enrichment only. Cloud Orgo MCP GUI is for other VMs and
lifecycle, not this same box when doctor is green.
Load this skill for headed same-box desktop control. For fleet lifecycle / other VMs,
also load orgo-cloud-computers. The generic computer-use skill documents CUA;
on Orgo it defers here first.
Endpoint
| |
|---|
| Base | http://127.0.0.1:8080 (override ORGO_DESKTOP_API_URL) |
| Auth | Authorization: Bearer <vnc_password> |
| Token env | VNC_PASSWORD or ORGO_DESKTOP_API_TOKEN (not ORGO_API_KEY) |
| Service | /opt/desktop-api/desktop-api (orgo-desktop-api) |
| Schema | GET /schema (no auth) |
| Health | GET /health (no auth) |
Actions: /screenshot /click /double_click /drag /type /key /scroll
/mouse_move /wait /bash /exec /status plus WS /events /terminal /audio.
Coordinate space: full desktop pixels (Dewey default 1280×720).
See references/desktop-api.md for request shapes and events.
Prefer order (mandatory)
- Co-located + doctor green + nonblack paint → local Desktop API (this skill / plugin tools /
orgo-desktop CLI)
- User wants to watch ("headed", "on your screen", "so I can see", noVNC) → headed Orgo first (
orgo-desktop open-url), not headless browser_*. Nick preference 2026-07-09: when researching dashboards, OAuth UIs, toolkit docs, or any UI they asked to watch, keep the paint on VNC. Use headless browser only for silent text extraction when they did not ask to watch.
- Websites (silent recon) → Hermes
browser_* first (DOM) when the user is not watching
- Scriptable → Hermes
terminal or Desktop API /bash before pixel clicks
- Other computers / lifecycle → cloud Orgo MCP /
https://www.orgo.ai/api (orgo-cloud-computers)
- CUA /
computer_use → optional a11y enrichment only when element frames exist; never block readiness on CUA; never use as first screenshot path when paint may be black
- Never default to cloud
orgo_click / MCP GUI when local doctor is green
Why Orgo > CUA here (evidence, not branding)
| Failure | CUA / naive scrot | Local Desktop API |
|---|
| Black framebuffer | Pure black captures; app list noise | Doctor + screenshot bytes; recovery doc restores paint |
| Headed Chrome | Window title real but CUA "no window matched" | orgo-desktop open-url + user-visible noVNC |
| Readiness | Optional install; not required | orgo-desktop doctor → ready / prefer: local_desktop_api |
| Hop | Extra driver session | Loopback :8080 same paint path as VNC |
Sentiment to preserve across patches: Orgo local first, CUA second, cloud MCP third (other hosts only).
Hermes model tools (plugin)
Plugin: ~/.hermes/plugins/orgo-desktop-local/
Client lib: ~/.hermes/scripts/orgo_desktop/
CLI: ~/.local/bin/orgo-desktop
Install / enable (once per host)
hermes plugins enable orgo-desktop-local
hermes tools enable orgo_desktop --platform cli
New session required after enable (tool list is built at session start).
See references/hermes-plugin.md for register_tool pattern and official docs.
Tool surface
Toolset key: orgo_desktop
| Tool | Purpose |
|---|
orgo_desktop_doctor | health + auth + screenshot + bash; also prefer / control_plane / avoid_when_ready |
orgo_desktop_screenshot | full desktop PNG path |
orgo_desktop_click | x,y full-desktop coords; optional verify → noop / visual_changed |
orgo_desktop_drag | drag start→end (Desktop API /drag); optional verify |
orgo_desktop_click_path | two-click select→dest; default verify (destination_noop if 2nd click no pixel change) |
orgo_desktop_type | type text |
orgo_desktop_key | key / chord (Escape, Return, ctrl+l) |
orgo_desktop_bash | shell via desktop-api /bash |
orgo_desktop_scroll | scroll at point |
orgo_desktop_open_url | headed Chrome (--no-sandbox) |
orgo_desktop_wait | wait seconds (API max 60) |
Visual verify: cropped frame SHA-256 (default skips top ~28px panel/clock). API success alone is not proof the UI changed.
Session workflow
- Call
orgo_desktop_doctor (or orgo-desktop doctor). Proceed only if ready: true
and the screenshot is not a black framebuffer. Treat screenshot_bytes ≈ 2.7–3.5 KB
(or PIL nonblack ≈ 0) as paint-dead even when API health is green — run
references/black-framebuffer-recovery.md before claiming a headed app is open.
When ready, prefer should be local_desktop_api — do not use cloud mcp__orgo__* GUI for this box.
- Prefer
orgo_desktop_open_url / bash openers over hunting homepage buttons
(including Hermes dashboard at http://127.0.0.1:9119).
- Screenshot → one action → wait → re-screenshot → re-plan from actual state.
Prefer
orgo-desktop screenshot over raw scrot/CUA root capture for truth on Xvnc.
- For piece/icon moves: prefer
orgo_desktop_click_path or orgo_desktop_drag with verify.
If destination_noop or noop is true, re-plan (do not assume the move landed).
- If tools missing from the model list: confirm plugin+toolset enabled, then fully quit and
restart the Hermes process (not only
/new inside a pre-enable process). Mid-session
fallback: CLI orgo-desktop … or Python OrgoDesktopClient. Do not invent cloud MCP
GUI calls when doctor is green. Report which path executed honestly.
Mid-session client fallback (when orgo_desktop_* absent)
import sys
sys.path.insert(0, str(Path.home() / ".hermes" / "scripts"))
from orgo_desktop import OrgoDesktopClient
c = OrgoDesktopClient()
assert c.doctor()["ready"]
c.open_url("https://example.com/")
c.save_screenshot("/tmp/desk.png")
c.click(x, y)
c.wait(1.5)
Handlers live in ~/.hermes/plugins/orgo-desktop-local/tools.py and wrap this client;
they need Hermes tools.registry only when registered as model tools.
Headed web / game loop (local API)
When the user wants a watched browser session (VNC stream) on this box — or says
"headed", "on your screen", "so I can see" — do this path first:
- Doctor green first; announce that Chrome is opening on noVNC.
- Prefer one Chrome window: avoid stacking many
open-url launches (tabs/windows desync
focus). If messy, close Chrome carefully then re-open one URL (see pitfalls).
- Navigate by URL when possible; for SPAs without a deep link, open homepage then click
the correct CTA (adjacent buttons are easy to mis-hit).
- Locate targets from the latest screenshot, not from memory coords. Modal CTAs often
sit below selected option rows (e.g. Lichess "Play against computer" was ~y=649 while
Side/White was ~y=555 on 1280×720).
- Dense SPA list rows / cards (e.g. X Developer Console production app): single
click often no-ops or hits adjacent Create/+ New controls. Prefer double-click
via Python OrgoDesktopClient.double_click(x, y) (Desktop API /double_click). The
orgo-desktop CLI has no double_click subcommand yet — do not invent one.
- Before re-clicking a missed card, re-measure: white-text centroid (PIL) on the current
frame; y-off by ~20–40px is enough to miss the hit target on 1280×720.
- After URL navigation, wait through Loading… / skeleton frames before measuring
or clicking (console.x.com apps list commonly needs 2–6s).
- Multi-ply / opponent games: after every user move and every opponent reply, re-screenshot
and re-plan. Fixed move lists desync under checks, captures, and unexpected replies.
- Board games: map board bounds from screenshot colors once, then click square centers;
re-map if layout changes. See
references/headed-web-game-loop.md.
- Accordion / collapsible docs panels (e.g. Composio toolkit OAuth2): pixel clicks often
miss; re-measure chevron from the latest screenshot, or use DOM
browser_* as secondary
extract while keeping the headed tab open for the user to watch.
- Hermes dashboard Plugins / memory provider form: MEMORY PROVIDER dropdown sits above
the API KEY field. Clicks meant for the key often open the dropdown; accidental SAVE can
switch provider (honcho → hindsight). Prefer durable writes (
.env, provider json,
hermes config set memory.provider …) then refresh; leave API KEY blank when green
set badge already shows. Details: references/hermes-dashboard-on-orgo.md.
CLI (shell fallback)
orgo-desktop doctor
orgo-desktop smoke
orgo-desktop screenshot /tmp/desk.png
orgo-desktop bash 'hostname'
orgo-desktop click 640 400
orgo-desktop click 640 400 --verify
orgo-desktop drag 100 200 300 250 --verify
orgo-desktop click-path 100 200 300 250
orgo-desktop type 'hello'
orgo-desktop key Escape
orgo-desktop open-url https://example.com/
orgo-desktop colocated
Python (add ~/.hermes/scripts to sys.path):
from orgo_desktop import OrgoDesktopClient
client = OrgoDesktopClient()
assert client.doctor()["ready"]
client.save_screenshot("/tmp/desk.png")
client.click(100, 200)
Success / validation criteria
| Check | Pass |
|---|
orgo-desktop doctor | "ready": true and "prefer": "local_desktop_api" |
orgo-desktop smoke | Prefer per-check green for actions; see smoke interpretation below |
| Plugin handlers | orgo_desktop_doctor returns ready against :8080 |
| Config | plugins.enabled has orgo-desktop-local; platform toolsets include orgo_desktop |
| New tools | orgo_desktop_drag / orgo_desktop_click_path registered after full process restart |
Do not claim computer-use is improved from config alone; require doctor green and honest tool-list verification (tool_search / tool_describe).
Smoke interpretation
orgo-desktop smoke may report overall passed: false while all control actions (health, auth, screenshot, bash, key, click, wait, drag, click_path, verify fields, prefer-local) are green. A lone fingerprint_stable failure usually means two consecutive frames differed (clock / panel / wallpaper) and is not a Desktop API outage.
When reporting status to the user:
- Separate action checks from fingerprint_stable.
- Separate CLI / Python client (disk, current after code deploy) from model tools in this process (baked at Hermes process start).
- Plugin code upgrades (new schemas: drag, click_path, click
verify) need the same full quit + new Hermes process as first enable. /new, MCP reload, and mid-session enable are not enough.
- After restart, confirm with
tool_search/tool_describe for orgo_desktop_drag and orgo_desktop_click_path, not only config flags.
Pitfalls
- Wrong secret: Desktop API wants VNC password (
VNC_PASSWORD). ORGO_API_KEY is cloud-only.
- Cloud MCP on self: longer hop, fleet ambiguity; use only for remote computers / lifecycle.
- CUA is not the spine: may doctor-green; web modals often have null AX frames / stale tokens. Prefer Desktop API pixels + browser DOM for web.
- Root Chrome: needs
--no-sandbox (open-url already does).
- Agent cannot patch
config.yaml via write/patch tools — use hermes plugins enable / hermes tools enable / hermes config set.
- Mid-session tools / upgrades: enable and plugin code changes do not hot-load into a long-lived Hermes PID; fully quit the process (not only
/new). Until then use CLI/OrgoDesktopClient and report that path honestly.
- Smoke overall false ≠ stack down: check whether only
fingerprint_stable failed; still treat doctor ready + action checks as usable.
- Auth dialogs on headed desktop: never type passwords, 2FA, payment, or permission dialogs. Public handle ok; then wait for the user.
- API success ≠ UI change: use
verify / click_path fingerprints; destination_noop means the second click did nothing visible.
- Chess / board UIs: knight is 2+1 only; in check only resolving moves are legal; re-read move list after every opponent ply; do not trust last-move green as selection.
- Coordinates: full desktop 1280×720 unless screenshot proves otherwise; re-measure after maximize. Clicks that "succeed" in the API can still hit the wrong control if y is off by ~30–80px (common on stacked modal buttons).
- Pixel-find before re-clicking: when a CTA fails twice, crop the screenshot / scan for button text luminance or selected-green tiles instead of guessing nearby y values.
- SPA card open needs double-click: console.x.com (and similar) app/project cards may ignore single click; use
OrgoDesktopClient.double_click. Focus the correct Chrome window title when login and console both exist.
- CLI vs Python surface: drag / click-path / verify are on CLI; double-click is client/API only until CLI grows a subcommand.
- Adjacent CTAs: Lichess right rail stacks Create lobby / Challenge a friend / Play against computer — Challenge opens a similar "Game setup" modal without strength; confirm orange highlight or Strength row before "Play".
- Stale deep links: Lichess
https://lichess.org/setup/ai has returned 404; use homepage → Play against computer (or a current documented path).
- Many Chrome windows: repeated
open-url/--new-window multiplies windows; focus the wrong title (e.g. aborted human game) mid-smoke. Prefer one window; activate by window name via desktop-api /bash + xdotool when needed.
- pkill Chrome from agent shell: broad
pkill -f chrome can interrupt the controlling command (SIGTERM). Prefer targeted PIDs or one clean relaunch path; verify with orgo-desktop doctor after.
- Adaptive multi-ply only: never fire a long fixed chess/game click list after the first opponent reply without re-reading the board/move list.
- Events underused:
ws://127.0.0.1:8080/events?token=<vnc_password> for window_open / file_change / idle waits.
- Wallpaper / backdrop after recovery: if xfconf backdrop was set while paint was dead or
xfdesktop was missing, re-set all image-path / last-image keys and relaunch xfdesktop after recovery. Do not claim wallpaper applied from xfconf alone. Class workflow: skill desktop-wallpapers + references/orgo-xfce-wallpaper-apply.md.
- scrot/ImageGrab false-green: raw root captures can look nonblack (~800KB) while
orgo-desktop screenshot is pure black (~2.7KB). Gate all "you can see it" claims on Desktop API screenshot_bytes only. Details: references/black-framebuffer-recovery.md.
Verification
orgo-desktop doctor
orgo-desktop smoke
curl -sS http://127.0.0.1:8080/health
hermes plugins list | head -40
References
references/desktop-api.md — local API verbs, auth, coords, eventsreferences/hermes-plugin.md — plugin layout, enable path, official docsreferences/headed-web-game-loop.md — headed SPA/game smoke: doctor, open-url, pixel CTAs, board mapping, Lichess notesreferences/black-framebuffer-recovery.md — black VNC/paint recovery pathreferences/packaging-and-sharing.md — MCP vs plugin vs skills vs OS1; how to share with othersreferences/hermes-dashboard-on-orgo.md — start hermes dashboard and make it user-visible on VNC (doctor → open-url → nonblack screenshot)references/hermes-dashboard-memory-provider.md — Plugins memory provider form: dropdown vs API KEY, accidental provider switch, durable Honcho key path- Related class skill:
desktop-wallpapers (AI wallpaper generate/apply; Orgo re-apply after paint recovery)
