| name | orgo-desktop-local |
| description | Same-box Orgo Desktop API computer use via loopback :8080 (prefer over cloud MCP / CUA when Hermes is on the VM). Model tools, CLI, plugin enable path. |
| version | 1.4.1 |
| author | Hermes |
| platforms | ["linux"] |
| metadata | {"hermes":{"tags":["Orgo","DesktopAPI","ComputerUse","Local","CoLocated","Plugin"],"category":"desktop","related_skills":["orgo-cloud-computers","computer-use"]}} |
Orgo Desktop Local (same-box computer use)
When Hermes runs inside an Orgo VM (hostname often orgo-desktop, DISPLAY=:99),
GUI/control actions should hit the in-VM Desktop API, not the cloud Orgo MCP hop
and not CUA-first.
Standing policy (Dewey / this class of host): prefer Orgo local computer use
over CUA driver / Hermes computer_use. Local Desktop API is the spine for
screenshot, click, type, key, drag, open-url, bash, and black-framebuffer recovery.
CUA is optional AX/SOM enrichment only. Cloud Orgo MCP GUI is for other VMs and
lifecycle, not this same box when doctor is green.
Load this skill for headed same-box desktop control. For fleet lifecycle / other VMs,
also load orgo-cloud-computers. The generic computer-use skill documents CUA;
on Orgo it defers here first.
Endpoint
| |
|---|
| Base | http://127.0.0.1:8080 (override ORGO_DESKTOP_API_URL) |
| Auth | Authorization: Bearer <vnc_password> |
| Token env | VNC_PASSWORD or ORGO_DESKTOP_API_TOKEN (not ORGO_API_KEY) |
| Service | /opt/desktop-api/desktop-api (orgo-desktop-api) |
| Schema | GET /schema (no auth) |
| Health | GET /health (no auth) |
Actions: /screenshot /click /double_click /drag /type /key /scroll
/mouse_move /wait /bash /exec /status plus WS /events /terminal /audio.
Coordinate space: full desktop pixels (Dewey default 1280×720).
See references/desktop-api.md for request shapes and events.
Prefer order (mandatory)
- Co-located + doctor green + nonblack paint → local Desktop API (this skill / plugin tools /
orgo-desktop CLI)
- Websites → Hermes
browser_* first (DOM), desktop only when headed watch / native chrome needed
- Scriptable → Hermes
terminal or Desktop API /bash before pixel clicks
- Other computers / lifecycle → cloud Orgo MCP /
https://www.orgo.ai/api (orgo-cloud-computers)
- CUA /
computer_use → optional a11y enrichment only when element frames exist; never block readiness on CUA; never use as first screenshot path when paint may be black
- Never default to cloud
orgo_click / MCP GUI when local doctor is green
Why Orgo > CUA here (evidence, not branding)
| Failure | CUA / naive scrot | Local Desktop API |
|---|
| Black framebuffer | Pure black captures; app list noise | Doctor + screenshot bytes; recovery doc restores paint |
| Headed Chrome | Window title real but CUA "no window matched" | orgo-desktop open-url + user-visible noVNC |
| Readiness | Optional install; not required | orgo-desktop doctor → ready / prefer: local_desktop_api |
| Hop | Extra driver session | Loopback :8080 same paint path as VNC |
Sentiment to preserve across patches: Orgo local first, CUA second, cloud MCP third (other hosts only).
Hermes model tools (plugin)
Plugin: ~/.hermes/plugins/orgo-desktop-local/
Client lib: ~/.hermes/scripts/orgo_desktop/
CLI: ~/.local/bin/orgo-desktop
Install / enable (once per host)
hermes plugins enable orgo-desktop-local
hermes tools enable orgo_desktop --platform cli
New session required after enable (tool list is built at session start).
See references/hermes-plugin.md for register_tool pattern and official docs.
Tool surface
Toolset key: orgo_desktop
| Tool | Purpose |
|---|
orgo_desktop_doctor | health + auth + screenshot + bash; also prefer / control_plane / avoid_when_ready |
orgo_desktop_screenshot | full desktop PNG path |
orgo_desktop_click | x,y full-desktop coords; optional verify → noop / visual_changed |
orgo_desktop_drag | drag start→end (Desktop API /drag); optional verify |
orgo_desktop_click_path | two-click select→dest; default verify (destination_noop if 2nd click no pixel change) |
orgo_desktop_type | type text |
orgo_desktop_key | key / chord (Escape, Return, ctrl+l) |
orgo_desktop_bash | shell via desktop-api /bash |
orgo_desktop_scroll | scroll at point |
orgo_desktop_open_url | headed Chrome (--no-sandbox) |
orgo_desktop_wait | wait seconds (API max 60) |
Visual verify: cropped frame SHA-256 (default skips top ~28px panel/clock). API success alone is not proof the UI changed.
Session workflow
- Call
orgo_desktop_doctor (or orgo-desktop doctor). Proceed only if ready: true
and the screenshot is not a black framebuffer. Treat screenshot_bytes ≈ 2.7–3.5 KB
(or PIL nonblack ≈ 0) as paint-dead even when API health is green — run
references/black-framebuffer-recovery.md before claiming a headed app is open.
When ready, prefer should be local_desktop_api — do not use cloud mcp__orgo__* GUI for this box.
- Prefer
orgo_desktop_open_url / bash openers over hunting homepage buttons
(including Hermes dashboard at http://127.0.0.1:9119).
- Screenshot → one action → wait → re-screenshot → re-plan from actual state.
Prefer
orgo-desktop screenshot over raw scrot/CUA root capture for truth on Xvnc.
- For piece/icon moves: prefer
orgo_desktop_click_path or orgo_desktop_drag with verify.
If destination_noop or noop is true, re-plan (do not assume the move landed).
- If tools missing from the model list: confirm plugin+toolset enabled, then fully quit and
restart the Hermes process (not only
/new inside a pre-enable process). Mid-session
fallback: CLI orgo-desktop … or Python OrgoDesktopClient. Do not invent cloud MCP
GUI calls when doctor is green. Report which path executed honestly.
Mid-session client fallback (when orgo_desktop_* absent)
import sys
sys.path.insert(0, str(Path.home() / ".hermes" / "scripts"))
from orgo_desktop import OrgoDesktopClient
c = OrgoDesktopClient()
assert c.doctor()["ready"]
c.open_url("https://example.com/")
c.save_screenshot("/tmp/desk.png")
c.click(x, y)
c.wait(1.5)
Handlers live in ~/.hermes/plugins/orgo-desktop-local/tools.py and wrap this client;
they need Hermes tools.registry only when registered as model tools.
Headed web / game loop (local API)
When the user wants a watched browser session (VNC stream) on this box:
- Doctor green first.
- Prefer one Chrome window: avoid stacking many
open-url launches (tabs/windows desync
focus). If messy, close Chrome carefully then re-open one URL (see pitfalls).
- Navigate by URL when possible; for SPAs without a deep link, open homepage then click
the correct CTA (adjacent buttons are easy to mis-hit).
- Locate targets from the latest screenshot, not from memory coords. Modal CTAs often
sit below selected option rows (e.g. Lichess "Play against computer" was ~y=649 while
Side/White was ~y=555 on 1280×720).
- Dense SPA list rows / cards (e.g. X Developer Console production app): single
click often no-ops or hits adjacent Create/+ New controls. Prefer double-click
via Python OrgoDesktopClient.double_click(x, y) (Desktop API /double_click). The
orgo-desktop CLI has no double_click subcommand yet — do not invent one.
- Before re-clicking a missed card, re-measure: white-text centroid (PIL) on the current
frame; y-off by ~20–40px is enough to miss the hit target on 1280×720.
- After URL navigation, wait through Loading… / skeleton frames before measuring
or clicking (console.x.com apps list commonly needs 2–6s).
- Multi-ply / opponent games: after every user move and every opponent reply, re-screenshot
and re-plan. Fixed move lists desync under checks, captures, and unexpected replies.
- Board games: map board bounds from screenshot colors once, then click square centers;
re-map if layout changes. See
references/headed-web-game-loop.md.
CLI (shell fallback)
orgo-desktop doctor
orgo-desktop smoke
orgo-desktop screenshot /tmp/desk.png
orgo-desktop bash 'hostname'
orgo-desktop click 640 400
orgo-desktop click 640 400 --verify
orgo-desktop drag 100 200 300 250 --verify
orgo-desktop click-path 100 200 300 250
orgo-desktop type 'hello'
orgo-desktop key Escape
orgo-desktop open-url https://example.com/
orgo-desktop colocated
Python (add ~/.hermes/scripts to sys.path):
from orgo_desktop import OrgoDesktopClient
client = OrgoDesktopClient()
assert client.doctor()["ready"]
client.save_screenshot("/tmp/desk.png")
client.click(100, 200)
Success / validation criteria
| Check | Pass |
|---|
orgo-desktop doctor | "ready": true and "prefer": "local_desktop_api" |
orgo-desktop smoke | Prefer per-check green for actions; see smoke interpretation below |
| Plugin handlers | orgo_desktop_doctor returns ready against :8080 |
| Config | plugins.enabled has orgo-desktop-local; platform toolsets include orgo_desktop |
| New tools | orgo_desktop_drag / orgo_desktop_click_path registered after full process restart |
Do not claim computer-use is improved from config alone; require doctor green and honest tool-list verification (tool_search / tool_describe).
Smoke interpretation
orgo-desktop smoke may report overall passed: false while all control actions (health, auth, screenshot, bash, key, click, wait, drag, click_path, verify fields, prefer-local) are green. A lone fingerprint_stable failure usually means two consecutive frames differed (clock / panel / wallpaper) and is not a Desktop API outage.
When reporting status to the user:
- Separate action checks from fingerprint_stable.
- Separate CLI / Python client (disk, current after code deploy) from model tools in this process (baked at Hermes process start).
- Plugin code upgrades (new schemas: drag, click_path, click
verify) need the same full quit + new Hermes process as first enable. /new, MCP reload, and mid-session enable are not enough.
- After restart, confirm with
tool_search/tool_describe for orgo_desktop_drag and orgo_desktop_click_path, not only config flags.
Pitfalls
- Wrong secret: Desktop API wants VNC password (
VNC_PASSWORD). ORGO_API_KEY is cloud-only.
- Cloud MCP on self: longer hop, fleet ambiguity; use only for remote computers / lifecycle.
- CUA is not the spine: may doctor-green; web modals often have null AX frames / stale tokens. Prefer Desktop API pixels + browser DOM for web.
- Root Chrome: needs
--no-sandbox (open-url already does).
- Agent cannot patch
config.yaml via write/patch tools — use hermes plugins enable / hermes tools enable / hermes config set.
- Mid-session tools / upgrades: enable and plugin code changes do not hot-load into a long-lived Hermes PID; fully quit the process (not only
/new). Until then use CLI/OrgoDesktopClient and report that path honestly.
- Smoke overall false ≠ stack down: check whether only
fingerprint_stable failed; still treat doctor ready + action checks as usable.
- Auth dialogs on headed desktop: never type passwords, 2FA, payment, or permission dialogs. Public handle ok; then wait for the user.
- API success ≠ UI change: use
verify / click_path fingerprints; destination_noop means the second click did nothing visible.
- Chess / board UIs: knight is 2+1 only; in check only resolving moves are legal; re-read move list after every opponent ply; do not trust last-move green as selection.
- Coordinates: full desktop 1280×720 unless screenshot proves otherwise; re-measure after maximize. Clicks that "succeed" in the API can still hit the wrong control if y is off by ~30–80px (common on stacked modal buttons).
- Pixel-find before re-clicking: when a CTA fails twice, crop the screenshot / scan for button text luminance or selected-green tiles instead of guessing nearby y values.
- SPA card open needs double-click: console.x.com (and similar) app/project cards may ignore single click; use
OrgoDesktopClient.double_click. Focus the correct Chrome window title when login and console both exist.
- CLI vs Python surface: drag / click-path / verify are on CLI; double-click is client/API only until CLI grows a subcommand.
- Adjacent CTAs: Lichess right rail stacks Create lobby / Challenge a friend / Play against computer — Challenge opens a similar "Game setup" modal without strength; confirm orange highlight or Strength row before "Play".
- Stale deep links: Lichess
https://lichess.org/setup/ai has returned 404; use homepage → Play against computer (or a current documented path).
- Many Chrome windows: repeated
open-url/--new-window multiplies windows; focus the wrong title (e.g. aborted human game) mid-smoke. Prefer one window; activate by window name via desktop-api /bash + xdotool when needed.
- pkill Chrome from agent shell: broad
pkill -f chrome can interrupt the controlling command (SIGTERM). Prefer targeted PIDs or one clean relaunch path; verify with orgo-desktop doctor after.
- Adaptive multi-ply only: never fire a long fixed chess/game click list after the first opponent reply without re-reading the board/move list.
- Events underused:
ws://127.0.0.1:8080/events?token=<vnc_password> for window_open / file_change / idle waits.
Verification
orgo-desktop doctor
orgo-desktop smoke
curl -sS http://127.0.0.1:8080/health
hermes plugins list | head -40
References
references/desktop-api.md — local API verbs, auth, coords, eventsreferences/hermes-plugin.md — plugin layout, enable path, official docsreferences/headed-web-game-loop.md — headed SPA/game smoke: doctor, open-url, pixel CTAs, board mapping, Lichess notesreferences/black-framebuffer-recovery.md — black VNC/paint recovery pathreferences/packaging-and-sharing.md — MCP vs plugin vs skills vs OS1; how to share with othersreferences/hermes-dashboard-on-orgo.md — start hermes dashboard and make it user-visible on VNC (doctor → open-url → nonblack screenshot)
