orgo-desktop-local

name	orgo-desktop-local
description	Same-box Orgo Desktop API computer use via loopback :8080 (prefer over cloud MCP / CUA when Hermes is on the VM). Model tools, CLI, plugin enable path.
version	1.4.1
author	Hermes
platforms	["linux"]
metadata	{"hermes":{"tags":["Orgo","DesktopAPI","ComputerUse","Local","CoLocated","Plugin"],"category":"desktop","related_skills":["orgo-cloud-computers","computer-use"]}}

Orgo Desktop Local (same-box computer use)

When Hermes runs inside an Orgo VM (hostname often orgo-desktop, DISPLAY=:99), GUI/control actions should hit the in-VM Desktop API, not the cloud Orgo MCP hop and not CUA-first.

Standing policy (Dewey / this class of host): prefer Orgo local computer use over CUA driver / Hermes computer_use. Local Desktop API is the spine for screenshot, click, type, key, drag, open-url, bash, and black-framebuffer recovery. CUA is optional AX/SOM enrichment only. Cloud Orgo MCP GUI is for other VMs and lifecycle, not this same box when doctor is green.

Load this skill for headed same-box desktop control. For fleet lifecycle / other VMs, also load orgo-cloud-computers. The generic computer-use skill documents CUA; on Orgo it defers here first.

Endpoint

Base	http://127.0.0.1:8080 (override ORGO_DESKTOP_API_URL)
Auth	Authorization: Bearer <vnc_password>
Token env	VNC_PASSWORD or ORGO_DESKTOP_API_TOKEN (not ORGO_API_KEY)
Service	/opt/desktop-api/desktop-api (orgo-desktop-api)
Schema	GET /schema (no auth)
Health	GET /health (no auth)

Actions: /screenshot /click /double_click /drag /type /key /scroll /mouse_move /wait /bash /exec /status plus WS /events /terminal /audio.

Coordinate space: full desktop pixels (Dewey default 1280×720).

See references/desktop-api.md for request shapes and events.

Prefer order (mandatory)

1. Co-located + doctor green + nonblack paint → local Desktop API (this skill / plugin tools / orgo-desktop CLI)
2. Websites → Hermes browser_* first (DOM), desktop only when headed watch / native chrome needed
3. Scriptable → Hermes terminal or Desktop API /bash before pixel clicks
4. Other computers / lifecycle → cloud Orgo MCP / https://www.orgo.ai/api (orgo-cloud-computers)
5. CUA / computer_use → optional a11y enrichment only when element frames exist; never block readiness on CUA; never use as first screenshot path when paint may be black
6. Never default to cloud orgo_click / MCP GUI when local doctor is green

Why Orgo > CUA here (evidence, not branding)

Failure	CUA / naive scrot	Local Desktop API
Black framebuffer	Pure black captures; app list noise	Doctor + screenshot bytes; recovery doc restores paint
Headed Chrome	Window title real but CUA "no window matched"	orgo-desktop open-url + user-visible noVNC
Readiness	Optional install; not required	orgo-desktop doctor → ready / prefer: local_desktop_api
Hop	Extra driver session	Loopback :8080 same paint path as VNC

Sentiment to preserve across patches: Orgo local first, CUA second, cloud MCP third (other hosts only).

Hermes model tools (plugin)

Plugin: ~/.hermes/plugins/orgo-desktop-local/
Client lib: ~/.hermes/scripts/orgo_desktop/
CLI: ~/.local/bin/orgo-desktop

Install / enable (once per host)

hermes plugins enable orgo-desktop-local
hermes tools enable orgo_desktop --platform cli
# optional: --platform telegram|discord|whatsapp|slack|signal

New session required after enable (tool list is built at session start).

See references/hermes-plugin.md for register_tool pattern and official docs.

Tool surface

Toolset key: orgo_desktop

Tool	Purpose
orgo_desktop_doctor	health + auth + screenshot + bash; also prefer / control_plane / avoid_when_ready
orgo_desktop_screenshot	full desktop PNG path
orgo_desktop_click	x,y full-desktop coords; optional verify → noop / visual_changed
orgo_desktop_drag	drag start→end (Desktop API /drag); optional verify
orgo_desktop_click_path	two-click select→dest; default verify (destination_noop if 2nd click no pixel change)
orgo_desktop_type	type text
orgo_desktop_key	key / chord (Escape, Return, ctrl+l)
orgo_desktop_bash	shell via desktop-api /bash
orgo_desktop_scroll	scroll at point
orgo_desktop_open_url	headed Chrome (--no-sandbox)
orgo_desktop_wait	wait seconds (API max 60)

Visual verify: cropped frame SHA-256 (default skips top ~28px panel/clock). API success alone is not proof the UI changed.

Session workflow

1. Call orgo_desktop_doctor (or orgo-desktop doctor). Proceed only if ready: true and the screenshot is not a black framebuffer. Treat screenshot_bytes ≈ 2.7–3.5 KB (or PIL nonblack ≈ 0) as paint-dead even when API health is green — run references/black-framebuffer-recovery.md before claiming a headed app is open. When ready, prefer should be local_desktop_api — do not use cloud mcp__orgo__* GUI for this box.
2. Prefer orgo_desktop_open_url / bash openers over hunting homepage buttons (including Hermes dashboard at http://127.0.0.1:9119).
3. Screenshot → one action → wait → re-screenshot → re-plan from actual state. Prefer orgo-desktop screenshot over raw scrot/CUA root capture for truth on Xvnc.
4. For piece/icon moves: prefer orgo_desktop_click_path or orgo_desktop_drag with verify. If destination_noop or noop is true, re-plan (do not assume the move landed).
5. If tools missing from the model list: confirm plugin+toolset enabled, then fully quit and restart the Hermes process (not only /new inside a pre-enable process). Mid-session fallback: CLI orgo-desktop … or Python OrgoDesktopClient. Do not invent cloud MCP GUI calls when doctor is green. Report which path executed honestly.

Mid-session client fallback (when orgo_desktop_* absent)

import sys
sys.path.insert(0, str(Path.home() / ".hermes" / "scripts"))
from orgo_desktop import OrgoDesktopClient
c = OrgoDesktopClient()
assert c.doctor()["ready"]
c.open_url("https://example.com/")
c.save_screenshot("/tmp/desk.png")
c.click(x, y)
c.wait(1.5)

Handlers live in ~/.hermes/plugins/orgo-desktop-local/tools.py and wrap this client; they need Hermes tools.registry only when registered as model tools.

Headed web / game loop (local API)

When the user wants a watched browser session (VNC stream) on this box:

1. Doctor green first.
2. Prefer one Chrome window: avoid stacking many open-url launches (tabs/windows desync focus). If messy, close Chrome carefully then re-open one URL (see pitfalls).
3. Navigate by URL when possible; for SPAs without a deep link, open homepage then click the correct CTA (adjacent buttons are easy to mis-hit).
4. Locate targets from the latest screenshot, not from memory coords. Modal CTAs often sit below selected option rows (e.g. Lichess "Play against computer" was ~y=649 while Side/White was ~y=555 on 1280×720).
5. Dense SPA list rows / cards (e.g. X Developer Console production app): single click often no-ops or hits adjacent Create/+ New controls. Prefer double-click via Python OrgoDesktopClient.double_click(x, y) (Desktop API /double_click). The orgo-desktop CLI has no double_click subcommand yet — do not invent one.
6. Before re-clicking a missed card, re-measure: white-text centroid (PIL) on the current frame; y-off by ~20–40px is enough to miss the hit target on 1280×720.
7. After URL navigation, wait through Loading… / skeleton frames before measuring or clicking (console.x.com apps list commonly needs 2–6s).
8. Multi-ply / opponent games: after every user move and every opponent reply, re-screenshot and re-plan. Fixed move lists desync under checks, captures, and unexpected replies.
9. Board games: map board bounds from screenshot colors once, then click square centers; re-map if layout changes. See references/headed-web-game-loop.md.

CLI (shell fallback)

orgo-desktop doctor
orgo-desktop smoke
orgo-desktop screenshot /tmp/desk.png
orgo-desktop bash 'hostname'
orgo-desktop click 640 400
orgo-desktop click 640 400 --verify
orgo-desktop drag 100 200 300 250 --verify
orgo-desktop click-path 100 200 300 250
orgo-desktop type 'hello'
orgo-desktop key Escape
orgo-desktop open-url https://example.com/
orgo-desktop colocated

Python (add ~/.hermes/scripts to sys.path):

from orgo_desktop import OrgoDesktopClient
client = OrgoDesktopClient()
assert client.doctor()["ready"]
client.save_screenshot("/tmp/desk.png")
client.click(100, 200)

Success / validation criteria

Check	Pass
orgo-desktop doctor	"ready": true and "prefer": "local_desktop_api"
orgo-desktop smoke	Prefer per-check green for actions; see smoke interpretation below
Plugin handlers	orgo_desktop_doctor returns ready against :8080
Config	plugins.enabled has orgo-desktop-local; platform toolsets include orgo_desktop
New tools	orgo_desktop_drag / orgo_desktop_click_path registered after full process restart

Do not claim computer-use is improved from config alone; require doctor green and honest tool-list verification (tool_search / tool_describe).

Smoke interpretation

orgo-desktop smoke may report overall passed: false while all control actions (health, auth, screenshot, bash, key, click, wait, drag, click_path, verify fields, prefer-local) are green. A lone fingerprint_stable failure usually means two consecutive frames differed (clock / panel / wallpaper) and is not a Desktop API outage.

When reporting status to the user:

1. Separate action checks from fingerprint_stable.
2. Separate CLI / Python client (disk, current after code deploy) from model tools in this process (baked at Hermes process start).
3. Plugin code upgrades (new schemas: drag, click_path, click verify) need the same full quit + new Hermes process as first enable. /new, MCP reload, and mid-session enable are not enough.
4. After restart, confirm with tool_search/tool_describe for orgo_desktop_drag and orgo_desktop_click_path, not only config flags.

Pitfalls

• Wrong secret: Desktop API wants VNC password (VNC_PASSWORD). ORGO_API_KEY is cloud-only.
• Cloud MCP on self: longer hop, fleet ambiguity; use only for remote computers / lifecycle.
• CUA is not the spine: may doctor-green; web modals often have null AX frames / stale tokens. Prefer Desktop API pixels + browser DOM for web.
• Root Chrome: needs --no-sandbox (open-url already does).
• Agent cannot patch config.yaml via write/patch tools — use hermes plugins enable / hermes tools enable / hermes config set.
• Mid-session tools / upgrades: enable and plugin code changes do not hot-load into a long-lived Hermes PID; fully quit the process (not only /new). Until then use CLI/OrgoDesktopClient and report that path honestly.
• Smoke overall false ≠ stack down: check whether only fingerprint_stable failed; still treat doctor ready + action checks as usable.
• Auth dialogs on headed desktop: never type passwords, 2FA, payment, or permission dialogs. Public handle ok; then wait for the user.
• API success ≠ UI change: use verify / click_path fingerprints; destination_noop means the second click did nothing visible.
• Chess / board UIs: knight is 2+1 only; in check only resolving moves are legal; re-read move list after every opponent ply; do not trust last-move green as selection.
• Coordinates: full desktop 1280×720 unless screenshot proves otherwise; re-measure after maximize. Clicks that "succeed" in the API can still hit the wrong control if y is off by ~30–80px (common on stacked modal buttons).
• Pixel-find before re-clicking: when a CTA fails twice, crop the screenshot / scan for button text luminance or selected-green tiles instead of guessing nearby y values.
• SPA card open needs double-click: console.x.com (and similar) app/project cards may ignore single click; use OrgoDesktopClient.double_click. Focus the correct Chrome window title when login and console both exist.
• CLI vs Python surface: drag / click-path / verify are on CLI; double-click is client/API only until CLI grows a subcommand.
• Adjacent CTAs: Lichess right rail stacks Create lobby / Challenge a friend / Play against computer — Challenge opens a similar "Game setup" modal without strength; confirm orange highlight or Strength row before "Play".
• Stale deep links: Lichess https://lichess.org/setup/ai has returned 404; use homepage → Play against computer (or a current documented path).
• Many Chrome windows: repeated open-url/--new-window multiplies windows; focus the wrong title (e.g. aborted human game) mid-smoke. Prefer one window; activate by window name via desktop-api /bash + xdotool when needed.
• pkill Chrome from agent shell: broad pkill -f chrome can interrupt the controlling command (SIGTERM). Prefer targeted PIDs or one clean relaunch path; verify with orgo-desktop doctor after.
• Adaptive multi-ply only: never fire a long fixed chess/game click list after the first opponent reply without re-reading the board/move list.
• Events underused: ws://127.0.0.1:8080/events?token=<vnc_password> for window_open / file_change / idle waits.

Verification

orgo-desktop doctor
orgo-desktop smoke
curl -sS http://127.0.0.1:8080/health
hermes plugins list | head -40
# after new session: model tools orgo_desktop_* available

References

• references/desktop-api.md — local API verbs, auth, coords, events
• references/hermes-plugin.md — plugin layout, enable path, official docs
• references/headed-web-game-loop.md — headed SPA/game smoke: doctor, open-url, pixel CTAs, board mapping, Lichess notes
• references/black-framebuffer-recovery.md — black VNC/paint recovery path
• references/packaging-and-sharing.md — MCP vs plugin vs skills vs OS1; how to share with others
• references/hermes-dashboard-on-orgo.md — start hermes dashboard and make it user-visible on VNC (doctor → open-url → nonblack screenshot)