| name | unix_computer_use |
| description | Local and remote desktop observation/input tools with coordinate normalization (local macOS/Linux by default; optional OSWorld HTTP and SSH Mac backends). |
| version | 0.4.0 |
| type | extension |
| entry | plugin.py |
| runtime | python3 |
| permissions | ["tool","subprocess","net"] |
| env_from_settings | [] |
| when_to_use | The user asks Ouroboros to inspect a desktop, take a screenshot, click/type/press keys, drag, or operate a local/explicitly configured remote GUI under human or benchmark supervision. |
Unix Computer Use
This bundled extension exposes a reviewable computer-use substrate for local
macOS/Linux desktops and explicitly configured remote targets. All actions are
for supervised, low-risk workflows under human or benchmark observation. Windows
support is deferred to a separate backend/skill.
Tools:
capabilities reports the platform, display-session type, and detected
screenshot/input backends plus the active connection/backend.list_connections, add_connection, test_connection, activate_connection,
and use_local manage the local/remote connection registry in skill state.screenshot captures the desktop, downscales the image to fit WXGA
(1280x800, configurable) and persists the exact image-to-input coordinate
transform. Pass coordinates read off the returned image directly to the
input tools — they are remapped automatically. The result carries the typed
auto_attach_image field, so the host attaches the image to the
conversation in the same round (v6.81.1) — no view_image call is needed
for a fresh capture; the returned path stays view_image-readable for
re-viewing later.click (left/right/middle, double/triple), the thin aliases
double_click/triple_click (models call these names unprompted), move,
left_click_drag, mouse_down/mouse_up, cursor_position, type_text,
key, hold_key, scroll, and wait execute input through platform tools
when available. Pointer coordinates pass through one normalizer that accepts
the malformations models actually emit (the pair packed into x with y
absent or duplicated) and fails loudly on anything ambiguous.window_list lists visible windows/processes where the platform exposes a
lightweight backend.ax_tree returns a set-of-marks accessibility snapshot of the frontmost
window on macOS: numbered interactive elements (role, title, center
coordinates in input space — click them with raw=true). It degrades
honestly to a process/window list when the AX walk fails or on Linux.remote_exec runs a shell command on the active remote backend only; it
refuses on local backend. Use ordinary Ouroboros shell/file tools for local
work.
