unix-computer-use

name	unix_computer_use
description	Local and remote desktop observation/input tools with coordinate normalization (local macOS/Linux by default; optional OSWorld HTTP and SSH Mac backends).
version	0.4.0
type	extension
entry	plugin.py
runtime	python3
permissions	["tool","subprocess","net"]
env_from_settings	[]
when_to_use	The user asks Ouroboros to inspect a desktop, take a screenshot, click/type/press keys, drag, or operate a local/explicitly configured remote GUI under human or benchmark supervision.

Unix Computer Use

This bundled extension exposes a reviewable computer-use substrate for local macOS/Linux desktops and explicitly configured remote targets. All actions are for supervised, low-risk workflows under human or benchmark observation. Windows support is deferred to a separate backend/skill.

Tools:

• capabilities reports the platform, display-session type, and detected screenshot/input backends plus the active connection/backend.
• list_connections, add_connection, test_connection, activate_connection, and use_local manage the local/remote connection registry in skill state.
• screenshot captures the desktop, downscales the image to fit WXGA (1280x800, configurable) and persists the exact image-to-input coordinate transform. Pass coordinates read off the returned image directly to the input tools — they are remapped automatically. The result carries the typed auto_attach_image field, so the host attaches the image to the conversation in the same round (v6.81.1) — no view_image call is needed for a fresh capture; the returned path stays view_image-readable for re-viewing later.
• click (left/right/middle, double/triple), the thin aliases double_click/triple_click (models call these names unprompted), move, left_click_drag, mouse_down/mouse_up, cursor_position, type_text, key, hold_key, scroll, and wait execute input through platform tools when available. Pointer coordinates pass through one normalizer that accepts the malformations models actually emit (the pair packed into x with y absent or duplicated) and fails loudly on anything ambiguous.
• window_list lists visible windows/processes where the platform exposes a lightweight backend.
• ax_tree returns a set-of-marks accessibility snapshot of the frontmost window on macOS: numbered interactive elements (role, title, center coordinates in input space — click them with raw=true). It degrades honestly to a process/window list when the AX walk fails or on Linux.
• remote_exec runs a shell command on the active remote backend only; it refuses on local backend. Use ordinary Ouroboros shell/file tools for local work.