cmux-cua

name	cmux-cua
description	Drive real macOS apps from a cmux agent session via the bundled computer-use engine (accessibility tree + screenshot perception, click/type/scroll/drag, branded agent cursor). Use when an agent should see and operate GUI apps on the local Mac, when computer-use tools are missing or failing, or when explaining how to grant permissions, brand the cursor, or focus the driving session.

cmux-cua

cmux bundles a local computer-use engine (packaged as cmux Computer Use with the MCP proxy named cmux-cua, from a pinned build of the manaflow-ai/cmux-cua fork) and attaches it as an MCP tool server named cmux-cua to every agent session cmux launches (Claude Code, Codex). The agent can then perceive and operate real macOS apps: read the accessibility tree, take screenshots, and click / type / scroll / drag.

Everything runs locally through the bundled cmux Computer Use helper. The helper has its own TCC identity, so Accessibility and Screen Recording never belong to the main cmux app and granting Screen Recording never requires restarting cmux. Upstream telemetry and update checks are disabled at runtime.

How it attaches

• The cmux-claude-wrapper and cmux-codex-wrapper inject cmux-cua as an MCP proxy using mcp --socket <cmux-owned socket> plus cursor-branding and state-dir env. Codex launches the exact tag-installed helper executable as its authenticated approval broker; Claude uses the bundled native-profile proxy client. The Codex wrapper additionally passes --codex-computer-use-compat; the Claude wrapper deliberately does not. No user setup per session — start claude or codex inside cmux and the corresponding tool profile is there.
• ComputerUseRuntimeService is the only helper lifecycle owner. It installs the nested helper under the tag-scoped ~/Library/Application Support/cmux/cmux-cua/helper/<scope>/ directory and launches that explicit app URL through LaunchServices.
• The native daemon uses the tag-scoped /tmp/cmux-cua-<uid>/<scope>/cmux-cua.sock; the Codex compatibility daemon uses cmux-cua-codex.sock beside it. Both fit Darwin's Unix-socket path limit and share the tag-scoped cmux Application Support state directory.
• The wrappers keep the signed, app-bundled skill discoverable in both agent pickers: each repairs the link in its agent's own root before launching — ~/.claude/skills/cmux-cua for Claude, ~/.agents/skills/cmux-cua for Codex (migrating any older cmux-owned cmux-computer-use link) — so the picker shows one plain cmux-cua entry. Codex falls back to an invocation-scoped skills.config entry only when that repair fails; there is deliberately no Claude plugin fallback and no plugin manifest in this directory, because a manifest makes Codex namespace the skill (cmux-cua:cmux-cua) and Claude display it plugin-qualified. A user-owned directory or unrelated symlink at that path is never replaced. Set CMUX_COMPUTER_USE_INSTALL_GLOBAL_SKILL=0 when a strictly session-local launch is required.
• While Computer Use is enabled, the helper daemon starts quietly at cmux startup with its internal permission gate disabled. Starting cmux or an agent never requests access or shows onboarding.
• Wrappers are pure forced proxies. They never copy or launch the helper and never fall back to in-process computer use. cmux owns the onboarding window and opens the permanent macOS permission panes directly; it does not ask the helper to raise an intermediate native prompt. The proxy keeps its external-flow flag on so the first driving call waits for both helper grants before it is forwarded.
• Kill switch: set CMUX_COMPUTER_USE_MCP_DISABLED=1, or toggle it off in Settings → Computer Use (persists to ~/.config/cmux/cmux.json and is exported to spawned terminals).
• Attaches only on cmux-launched, live-socket sessions (same authority bar as cmux hooks); hooks-disabled and stale-socket sessions do not attach.

Permissions (one-time, granted to the helper)

Two macOS permissions are required and are owned by cmux Computer Use, not the main cmux app:

• Accessibility — inspect and drive app UI (AXIsProcessTrusted).
• Screen Recording — screenshots / vision (CGPreflightScreenCaptureAccess).

Onboarding appears on the first real Computer Use tool invocation, not on cmux or agent startup. Settings → Computer Use always shows the two authoritative permission states; choosing Grant… for an ungranted permission opens that same permission step and its draggable helper-app recovery path. Each Allow action opens the matching permanent System Settings pane in one step and stays labeled Allow until the helper reports the grant; pressing it again simply reopens the same pane. If macOS has not listed the helper yet, drag or add the cmux Computer Use app tile to the list, then turn it on. cmux reads status from the helper over its Unix socket, advances beside System Settings to the next missing permission, and shows completion in place once both are granted. On macOS Tahoe a third confirmation follows Screen Recording: the system's direct-capture consent, an alert that says cmux Computer Use "is attempting to bypass the system private window picker". That alert is expected — it comes from onboarding's host-authenticated capture probe, onboarding explains it in place, and the user must allow it before setup completes. Never "fix" it by suppressing the probe; without that consent, agent screenshots on Tahoe fail. The consent follows the helper's code signature, so every rebuilt (ad-hoc signed) dev helper re-triggers it: cmux invalidates its cached direct-capture-ready flag whenever it replaces the installed helper build, which re-presents onboarding so the alert always lands with its explanation. Do not invoke check_permissions {prompt:true} or any standalone helper while this flow is active: that creates the stray native permission dialogs this onboarding deliberately avoids. The main cmux process never calls a TCC API or executes the cmux-cua binary.

A TCC prompt naming Codex Computer Use (com.openai.sky.CUAService) is not from cmux. The codex CLI ships its own computer-use helper; when codex runs inside a cmux terminal and pokes that helper with an Apple Event, macOS attributes the request to the responsible parent — the cmux app — so the dialog reads as cmux asking to control "Codex Computer Use". Nothing in cmux or the cmux-cua engine references that service; denying the prompt does not affect cmux computer use.

If actions fail with a permission error, grant Accessibility to cmux Computer Use. If screenshots come back blank, grant Screen Recording to cmux Computer Use. The helper daemon refreshes/restarts to pick up the grant while cmux stays open. Retry the tool call after onboarding reports both grants.

Using the tools (agent-facing)

cmux already owns the MCP connection's session identity. Do not call start_session / end_session, and do not pass a custom session argument. The proxy binds every call to the originating cmux surface so the menu-bar item, cursor, recording cleanup, and background/focus controls stay attached to the right agent.

Codex profile

Codex gets the exact ten-tool Computer Use roster, in order:

list_apps, get_app_state, click, perform_secondary_action, set_value, select_text, scroll, drag, press_key, type_text.

Use it like the built-in Computer Use connector:

1. Call get_app_state with the app name, full path, or unambiguous bundle id before acting. It launches the app if needed and returns the logical-size JPEG screenshot plus the compact accessibility tree.
2. Prefer the current snapshot's string element_index; use screenshot-local x/y coordinates only as fallback.
3. Use xdotool-style key strings such as super+l with press_key.
4. Operate controls with visible pointer clicks by default: the branded cursor gliding to each button and clicking is the product experience, so requests like “click 100 + 105” mean literal button-by-button pointer interaction on Calculator's buttons. Reserve type_text for entering text into text fields (search boxes, forms, editors) — not as a shortcut around clicking on-screen controls. Only fall back to a single keyboard sequence when the user explicitly asks for speed over visibility or a control has no clickable element.
5. Actions return a compact dispatch acknowledgement, not a screenshot or accessibility tree. After one or more actions, call get_app_state before deciding what to do next. The returned screenshot/tree is the authoritative verification surface, matching the built-in Computer Use connector.
6. Numeric element_index values belong only to the state that displayed them. Re-snapshot before using an index that may have been renumbered by a layout change (Calculator's All Clear removes display nodes, for example). Multiple coordinate actions against a stable layout may be issued in one host turn; do not batch element-index actions across a state-changing step.

Do not expect native cmux extensions such as get_window_state, tokens, perform_actions, cursor controls, diagnostics, recordings, or browser/CDP in this profile. Their absence is required for Codex schema parity.

Claude/native cmux profile

Perceive, act in logical groups, then verify:

1. get_window_state (pid + window_id) returns the accessibility tree and a screenshot. Ground on both. Prefer element addressing.
2. Act by element: click with element_token (or element_index + pid + window_id) is the robust path. Pixel addressing (x,y) is the fallback.
3. For a stable, already-snapshotted control set, call perform_actions once with the ordered click / type_text / press_key / other input steps. This reuses the existing element-token cache and visible cursor inside the persistent proxy instead of paying one model/MCP round trip and AX scan per click. Do not put navigation, modal-opening, or layout-changing actions before later control references in the same group; re-snapshot immediately after any action that can invalidate those controls.
4. Verify the completed group by re-snapshotting and reading the element value / screenshot — do not assume actions landed (clicks are never helper-verified). Operate on-screen controls with visible pointer clicks by default — the gliding branded cursor is the product experience. Use type_text for entering text into text fields, and fall back to a pure keyboard sequence only when the user explicitly prefers speed over visibility.

Notes:

• In the native profile, use list_apps / launch_app / list_windows to find targets; get_window_state needs a window_id from list_windows.
• Catalyst apps (e.g. Calculator) can expose an empty AX tree briefly after launch and return spurious AX error codes (-25204) even when the action landed — re-snapshot and check the result rather than trusting the code.
• Pixel input is obstruction-checked: if another window covers the target point cmux-cua refuses with background_occluded naming the occluder instead of clicking the wrong window. Retry with delivery_mode:"foreground" or front the target.

The branded agent cursor

The agent's pointer shows as the cmux logo gradient (#12c7f5 → #2d8cff → #6c5cff) with a cmux label, so it is visually distinct from the user's cursor. It is configured by env the wrapper injects (CMUX_CUA_CURSOR_GRADIENT / _BLOOM / _LABEL) and is auto-active while the helper daemon is driving. It remains visible across normal reasoning gaps and is removed when the driving session ends or the proxy control connection closes. Each later action reasserts the cursor directly above the driven target. If no cursor appears during an action, confirm the MCP config uses the helper socket, has a stable CMUX_CUA_DEFAULT_SESSION, and uses the pinned cmux-cua build.

Finding and focusing the driving session

While an agent is driving, the cmux Computer Use menu-bar item projects only the most recently active live agent session and offers two presentation modes:

• Focus Computer Use — bring forward the app the agent is driving and resume automatically following new targets.
• Focus Calling Terminal — return to the terminal that invoked Computer Use and reveal the exact workspace + surface running that agent while automation continues in the background.

The helper pins its cursor window directly above the driven target window at the normal application window level. That keeps the cursor visible on the target while allowing any app the user places in front of that target to cover the cursor naturally; presentation mode never promotes it to an always-on-top layer.

The active target and session ordering come from cmux-cua's per-session state files under ~/Library/Application Support/cmux/cmux-cua/runtime/<scope>/state/.

The item hides when there is no live or recent session. Toggle visibility in Settings → Computer Use.

Troubleshooting

• Agent has no computer-use tools — Settings → Computer Use must be on; start a new session (tools attach at launch).
• Clicks do nothing / not permitted — grant Accessibility to cmux Computer Use.
• Black/empty screenshots — grant Screen Recording to cmux Computer Use; restart only the helper if its automatic refresh has not completed yet.
• No menu-bar icon — needs a live/recent session; check the visibility toggle.
• Prompts name the main cmux app — a non-cmux fallback executed the helper directly. Stop there and report the failure; the bundled path must use the tag-scoped socket with CMUX_CUA_MCP_FORCE_PROXY=1.
• Prompts name CmuxCua — a stale /Applications/CmuxCua.app daemon or a standalone helper launch is active. Stop it and reset/remove its TCC entry; the bundled path never uses that identity.

Development

• Engine source: manaflow-ai/cmux-cua (libs/cmux-cua/rust). cmux consumes it via CMUX_CUA_PINNED_SHA in scripts/build-cmux-cua.sh, which builds, lipos, and codesigns the binary plus the nested helper into the app bundle.
• The helper daemon's CMUX_CUA_EXTERNAL_PERMISSION_FLOW=1 prevents agent-supplied check_permissions {prompt:true} from bypassing cmux onboarding. The wrappers set CMUX_CUA_MCP_FORCE_PROXY=1 and preserve the external-flow contract in the proxy process so protected calls wait for cmux's post-verification readiness signal. There is no ambient executable override. Both profiles resolve the app-bundled cmux-cua executable and the tag-scoped helper only; missing ownership fails closed instead of running a user-supplied executable.
• If the cmux-owned daemon is unavailable, do not invoke cmux-cua directly through Bash and do not start its default socket. Tell the user to open Settings → Computer Use or restart the tagged cmux build, then retry the MCP tool after the helper runtime is healthy.
• Never hand-edit docs/.../cmux-cua/mcp-tools.mdx in the fork — it is generated from the Rust tool descriptions.
• cmux-side UX lives in Sources/App/ComputerUse*.swift, Packages/macOS/CmuxSettingsUI/.../Sections/ComputerUseSection.swift, and the two wrappers under Resources/bin/.