ui-control

name	ui-control
description	Infrastructure skill - application UI observation and scoped action tools for DCC-adjacent workflows. Use ui_control__snapshot, ui_control__find, ui_control__act, ui_control__wait_for, and ui_control__stop_computer_use for DCC UI Control when a host UI state is not exposed through native DCC APIs. Use the separate, operator-granted ui_control__system_operation only for bounded Windows plug-in setup. Prefer DCC-native skills first, then use ui_control as a policy-controlled fallback.
license	MIT
metadata	{"dcc-mcp":{"dcc":"python","version":"0.3.0","layer":"infrastructure","search-hint":"dcc ui control, ui-control, ui control, ui automation, operate control menu dialog window button click keyboard, windows uia, chrome cdp, edge cdp, agent-browser, modal, settings panel, screenshot, snapshot, find control, custom control, face shaping, sculpt slider, modifier drag, registry, symlink, plugin setup, remote control, scroll, type, keypress, wait for ui, stale control, dcc debugging, 操作, 控制, 界面, 菜单, 弹窗, 窗口, 按钮, 点击, 键盘, 捏脸","tags":"ui-control, dcc-ui-control, ui-automation, windows-uia, chrome-cdp, edge-cdp, agent-browser, diagnostics, infrastructure, mock","tools":"tools.yaml"}}

DCC UI Control

Application UI automation primitives for cases where native DCC tools cannot observe or drive the interface state directly.

DCC UI Control is the public capability name. The canonical skill is ui-control, its tools use the ui_control__* prefix, and configuration uses DCC_MCP_UI_CONTROL_*. Shell agents use dcc-mcp-cli ui-control; MCP-native agents call the underlying tools after search and describe.

ui-control is an escape hatch, not the first tool choice. Discover and call a structured DCC skill, host API, or adapter script first. Enter ui-control only when that path reports unsupported or capability_missing. Policy denial, user interruption, authentication, or desktop unavailability are stop conditions, not fallback signals.

The default backend is deterministic mock state for CI and adapter authoring. Set DCC_MCP_UI_CONTROL_BACKEND=chrome to use the experimental CDP backend through the same ui_control__* contract.

Set DCC_MCP_UI_CONTROL_BACKEND=windows-uia on Windows to use the isolated dcc-mcp-ui-control-host.exe. Bind it at adapter startup with exactly one DCC_MCP_UI_CONTROL_UIA_PROCESS_ID or DCC_MCP_UI_CONTROL_UIA_WINDOW_HANDLE; request parameters may narrow that scope but cannot create or widen it. Whole-desktop and title/process-name-only native sessions are disabled.

Windows Reference Backend

The Windows backend exposes DCC UI Control through the existing ui_control tools:

DCC UI Control operation	DCC-MCP tool
screenshot	ui_control__snapshot
semantic click, set_text, toggle, set_checked, select_option, focus	ui_control__act with an exact control_id
raw click, move, double_click, scroll, drag, keypress	ui_control__act with the latest snapshot_id
typed HKCU value or symbolic-link ensure	ui_control__system_operation
wait	ui_control__wait_for (condition-based polling)
stop	ui_control__stop_computer_use

Shell agents use the product-level wrapper, which maps to those tools without requiring a hand-built slug:

dcc-mcp-cli ui-control snapshot --instance-id <id> --json '{"session_id":"ui","process_id":1234}'
dcc-mcp-cli ui-control find --instance-id <id> --json '{"session_id":"ui","label":"Settings"}'
dcc-mcp-cli ui-control act --instance-id <id> --json '{"session_id":"ui","control_id":"settings","action":"click","snapshot_id":"<snapshot_id>"}'
dcc-mcp-cli ui-control system-operation --instance-id <id> --json '{"operation_id":"enable-remote-control"}'
dcc-mcp-cli ui-control wait --instance-id <id> --json '{"session_id":"ui","condition":{"kind":"control_exists","label":"Preferences"}}'
dcc-mcp-cli ui-control stop --instance-id <id> --json '{"session_id":"ui"}'

system-operation is a separate, windowless setup path. Before the shared Windows-session host starts, the operator supplies an exact JSON catalog with DCC_MCP_UI_CONTROL_SYSTEM_GRANTS_FILE and selects one entry with DCC_MCP_UI_CONTROL_SYSTEM_GRANT_ID; changing either requires a host restart. Each request carries only a non-sensitive operation_id; the native host resolves the corresponding typed operation locally and still requires native action-time confirmation. Registry values and link paths never enter model context, tool arguments, or the host pipe. Only HKCU String/DWORD values and file/directory symbolic links are supported. The operator-owned catalog is trusted configuration, not a credential store; use an opaque credential broker or a host-owned secure prompt for secrets. There is no command, deletion, overwrite, alternate-hive, elevation, UAC, or security-settings form. Stop on system_operation_not_granted, approval_required, or elevation_required.

The CLI prints compact JSON by default: it keeps routing ids, messages/errors, observation ids, snapshot metadata, semantic matches, and materialized image paths while omitting the repeated MCP envelope and full UIA tree. Add --full-output only for targeted raw protocol or tree diagnostics.

All ui-control tools require the adapter's persistent in-process executor so one thin named-pipe client survives across snapshot/action calls. The independent per-Windows-session host owns screenshots, UIA, observation ids, the Ctrl+Alt+Esc stop latch, visible overlay, global input owner, confirmation, and native input; adapters do not retain an alternate native path. Every snapshot, find, action, wait, stop, and rejected operation also appends a redacted ui_control_operation event to the shared DCC-MCP log directory, so the existing Admin Logs panel can display it without exposing entered text or screenshot coordinates.

Use semantic UI Automation first: resolve a stable control_id with ui_control__find, then use click, set_text, toggle, set_checked, or focus. Use screenshot coordinates and native input only when the required control is not exposed semantically.

The adapter/operator must bind a trusted DCC target with DCC_MCP_UI_CONTROL_UIA_PROCESS_ID or DCC_MCP_UI_CONTROL_UIA_WINDOW_HANDLE before any Windows snapshot or mutation. The host resolves an exact PID/HWND, validates the caller's Windows session, user, and integrity level, then starts the prominent non-modal control notice before minting its opaque capability. Routine session start does not open a confirmation dialog.

The native DCC UI Control boundary imports that PID/HWND as a separate trusted scope, rejects construction without it, and revalidates the resolved native identity before the capsule, every capture, and every action. A request-supplied title, process name, PID, or HWND cannot authorize a different process.

Before a semantic mutation, the host re-resolves the actual descendant control and checks it plus every ancestor back to the scoped root. Password controls, cross-process descendants, and authentication or credential subtrees are hard denied even when the outer DCC window itself is allowed.

Raw pointer and keyboard input have a second gate and are disabled by default: the operator must also set DCC_MCP_COMPUTER_USE_ALLOW_RAW_INPUT=true in the adapter process. This environment variable is a hard ceiling: a request cannot enable allow_raw_coordinates or allow_keyboard_shortcuts while it is unset or false. Request PID, HWND, title, policy, and environment scopes are intersected; a request can narrow the trusted target but cannot replace it with another application. Process-name scopes are observation-only. Never widen the scope to the desktop.

The Windows backend hard-denies raw type. Enter non-sensitive text only with semantic set_text against an exact control_id; passwords, authentication codes, and other credential material require a user hand-off or a host-owned secure credential flow. The mock and CDP backends may retain backend-specific type support, so callers must not treat it as a portable Windows action. Windows keypress also rejects ordinary printable characters, including Shift-modified and AltGr text. Use it only for navigation/control/function keys or a genuine Ctrl/Alt shortcut; it is not a one-character text-entry bypass.

ui_control__act advertises a destructive annotation and accepts an optional intent consequence hint. The native host independently classifies the UIA control, focused/pointed control, keyboard chord, and requested intent; the hint can only raise the tier. Tier 2/3 operations use a trusted host-owned Windows confirmation dialog. There is no confirmed, approved, or environment-based approval field. A missing or denied confirmation returns approval_required.

With that operator-bound exact scope, ui_control__snapshot returns a bounded PNG through versioned shared memory plus a UIA tree, even when raw input is disabled. Host absence, protocol mismatch, UIA failure, and capture failure fail closed with no in-process or alternate-input fallback.

If that exact HWND is valid but minimized or hidden, do not search the desktop or switch input backends. ui_control__act supports the pre-snapshot actions get_window_state, restore_window, show_window, and activate_window. They carry the existing task grant and opaque HWND capability, never accept a replacement target, and use no pointer or keyboard input. The host revalidates PID/HWND ownership and hard target policy, audits each operation, and invalidates any old observation. After recovery, take a fresh snapshot before interacting with content.

For native DCC UI Control actions, keep one session_id and use this loop:

1. Call ui_control__snapshot with the exact target scope. It returns a PNG image, a snapshot_id, and observation metadata for that window generation.
2. Inspect the screenshot and UIA tree. Prefer a semantic action when a stable control is available. On Windows, enter non-sensitive text only with set_text and that control's exact control_id.
3. For visual fallback, call ui_control__act with the same snapshot_id and screenshot-relative x/y or path values. Use keys for keypress or Ctrl/Shift/Alt modifiers held during a pointer action. Windows keypress accepts navigation/control/function keys and genuine Ctrl/Alt shortcuts, not printable text. Do not use raw type on Windows.
4. Call ui_control__snapshot after every native action. Each native action consumes its observation; a newer snapshot or a moved/resized window makes old coordinates stale.
5. Use ui_control__wait_for for a UI condition, then snapshot again to verify.
6. Call ui_control__stop_computer_use in the success, failure, and abandoned-task cleanup path so the capsule, corner brackets, hotkey, and global input owner are released. If it returns cleanup_pending=true, retry cleanup and do not start another session; the cross-process input owner remains fenced until every pending key/button release is confirmed. Stopping does not clear an Ctrl+Alt+Esc interruption latch created during a UI Control session.

ui_control__wait_for remains interruptible while polling: Ctrl+Alt+Esc, an explicit ui_control__stop_computer_use, desktop loss, or backend cleanup cancels the wait without waiting for its condition timeout.

Agents should enter this loop only after a structured DCC operation returns unsupported or capability_missing; they should not ask the user to manually perform that missing GUI step. Keep native calls scoped to the same process_id or window_handle. Perform one action at a time and re-observe after every action; never chain guessed coordinates from an old image.

Coordinates are pixels in the returned PNG, which may be scaled down for a bounded MCP payload; they are not desktop coordinates. Never reuse them across actions or snapshots. On stale_control or stale_observation, restart from ui_control__snapshot.

The native session requires a visible, unlocked interactive desktop, a live target window, and the adapter and DCC process at the same Windows integrity level. While input control is active, click-through corner brackets mark the target window and a bottom-center capsule reads DCC UI Control · <app> | Ctrl+Alt+Esc to stop. Pointer actions display a transient cursor marker (and a following marker during drag) so the user can see where the agent is acting. The user stops control with Ctrl+Alt+Esc; ordinary Esc remains available to the target application. On user_interrupted, stop immediately, do not retry, do not switch to another input path, do not change session_id, and do not automatically start a new DCC UI Control session. The stop is latched across all DCC adapter processes in the same Windows logon session. Return control to the user. Resume only through an explicit ui_control__snapshot call with resume_computer_use=true. That flag only requests the flow: the native host always displays its own confirmation surface before clearing the global latch, so a model or adapter cannot approve itself. The native backend releases any held keys or mouse buttons before allowing more input. If Windows disconnects after a partial injection, those releases remain pending and retain the global input owner until reconnect makes them confirmable.

Windows desktop availability

Lock, disconnect, and secure-desktop transitions pause live UIA and raw input. They return desktop_unavailable without sending input or ending the logical ui_control session. Stop issuing UI operations, ask the user to unlock or reconnect, and do not poll autonomously. Keep the same session_id; structured DCC skills and MCP calls may continue only while the host adapter remains ready.

Never target LockApp, Windows Security, credential/authentication/password manager windows, the Windows Run dialog, terminals, PowerShell, or cmd. These are hard backend-enforced boundaries and must not be bypassed by another UI automation path. A DCC application's own script editor remains in scope because its target process is still the bound DCC process.

An ordinary user process cannot display the DCC UI Control capsule over the Windows lock screen or secure desktop. After the user unlocks or reconnects, discard all prior snapshot, observation, and control ids and call ui_control__snapshot again with the same exact target scope. A successful fresh snapshot re-establishes the corner brackets/capsule; resume_computer_use is still only for an explicit post-interruption resume. Run the DCC in a dedicated, always-unlocked VM when Windows GUI control must continue without interruption.

DCC UI Control executes on the adapter host, inside the specific interactive Windows logon session that owns the DCC process. A central gateway routes the tool call; it does not own the screenshot coordinate space. Never apply coordinates captured on the gateway, another machine, or another logon session to a remote DCC. An RDP disconnect or Windows session switch returns desktop_unavailable and retains the logical session. Reconnect to the DCC's session, then take a fresh exact-target snapshot before any UI action.

The returned image is bounded to the scoped target window; it is never a whole-desktop screenshot. That window may occupy or span monitors whose virtual desktop origins are negative and whose DPI scales differ. Continue to send coordinates relative to the returned PNG; the backend maps them through that observation's source rectangle and DPI metadata. A monitor add/remove, display-layout change, resolution change, or DPI/scaling change invalidates the observation. Discard its ids and take a fresh snapshot. The capsule follows the scoped target window across monitors in that same logon session.

CDP presets:

• DCC_MCP_UI_CONTROL_CDP_PRESET=reuse (default): attach to an existing DevTools endpoint first so the current browser profile, cookies, and tokens can be reused. Set DCC_MCP_UI_CONTROL_CDP_URL for an explicit HTTP or WebSocket CDP endpoint, or expose Chrome on DCC_MCP_UI_CONTROL_CDP_PORT / port 9222.
• DCC_MCP_UI_CONTROL_CDP_PRESET=isolated: launch Chrome with a temporary --user-data-dir for hermetic tests and demos.
• DCC_MCP_UI_CONTROL_CDP_PRESET=auroraview: attach to AuroraView's CDP endpoint. It uses DCC_MCP_UI_CONTROL_AURORAVIEW_CDP_PORT, then AURORAVIEW_CDP_PORT, then DCC_MCP_UI_CONTROL_CDP_PORT, and finally port 9222.
• DCC_MCP_UI_CONTROL_CDP_PRESET=edge: attach to or launch Microsoft Edge via CDP. It uses DCC_MCP_UI_CONTROL_EDGE_CDP_URL / _PORT before the shared CDP URL/port, and DCC_MCP_UI_CONTROL_EDGE_PATH when launching.
• DCC_MCP_UI_CONTROL_CDP_PRESET=agent-browser: use Vercel's agent-browser CLI, reading its CDP WebSocket URL through agent-browser get cdp-url after agent-browser open about:blank. Override the binary with DCC_MCP_UI_CONTROL_AGENT_BROWSER_BIN; this preset is suitable for CI when agent-browser install has provisioned Chrome for Testing.

Agent Loop

Use this loop:

1. Try the structured DCC skill, host API, or adapter script.
2. If it returns unsupported or capability_missing, call ui_control__snapshot for the exact application window.
3. ui_control__find to resolve a control by label, role, text, or object name.
4. ui_control__act to perform one scoped action using the resolved control id or screenshot coordinates when no semantic control is available.
5. ui_control__snapshot immediately to verify the result before another action.
6. Use ui_control__wait_for only for a known UI condition, then snapshot again.
7. Call ui_control__stop_computer_use when the fallback is complete or abandoned.

If an action returns stale_control, restart at ui_control__snapshot. If an action returns policy_disabled, prefer a native DCC skill or ask for an explicit policy change. On user_interrupted or desktop_unavailable, stop; do not follow a generic retry or fallback route.

Workflow Examples

Custom-drawn DCC controls and face shaping: prefer a native rig/parameter tool, then try ui_control__find for a semantic slider or handle. Only when the canvas, viewport manipulator, or face control is not exposed semantically, enable raw input at adapter startup and use one observation-fenced drag. Coordinates and the path below must come from the latest returned PNG; never infer them from an older frame. keys may hold Ctrl, Shift, or Alt for pointer actions when the DCC uses those modifiers for fine adjustment or an alternate manipulation mode.

dcc-mcp-cli ui-control snapshot --instance-id <id> --json '{"session_id":"face-shape","process_id":1234}'
dcc-mcp-cli ui-control act --instance-id <id> --json '{"session_id":"face-shape","process_id":1234,"action":"drag","intent":"ordinary_edit","button":"left","keys":["Shift"],"path":[{"x":612,"y":428},{"x":628,"y":424},{"x":646,"y":419}],"duration_ms":350,"snapshot_id":"<latest-snapshot-id>"}'
dcc-mcp-cli ui-control snapshot --instance-id <id> --json '{"session_id":"face-shape","process_id":1234}'

The second snapshot is mandatory immediately after the drag. Verify the visual result before deriving the next path; one native action consumes the preceding observation even when the face control moved only a few pixels.

Modal dialog: snapshot the scoped DCC/app window, find the button by label or role, click with the returned snapshot_id, then wait_for the button or dialog root to disappear. Verify completion through a native DCC skill when possible.

Settings panel: snapshot, find the labeled field or checkbox, set_text / toggle / set_checked, click Apply, then wait_for a status label such as Applied and snapshot again. Typed text is redacted from audit unless policy allows sensitive values. Use intent: account_or_access_change for a remote control/connection switch; the host also recognizes that label and always confirms it. Non-sensitive account fields are in scope, but password controls remain a user hand-off or application-owned OAuth/browser flow.

Recovery: on missing_window, confirm process/window scope instead of widening to the desktop. On timeout, inspect the last snapshot and either wait once more with a justified budget or switch to host diagnostics.