windows-desktop-control

name	windows-desktop-control
description	Operate Windows applications from structured accessibility, application-object, and graphics draw-call scenes without screenshots or coordinate clicks. Use for isolated background apps, custom canvases such as NI Multisim, and game menus such as Minecraft Bedrock pause/settings UI.

Windows Scene Control

Use desktop-control-v2 as a scene-graph controller. The live preview is for the human only. Never derive an action from its pixels and never request or return its image.

Required workflow

1. Call windows_capabilities and require agent_observation: scene_json_only.
2. Create a named isolated session. Keep its human-only preview open.
3. Launch the target process and attach the architecture-matched scene hook when a graphics adapter is required.
4. Call windows_scene_refresh, then inspect windows_scene_snapshot or query with windows_scene_find.
5. Select a node by ID, role, name, state, adapter, or declared action.
6. Call windows_scene_action only with an action exposed by that node.
7. Verify mutations from a later semantic state or scene revision. A transport success is not proof of application success.

Hard boundaries

• Do not use screenshots, raster frames, OCR, absolute screen coordinates, or a generic click action.
• Do not invent semantics for an anonymous mesh. Preserve its render-object fidelity until an application adapter identifies it.
• Do not expose private action bindings in observations. The server resolves bindings from node IDs.
• Do not send arbitrary Win32 messages. Only execute an allowlisted binding attached to an observed node.
• Do not move the user's real pointer or send input to the visible desktop.

Adapter order

Prefer the richest available source and merge rather than replace lower tiers:

1. Application object model: component/net/pin graphs, menu models, game narrator/focus state.
2. Accessibility: UI Automation or MSAA patterns.
3. Native control model: HWND controls, command IDs, list/combo state.
4. Render model: draw groups, text/layout, paths, meshes, transforms, materials, and z-order.

For game menus, expose focus navigation such as focus_next, focus_previous, activate, and back; do not expose world-play controls unless the user separately scopes them.