winapp-ui-automation

**If you can't run `winapp` yourself** (no shell, or the command is denied), give the user the exact `winapp` command(s) for their project instead of only describing the steps. ## When to use - Inspecting a running Windows app's UI from the command line - AI agents interacting with Windows applications (clicking buttons, reading text, taking screenshots) - Verifying UI state during development or testing - Automating UI workflows without Playwright or Selenium - Debugging WinUI 3, WPF, WinForms, Win32, or Electron app UIs ## Prerequisites To measure WinUI work while driving a scenario, use the [performance skill](../winapp-performance/SKILL.md) to bracket UI actions with capture markers. - For UIA mode (any app): No setup needed — works with any running Windows app - For input-injecting verbs (`click`, `hover`, `drag`, `touch`, `pen`, `scroll --wheel`/`--wheel-horizontal`, `send-keys --via send-input`): an **unlocked, interactive desktop** with the target window foregroundable. On a locked/secure desktop they fail fast with `no_interactive_desktop`. The UIA-pattern verbs (`inspect`, `search`, `get-*`, `wait-for`, `set-value`, `invoke`, `scroll --direction/--to`) are headless/locked-session friendly — prefer them in CI. - `screenshot` is **not** in that group: it always takes an exclusive turn, so it queues behind other UI workflows, and capture can need a usable interactive desktop — the engine restores the target if it is minimized, and falls back to foregrounding it when frame capture is unavailable or `--capture-screen` is used. - `--capture-screen` needs **exactly one window**. `-w <hwnd>` gives it one: that window's screen region, including anything visibly on top of it. If `-a` matches several top-level or owned windows the command fails with `invalid_arguments` before capturing; run `winapp ui list-windows -a <app>` and retry with `-w <hwnd>`. - **If other UI workflows may run at the same time**, set one workflow id per logical workflow (see below). Nothing breaks without it, but your commands will not be recognized as belonging together. ## Coordinating with other UI workflows Windows has one foreground window, one keyboard focus, one cursor, and one input stream. `winapp ui` therefore makes desktop-driving commands take **cooperative turns** so concurrent workflows cannot steal each other's focus or dismiss each other's menus. That is always on. Read-only commands never wait. Keeping the desktop *across* commands is opt-in — without an id, each command is a one-shot that releases the desktop the moment it finishes: ```powershell # Set once per logical UI workflow — same value for cooperating calls, different values for # independent workflows (even from the same agent). $env:WINAPP_UI_WORKFLOW_ID = [guid]::NewGuid().ToString() ``` Rules that matter when driving this from an agent: - **Each tool call usually gets a fresh shell**, and there is no process-ancestry fallback, so commands are grouped ONLY by the id you inject. Without it every call is its own workflow. - **A workflow with an id keeps its turn for four seconds** after its last command. That covers back-to-back commands in one script; it deliberately expires while you are reasoning. Treat it as a fallback for when you cannot say you are done — not as the way to finish. - **Run `winapp ui yield` when you finish a known sequence.** It hands the desktop over immediately instead of making a waiting workflow sit out a four-second grace nobody needs. Yielding twice, or after the grace lapsed, is a harmless success. - **After a reasoning gap, replay your setup.** Another workflow may have used the desktop, so reopen the menu / re-navigate, re-resolve the element, then act. Do not assume transient UI survived. - **Prefer one tight script over many round trips** for a known sequence: `winapp ui invoke View -w $hwnd; winapp ui search "Status bar" -w $hwnd; winapp ui click "Status bar" -w $hwnd; winapp ui yield`. - **`record` shares the turn with its own workflow**, so same-workflow clicks and typing are captured while it runs — but only if both commands carry the same id. A `record` with no id blocks everyone else for its whole duration. - **Ordering is owner affinity, then FIFO.** An active workflow may keep issuing commands ahead of others already waiting; once it yields or its grace expires, waiters are served in arrival order. - **Waiting is indefinite and cancellable.** After one second a notice on stderr says what holds the desktop and for how long (silent under `--json` and `--quiet`); Ctrl+C exits `130` with error code `cancelled` and the command never ran. - **There is no hard cap** — a long script, unbounded recording, or failure loop can block other mutating workflows until it finishes or is stopped. Commands that never wait: `status`, `list-windows`, `inspect`, `search`, `get-*`, `wait-for`. Commands that wait for the turn but never take the desktop (headless/locked-session friendly): `set-value`, `scroll-into-view`, `scroll --direction`/`--to`, `record`. They mutate the app, so they queue behind another workflow. Inside your own workflow they overlap with other shared work — that is how `record` captures the `set-value` calls it is recording — but they still wait behind an earlier `DesktopExclusive` command of your own workflow, so a `click` followed by a `set-value` runs in the order you wrote it. Commands that take the desktop exclusively: `invoke`, `click`, `drag`, `hover`, `scroll --wheel`/`--wheel-horizontal`, `touch`, `pen`, `focus`, `send-keys`, `screenshot`. ```powershell # Finish a workflow deliberately rather than leaving the desktop reserved for four more seconds. winapp ui yield ``` ## Common patterns ### Read within a specific container ```powershell winapp ui search "Welcome to MyApp" -a myapp --root MailRow --type Text --class-name TextBlock winapp ui get-value Subject -a myapp --root MailRow --type TextBox winapp ui wait-for Subject -a myapp --root MailRow --type Edit --value Ready --timeout 10000 winapp ui search --class-name CalendarSurface -a myapp # no text selector needed ``` Use `--root`, `--type`, and `--class-name` together or separately on `search`, `get-property`, `get-value`, `wait-for`, and the commands that act on one selected element (`invoke`, `set-value`, `click`, `focus`, and so on; not `send-keys --target`). `--type` and `--class-name` work without a selector; commands other than `search` then need exactly one match, and an ambiguous match lists the first five with their slugs. `--root` alone is not a query. For `set-value`, pass `""` as the selector. For `click`, `hover`, `touch`, and `pen`, filters can't be combined with `--at` or `--path`. The root must be unique; only its descendants match. `wait-for` re-resolves it every poll, including when it is initially absent. Type names and literal whole ClassName values ignore case. The only type aliases are `TextBox` → `Edit` and `TextBlock` → `Text`. See [Scoped and typed queries](https://github.com/microsoft/WinAppCli/blob/main/docs/ui-automation.md#scoped-and-typed-queries) for the full type vocabulary, boundaries, and error behavior. ### Discover and interact ```powershell # See what's clickable, then screenshot for context winapp ui inspect -a myapp --interactive; winapp ui screenshot -a myapp # Click and verify the page changed winapp ui invoke btn-settings-a1b2 -a myapp; winapp ui wait-for pn-settingspage-c3d4 -a myapp --timeout 3000; winapp ui screenshot -a myapp # Fill a form and submit winapp ui set-value txt-searchbox-e5f6 "hello" -a myapp; winapp ui invoke btn-submit-7a90 -a myapp; winapp ui screenshot -a myapp ``` ### Find visible text and click it ```powershell # Search by text — output shows invokable ancestor winapp ui search "Save changes" -a myapp # Output: # lbl-savechanges-a1b2 "Save changes" (120,40 80x20) # ^ invoke via: btn-save-c3d4 "Save" # Invoke by text — auto-walks to parent Button winapp ui invoke 'Save changes' -a myapp ``` ### Navigate multi-page apps ```powershell # Click nav item, wait for page, inspect what's available winapp ui invoke itm-samples-3f2c -a myapp; winapp ui wait-for pn-samplespage-b4e7 -a myapp; winapp ui inspect -a myapp --interactive ``` ### Choose an exact action in tests ```powershell winapp ui invoke SettingsCategory -a myapp --action select winapp ui invoke AgreeCheckbox -a myapp --action toggle-on --json winapp ui invoke Open -w <dialog-HWND> --type Button --action invoke ``` Use `--action` to avoid automatic pattern and ancestor fallback. Omit it for the existing automatic behavior. With `--root`, `--type`, or `--class-name`, the filters follow the same matching rules as read queries: exactly one element must match inside the selected app/window, or the command fails without acting, and the ancestor fallback is skipped. See the [action reference](https://github.com/microsoft/winappCli/blob/main/docs/ui-automation.md#invoke) for supported actions, scope, idempotent toggles, and failure recovery, and the [JSON envelope](references/ui-json-envelope.md#ui-invoke---json) for action results. ### Disambiguate duplicate elements ```powershell # Text matches by substring, but an element named exactly the text wins winapp ui invoke Submit -a myapp # acts on Button "Submit", not Button "Submit Order" # When several elements share that name, the error shows slugs for each — pick the right one winapp ui invoke Save -a myapp # → Selector matched 2 elements: # [0] Button "Save" → btn-save-a1b2 # [1] Button "Save" → btn-save-c3d4 # Use the slug: winapp ui invoke btn-save-c3d4 -a myapp ``` ## Key concepts - **Selector brackets**: `inspect` and `search` output shows selectors in `[brackets]` — use the bracketed value with other `ui` commands. Selectors are either AutomationId (stable, developer-set) or generated slug (e.g., `btn-name-hash`). - **AutomationId selectors**: When an element has a unique AutomationId, it becomes the selector directly (e.g., `[MinimizeButton]`). These survive layout changes and localization — preferred for stable targeting. - **Slug selectors**: When no unique AutomationId exists, a generated slug is used (e.g., `[btn-close-a2b3]`). Format: `prefix-name-hash`. May go stale after UI changes. - **Plain text search**: `search` and `invoke` accept plain text — `search Minimize` finds elements with "Minimize" in their Name or AutomationId (substring, case-insensitive). No special syntax needed. Commands that act on one element prefer an exact AutomationId match, then the only element whose Name is exactly the text (ignoring case); `search` still lists every match. - **`--interactive` flag**: Filters to elements you can invoke, click, or set-value (including text documents) with auto-depth 8 — the fastest way to see what you can act on - **Invokable ancestor surfacing**: When a search result isn't invokable, the nearest invokable parent is shown with its selector - **`;` chaining**: Chain commands with `;` to run multiple operations in one call, reducing agent round-trips - **`-a` vs `-w`**: Use `-a` to find apps by name/title/PID. Use `-w <HWND>` for stable window targeting - **Element markers**: `[on]`/`[off]` for toggles, `[collapsed]`/`[expanded]`, `[scroll:v]`/`[scroll:h]`/`[scroll:vh]` for scrollable containers, `[offscreen]`, `[disabled]`, `value="..."` for editable elements ## Usage ### Connect and discover ```powershell # Core loop and examples: `winapp ui --help`. Connect and see interactive elements in one call: winapp ui status -a myapp; winapp ui inspect -a myapp --interactive ``` ### Inspect element tree ```powershell winapp ui inspect -a myapp --interactive # elements you can invoke, click, or set-value; auto-depth 8 winapp ui inspect -a myapp --depth 5 # deeper tree at depth 5 winapp ui inspect txt-searchbox-e5f6 -a myapp # subtree rooted at element winapp ui inspect btn-settings-a1b2 -a myapp --ancestors # walk up from element to root winapp ui inspect -a myapp --hide-offscreen # hide offscreen elements ``` ### Find elements ```powershell winapp ui search Close -a myapp # finds elements with "Close" in name or automationId winapp ui search Button -a myapp # finds elements with "Button" in name (also matches type names) winapp ui search image -a myapp # case-insensitive substring match ``` ### Screenshot ```powershell # App screenshot (may include owned windows in a labeled composite) winapp ui screenshot -a myapp --output page.png # Crop to element; capture with popups visible winapp ui screenshot txt-searchbox-e5f6 -a myapp --output search.png winapp ui list-windows -a myapp # use the main window's HWND below winapp ui screenshot -w <hwnd> --capture-screen --output with-popups.png # Bring window to foreground first (matches what the user is currently seeing) winapp ui screenshot -a myapp --focus --output focused.png ``` Default capture includes owned windows even with an explicit main HWND; it produces one labeled composite, not separate image files. An element selector for an item in an open menu, flyout, tooltip, or teaching tip crops from that item's popup window, so you don't need `--capture-screen` for it. For scope and on-screen overlay placement, see [Screenshot](https://github.com/microsoft/WinAppCli/blob/main/docs/ui-automation.md#screenshot). With `--on sandbox`, the reported screenshot path is the delivered host destination. ### Record video (H.264 MP4) Record a window or element region to MP4. Prefer a positive `--duration-sec N` for agents and scripts; otherwise recording waits for a stop signal. npm helpers require `durationSec` (integer 1–86400). Their abort signal is forceful cancellation, not graceful stop. ```powershell # Record a window for 10s at 15 fps winapp ui record -a myapp --duration-sec 10 --fps 15 --output demo.mp4 # Recommended agent evidence: MP4 plus timestamped JPEGs and an NDJSON index winapp ui record -a myapp --frames --duration-sec 10 --fps 10 --output evidence.mp4 --json # Include overlays/popups (captures from screen DC; may include occluding windows) winapp ui record -a myapp --capture-screen --duration-sec 5 --output with-popups.mp4 # Programmatic stop: pipe a newline to stop and finalize the MP4 (for agent/script callers) "" | winapp ui record -a myapp --json --output capture.mp4 ``` - Default `--duration-sec 0` records until Ctrl+C, a newline, or EOF on redirected stdin. - `--frames` writes `<output-name>.frames` with a manifest, NDJSON index, and changed JPEGs. It supports 1-30 fps and `--max-edge` 64-4096 (default 1280), with a 1 GiB cap. Use `elapsedMs` to bound transitions. - Existing recording outputs are rejected by default. Use a fresh path, or explicitly request `--overwrite` to replace them after the new take finishes. Previous frame directories are archived, not deleted. On partial failure, keep the reported evidence and follow `recoveryHint`. - `--capture-screen` captures from the screen DC so overlays and popups are included; the window is brought to the foreground first. When WGC is unavailable and `--capture-screen` is not passed, the CLI returns an error — re-run with `--capture-screen` to consent to screen-DC capture. Because the screen DC captures whatever is genuinely in front, the target's foreground is **verified immediately before capture**; if activation was refused the command fails with `foreground_not_target` and writes nothing rather than returning an image of the wrong window. - Providing a selector that doesn't match any element fails immediately with `element_not_found` (rather than silently recording the whole window). - A selector for an item in an open menu, flyout, tooltip, or teaching tip records from that item's popup window, not the window behind it. - `--json` writes the final result to stdout and one JSON event per line to stderr. ### Hover (for tooltips, flyouts, hover states) `--dwell-time <ms>` sets how long to wait after hovering (default: 800, range: 0–10000). ```powershell winapp ui list-windows -a myapp # use the main window's HWND below, not the tooltip's # Hover to trigger tooltip, then capture it (default 800ms dwell) winapp ui hover btn-info-a1b2 -a myapp; winapp ui screenshot -w <hwnd> --capture-screen --output tooltip.png # Longer dwell for apps with slow tooltip timers winapp ui hover btn-info-a1b2 -a myapp --dwell-time 1200; winapp ui screenshot -w <hwnd> --capture-screen ``` ### Click or hover a screen point When the press needs a point rather than an element's center (inside a WebView2 page, or one part of a large control), pass `--at x,y` with screen coordinates from `ui inspect` instead of a selector. `--at` can't be combined with a selector. ```powershell winapp ui click --at 1520,780 -a myapp # add --right or --double as needed winapp ui hover --at 1520,780 -a myapp ``` ### Send keyboard input Synthesize keystrokes — the keyboard counterpart to `click`. Use for arrow/Tab/Enter navigation, shortcuts, and per-keystroke typing (vs `set-value`'s atomic write). Tokens are whitespace-separated: named keys (`enter`, `down`, `tab`, `esc`, `f5`), modifier combos (`ctrl+shift+t`), literal text (`hello`), and raw virtual keys (`vk=0xNN`). ```powershell # Keyboard navigation then commit winapp ui send-keys "down down enter" -a myapp # Type the literal words "down down enter" instead of pressing those keys (text= escapes each token) winapp ui send-keys "text=down text=down text=enter" -a myapp # Same intent, less typing: --verbatim types the whole argument literally (and keeps exact whitespace) winapp ui send-keys "down down enter" -a myapp --verbatim # Shortcut: select all and delete winapp ui send-keys "ctrl+a delete" -a myapp # Focus a field, then type text into it winapp ui send-keys "Hello world" --target txt-name-a1b2 -a myapp # Transport: --via post-message (default, HWND-targeted, bypasses UIPI) or send-input (OS-wide)