argent-device-interact

## Unified tool surface All interaction tools below accept a `udid` parameter and auto-dispatch iOS vs Android based on its shape (UUID → iOS simulator, anything else → Android adb serial). You use the same tool names on both platforms. For platform-specific caveats (Metro `adb reverse`, locked-screen describe errors, etc.), see § 9 Platform-specific notes at the bottom. ## 1. Before You Start If you delegate simulator tasks to sub-agents, make sure they have MCP permissions. Use `list-devices` to get a target id. Results are tagged with `platform` (`ios` or `android`); booted/ready devices come first. Pick the first entry that matches the platform you need — if none are ready, call `boot-device` with `udid` (iOS) or `avdName` (Android). See `argent-ios-simulator-setup` / `argent-android-emulator-setup` for full setup flow. **Load tool schemas before first use.** Gesture tools (`gesture-tap`, `gesture-swipe`, `gesture-pinch`, `gesture-rotate`, `gesture-custom`) may be deferred — their parameter schemas are not loaded until fetched. Always use ToolSearch to load the schemas of all gesture tools you plan to use **before** calling any of them. If you skip this step, parameters may be coerced to strings instead of numbers, causing validation errors. ## 2. Best Practices 1. **Always refer to tapping_rule** from your argent.md rule before tapping. 2. Before performing interactions, consider whether they can be **dispatched sequentially** - more on that in `run-sequence`. 3. **Use `gesture-swipe` for lists/scrolling**, not `gesture-custom`, unless you need non-linear movement. Consider whether you need multiple swipes, if yes - use `run-sequence`. 4. **Tap a text field before typing** — on iOS try `paste` first then fall back to `keyboard`; on Android use `keyboard` directly (`paste` is iOS-only). 5. **Coordinates are normalized** — always 0.0–1.0, not pixels. 6. **For native iOS app navigation, prefer `describe` first.** It works on any screen without app restart. Do not navigate from screenshots on regular in-app screens unless `describe` failed to expose a reliable target. Use `native-describe-screen` only when you need app-scoped UIKit properties. ## 3. Opening Apps **Never navigate to an app by tapping home-screen icons.** Use `launch-app` or `open-url` — they are instant and reliable. ### launch-app — by bundle ID ```json { "udid": "<UDID>", "bundleId": "com.apple.MobileSMS" } ``` Common IDs: `com.apple.MobileSMS` (Messages), `com.apple.mobilesafari` (Safari), `com.apple.Preferences` (Settings), `com.apple.Maps`, `com.apple.Photos`, `com.apple.mobilemail`, `com.apple.mobilenotes`, `com.apple.MobileAddressBook` (Contacts) ### open-url — by URL scheme ```json { "udid": "<UDID>", "url": "messages://" } ``` Common schemes: `messages://`, `settings://`, `maps://?q=<query>`, `tel://<number>`, `mailto:<address>`, `https://...` (Safari) ## 4. Choosing the Right Tool | Action | Tool | Notes | | ---------------- | ---------------- | ---------------------------------------------------------------------- | | Multiple actions | `run-sequence` | Batch steps in one call (no intermediate screenshots) | | Open an app | `launch-app` | **Always — never tap home-screen icons** | | Restart an app | `restart-app` | Terminate and relaunch by bundle ID | | Open URL/scheme | `open-url` | Web pages, deep links, URL schemes | | Single tap | `gesture-tap` | Buttons, links, checkboxes | | Scroll/swipe | `gesture-swipe` | Straight-line scroll or swipe | | Long press | `gesture-custom` | Context menus, drag start | | Drag & drop | `gesture-custom` | Complex drag interactions | | Pinch/zoom | `gesture-pinch` | Two-finger pinch with auto-interpolation | | Rotation | `gesture-rotate` | Two-finger rotation with auto-interpolation | | Custom gesture | `gesture-custom` | Arbitrary touch sequences, optional interpolation | | Hardware key | `button` | Home, back, power, volume, appSwitch, actionButton | | Type text (fast) | `paste` | iOS only. Form fields — uses clipboard | | Type text | `keyboard` | iOS+Android. Fallback when paste fails; supports Enter, Escape, arrows | | Rotate device | `rotate` | Orientation changes | ## 5. Finding Tap Targets IMPORTANT. When moved to a different screen after an action or do not know the coordinates of component, **always** perform proper discovery first. | App type | Discovery tool | What it returns | | --------------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Target app discovery | `describe` | Accessibility element tree for the current device screen (iOS AX-service or Android uiautomator) with normalized frame coordinates. Works on any app, system dialogs, and Home screen — no app restart or `bundleId` required | | React Native | `debugger-component-tree` | React component tree with names, text, testID, and (tap: x,y) | | App-scoped native | `native-describe-screen` | Low-level app-scoped accessibility elements with normalized and raw coordinates; requires `bundleId` | | Permission / system modal overlay | `describe` | `describe` detects system dialogs automatically and returns dialog buttons with tap coordinates. Fall back to `screenshot` only if `describe` does not expose the controls | | Final visual fallback | `screenshot` | Use only when discovery tools cannot inspect the current UI reliably. Do not derive routine in-app navigation targets from screenshots | Point follow-up native diagnostics after you already have a candidate point: - `native-user-interactable-view-at-point`: deepest native view that would receive touch at a known raw iOS point; requires `bundleId` - `native-view-at-point`: deepest visible native view at a known raw iOS point; requires `bundleId` ### If `describe` Fails Read the exact error and choose the action that matches it: - Error mentions `ax-service` not available or daemon startup failure: the ax-service daemon could not start. Check that the simulator is booted. Use `screenshot` as a temporary fallback, or use `native-describe-screen` with an explicit `bundleId` if the app has native devtools injected. - `describe` returns an empty element list: the screen may be blank, loading, or showing content without accessibility labels. Use `screenshot` to see what is visible, then retry after the content has loaded. - `describe` succeeds but is not detailed enough for a React Native app: use `debugger-component-tree` next. - You need app-scoped inspection with full UIKit properties (`accessibilityIdentifier`, `viewClassName`): use `native-describe-screen` with an explicit `bundleId`. This requires native devtools (dylib) injection — call `restart-app` first if needed. - You already have a candidate point and want to confirm what would actually receive touch: use `native-user-interactable-view-at-point`. Use `native-view-at-point` when you want the visually deepest view instead of the hit-test target. ## 6. Tool Usage ### gesture-tap — Single tap at a point ```json { "udid": "<UDID>", "x": 0.5, "y": 0.5 } ``` Coordinates: `0.0` = left/top, `1.0` = right/bottom. Before tapping near the bottom of the screen in React Native apps, check that "Open Debugger to View Warnings" banners are not visible — tapping them breaks the debugger connection. Close them with the X icon if present. ### gesture-swipe — Straight-line gesture ```json { "udid": "<UDID>", "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } ``` Swipe **up** (`fromY > toY`) = scroll content **down**. Default duration: 300ms. Optional: `"durationMs": 500` for slower swipe. ### gesture-pinch — Two-finger pinch ```json { "udid": "<UDID>", "centerX": 0.5, "centerY": 0.5, "startDistance": 0.2, "endDistance": 0.6 } ``` All values are normalized 0.0–1.0 (fractions of screen, not pixels) — same as all other gesture tools. `startDistance: 0.2` means fingers start 20% of the screen apart; `endDistance: 0.6` means they end 60% apart. `startDistance < endDistance` = pinch out (zoom in). `startDistance > endDistance` = pinch in (zoom out). Defaults: `angle: 0` (horizontal), `durationMs: 300`. Optional: `"angle": 90` for vertical axis, `"durationMs": 500` for slower pinch. ### gesture-rotate — Two-finger rotation ```json { "udid": "<UDID>", "centerX": 0.5, "centerY": 0.5, "radius": 0.15, "startAngle": 0, "endAngle": 90 } ``` All positions and radius are normalized 0.0–1.0 (fractions of screen, not pixels). `radius: 0.15` means each finger is 15% of the screen away from center. `endAngle > startAngle` = clockwise. Default duration: 300ms. Optional: `"durationMs": 500` for slower rotation. ### gesture-custom — Custom touch sequence For long-press, drag-and-drop, and other complex sequences, see `references/gesture-examples.md`. Set `"interpolate": 10` to auto-generate smooth intermediate Move events between keyframes. ### button — Hardware button press ```json { "udid": "<UDID>", "button": "home" } ``` Values: `home`, `back`, `power`, `volumeUp`, `volumeDown`, `appSwitch`, `actionButton` ### paste — Type text into focused field (iOS only) ```json { "udid": "<UDID>", "text": "Hello, world!" } ``` Tap the field first, then paste. Fall back to `keyboard` if it doesn't work. On Android the call is rejected by the capability gate ("Tool 'paste' is not supported on android") — use `keyboard` directly. ### keyboard — Type text or press special keys ```json { "udid": "<UDID>", "text": "search query", "key": "enter" } ``` Special keys: `enter`, `escape`, `backspace`, `tab`, `space`, `arrow-up`, `arrow-down`, `arrow-left`, `arrow-right`, `f1`–`f12`. Optional: `"delayMs": 100` between keystrokes (default 50ms). ### rotate — Change orientation ```json { "udid": "<UDID>", "orientation": "LandscapeLeft" } ``` Values: `Portrait`, `LandscapeLeft`, `LandscapeRight`, `PortraitUpsideDown` --- ## 7. Screenshots Use the explicit `screenshot` tool only when: - You need the initial screen state before any action. - The auto-attached screenshot shows a transitional or loading frame. - You require extra context. - You want to check state after a delay (e.g. waiting for a network response). - A permission dialog, system alert, or native modal overlay is visible and `describe` did not expose reliable targets. When using `screenshot` for permission or native modal navigation: - Do not switch to screenshot-driven navigation just because a modal is visible. On regular app screens and in-app modals, keep using `describe`. - Prefer obvious, centered alert buttons such as `Allow`, `OK`, `Don't Allow`, `Not Now`, or `Continue`. - Tap one control at a time and inspect the returned auto-screenshot before doing anything else. - After the modal is dismissed, return to normal discovery with `describe`, `native-describe-screen`, or `debugger-component-tree`. Optional rotation parameter: `{ "udid": "<UDID>", "rotation": "LandscapeLeft" }` — rotates the capture without changing simulator orientation. Screenshots are downscaled by default (30% of original resolution) to reduce context size. `scale` accepts values from 0.01 to 1.0. If UI elements are hard to read or you need to inspect fine detail, pass `scale: 1.0` to get full resolution: `{ "udid": "<UDID>", "scale": 1.0 }`. ### Troubleshooting | Problem | Solution |