| name | sim-use |
| description | Drive iOS Simulator and Android emulator/device screens for AI agents. Use when asked to automate a simulator or emulator, tap/swipe/type on a device, describe UI, take a screenshot, or interact with a mobile app. |
0. Preflight
Before first interaction with a device, run the preflight check:
python3 scripts/preflight.py --device <UDID>
This verifies sim-use is installed, the device is reachable, and the daemon is healthy. If you don't have the script, do the checks manually:
sim-use --version — confirm sim-use is on PATH.sim-use devices — confirm the target device is listed and booted/connected.sim-use ui --device <UDID> — confirm you can read the screen.
--device is optional when only one simulator is booted or one daemon is running. For Android, run sim-use android init --device <serial> once to install the bridge APK.
1. The observe-act loop
Every interaction follows the same cycle: observe → act → verify.
Observe
sim-use ui --device <UDID>
Read the outline. Each element has an @N alias and optionally a #<id> identifier. List cells carry #N (dominant list) or #N@M (scoped).
Frames in the JSON output (--json: entries[].frame, screen) are in platform-native units — iOS points, Android pixels. Key off the envelope's platform field before doing math on coordinates across platforms.
Act
Pick a selector, in order of preference:
| Selector | When to use |
|---|
tap @N | Right after ui. Fastest, cache-backed. |
tap #<id> | Stable across minor layout changes. Paste from the outline. |
tap --label 'X' | Scripted flows. Combine with --wait-timeout for transitions. |
tap --label-regex '...' | Dynamic labels with counters/timestamps. Anchor with ^...$. |
tap --label-contains 'X' | Substring match when exact label is unknown. |
tap -x N -y N / tap --point x,y | Last resort for elements with no AX data. |
Disambiguate collisions with --element-type or --frame minY=0.7r (see references/cheatsheet.md).
Verify
Always verify after acting — commands are fire-and-forget:
sim-use ui --device <UDID>
sim-use screenshot --device <UDID> --output after.png
Common moves
| Task | Command |
|---|
| Scroll down | sim-use gesture scroll-up --device <UDID> (scroll-up = content moves up = page down) |
| Type text | sim-use type 'hello' --device <UDID> |
| Paste unicode | sim-use paste 'こんにちは 🎉' --device <UDID> (iOS: needs hardware keyboard) |
| Hardware button | sim-use button home --device <UDID> |
| Android back | sim-use button back --device <UDID> |
| Wait for animation | sleep 0.4 between commands, or --pre-delay 0.5 |
| Toggle/switch | sim-use tap @N --duration 0.05 --device <UDID> (UISwitch needs a brief hold) |
| Swipe | sim-use swipe --from 50,500 --to 350,500 --device <UDID> |
| Pinch zoom in | sim-use gesture pinch-out --device <UDID> (two-finger spread) |
| Rotate | sim-use gesture rotate-cw --angle 90 --device <UDID> |
2. Pitfalls
Quick symptom index — see references/pitfalls.md for detailed recipes.
| Symptom | Cause | Fix |
|---|
tap --label hits wrong element | Label collision (e.g. header and tab bar share text) | Add --frame minY=0.7r or --element-type to narrow |
tap @N fails after navigation | Alias cache is stale | Re-run ui before tapping |
App: line shows wrong app | System layer (alert, share sheet) is on top | Dismiss it first, then re-run ui |
multipleMatches error | Several elements share the selector | Use --frame, --element-type, or a more specific selector |
| Tap lands but nothing happens | Animation in progress, or element not yet interactive | Add --pre-delay 0.3 or --wait-timeout 3 |
iOS: paste drops text | Soft keyboard only; HID Cmd+V is ignored | Use paste --via-menu --target-id <id> |
Android: paste denied | Background clipboard access blocked | Use type instead |
Outline shows U+FFFC in label | iOS icon placeholder character | Match with --label-regex excluding the prefix |
[i] … covers ~N% of the screen warning (text output, or --json top-level advisory key) | The selector resolved to a near-full-screen wrapper (common on Flutter/canvas UIs) and the tap hit its center, likely missing the intended control | Re-run ui and target the control via @N/#<id>, or pass explicit -x/-y/--point |
[i] Screen orientation could not be confirmed… / …coordinates may be stale… advisory | Device/app is rotated (the App: header shows a tag like (landscape-right)) and orientation self-calibration couldn't verify the mapping, or the @N snapshot predates a rotation | Re-run ui and tap again; selectors handle rotation automatically once calibration succeeds. Explicit -x/-y/--point is always device-native portrait space |
3. Crash awareness
See references/crash-awareness.md for the full protocol. Summary:
sim-use watches for the target process disappearing between commands. When it detects a crash:
================ PROCESS DISAPPEARED ================
com.example.app (pid 12345) was alive at the previous command and is GONE now.
On Android, ui also detects the AOSP system crash dialog directly from the accessibility tree.
Mandatory response:
- STOP. Do not silently relaunch or continue.
- Report the crash to the user with the banner text.
- Wait for instructions before proceeding.
After an intentional relaunch, call sim-use app-state --reset to clear the signal.
4. Escalation
Stop and ask the user when:
- A selector collision cannot be resolved with available disambiguators.
- Preflight fails and autofix does not recover.
- The task requires a destructive action (deleting data, uninstalling an app).
- You've retried the same action 3 times without progress.
5. Exit checklist
Before reporting a task as complete:
- Run
sim-use ui (or screenshot) to capture the final state.
- Confirm the screen matches the intended outcome.
- If the outcome is ambiguous, show the final
ui output or screenshot to the user.
