| name | interactive-sim-ux-audit |
| description | Drive the running game apps in the iOS Simulator with idb (tap / describe / screenshot) to find UX + layout bugs that snapshot tests cannot — navigation, modals, back-stack, completion, safe-area/Dynamic-Island clipping, offline/signed-out flows. Use when asked to "test the UI", "find UX problems", or verify an interactive flow. |
Interactive Simulator UX Audit (idb-driven)
Snapshot tests render views in a fixed-size NSHostingView — they do not model
navigation, taps, the device safe area, or the Dynamic Island. A whole class of bugs
only shows when you actually drive the app: blank-board-after-tap (#491), unplayable
when iCloud signed out (#512), completion icon clipped by the Dynamic Island (#518).
This skill is how to run that audit.
Prereq: idb (one-time install — NOT Homebrew)
The no-Homebrew/CocoaPods policy does not forbid idb. Install via direct GitHub
release download (user-authorized path). Verified on macOS 26 / Xcode 26 / arm64:
idb_companion: download idb-companion.universal.tar.gz (latest = v1.1.8; 2022 but
works) from https://github.com/facebook/idb/releases → extract to
~/idb-tools/companion/idb-companion.universal/ (binary in bin/, Frameworks/
sibling). The objc duplicate-class warning for FBProcess is non-fatal.idb CLI: pip3 install --user fb-idb → ~/Library/Python/3.9/bin/idb.- Put both on PATH at
~/.local/bin (already on the MCP-inherited PATH): symlink idb;
for the companion use a wrapper script that execs the real binary's absolute
path (a bare symlink breaks @executable_path/../Frameworks rpath).
- Verify:
idb list-targets, then idb ui describe-all --udid <booted-sim> returns the
a11y tree (frames + AXLabel) in device-point space (iPhone 17 Pro = 402×874 pt).
See [[idb-sim-ui-driving-works]] (memory) for the exact commands.
Build + install the app under test — use a CURRENT build, from the MAIN checkout
- Check the installed version first. A stale build silently invalidates findings —
e.g. the sim once held a
1.0.0 build; Settings → Version exposes it. Confirm it's the
version you mean to test before reporting any bug.
- Build for the sim:
xcodebuild -workspace Game.xcworkspace -scheme <Sudoku|Minesweeper> -sdk iphonesimulator -configuration Debug -destination 'platform=iOS Simulator,name=iPhone 17 Pro' -derivedDataPath build/sim build, then xcrun simctl install <udid> <App.app>.
- Build from the main checkout, NOT an agent worktree. Worktrees lack the gitignored
Tuist/AdMob.xcconfig + secrets/.env, so the Debug build's AppComposition.live()
hits an assertion at launch (EXC_BREAKPOINT) → app drops to springboard. That crash is
an env artifact, not a code bug. (Tuist needs tuist install && tuist generate --no-open
first; the workspace is gitignored.)
The drive loop
idb ui describe-all --udid <udid> # get element frames + AXLabel (pipe to python/jq)
idb ui tap --udid <udid> <x> <y> # tap at device-point coords (center of a frame)
xcrun simctl io <udid> screenshot <path.png> # capture; then Read the PNG and eyeball
- Get coords from
describe-all, not from screenshot pixels (screenshot is 3× the
point space). Tap the center of an element's frame.
- EYEBALL every screenshot (Read the PNG). The a11y tree tells you what nodes exist;
only the image shows clipping, overlap, empty space, tofu glyphs, z-order.
- After each tap,
describe-all again to confirm where you landed (a tap can miss / hit
springboard / over-navigate on double-back).
Gotchas (each cost a real failure here)
- zsh does NOT word-split unquoted
$var. idb ui tap --udid $S $xy with xy="201 488"
fails (invalid int value: '201 488'). Use ${=xy} (zsh split) or literal ints. Bare
literals work; variables need ${=...}.
idb ui describe-all / ui_tap need idb on PATH — without it the ios-simulator-MCP
errors spawn idb ENOENT. simctl can screenshot but cannot tap.- One booted sim = serialize. Don't run concurrent sim-driving subagents against the
same sim — they collide. Leader drives; analysis can fan out afterward.
- Layout stress:
xcrun simctl ui <udid> content_size accessibility-extra-extra-extra-large
then relaunch to test Dynamic Type; reset with content_size large. appearance dark|light
for color scheme. System alerts (e.g. StoreKit "Sign in to Apple Account") persist across
app relaunch — dismiss them before reading the app.
- Reaching win/completion: blind taps can't solve Sudoku or clear Minesweeper. Use the
DEBUG near-win launch hook (
-uitest-near-win) to land one move from a win. A loss is
reachable by tapping cells until a mine.
What to probe (this finds what snapshots miss)
- Negative / offline: iCloud signed out, no Game Center, airplane mode mid-game, Retry,
Restore-with-nothing, IAP cancel. (Core gameplay must never gate on iCloud — #512.)
- Online + iCloud-signed-out ≠ offline — test BOTH. For any CloudKit-dependent screen
(Daily Hub, ResumePill, leaderboard) these diverge: offline, CK calls fail-fast (throw
immediately); online-but-signed-out, CK calls can HANG (network round-trip stalls
with no authenticated container). A prior offline "pass" can mask an online hang — #536's
Daily Hub infinite spinner only reproduced online+signed-out (the AdMob banner's network
load triggered the re-render that exposed it). Drive the sim with network ON + iCloud OUT
and watch for a spinner that never resolves.
- Resume / save / Game Center cases need an iCloud-signed-in sim. Signed out,
latestInProgress() returns nil by graceful-degrade design (#515) so no ResumePill ever
shows — #4(pill)/#12/#19/#20/#38 + GC completion are un-testable until the user signs a
sandbox Apple ID into Simulator → Settings → Apple Account (iCloud and Game Center are
SEPARATE sign-ins — GC alone does not enable CloudKit). GKGameCenterViewController is a
system overlay idb can't introspect; absence of the #513 sign-in alert is the signal GC is
recognized.
- Navigation / modals: does the board actually appear after picking difficulty (#491);
Close → Leave-Game confirmation; pause; back-stack pressure.
- Safe area / Dynamic Island: completion/overlay content clipped or overlapping board
chrome on a notch/island device (#518) — snapshots can't see this.
- End-game: win + loss completion flow, leaderboard submit when GC unavailable.
File each finding with the screenshot evidence + repro; note when it's environmental
(no iCloud / stale build) vs a genuine current bug, and re-validate on a fresh main build
before calling it real.
