ios-simulator-microscope

# Purpose Reproduce and measure layout, rendering, and interaction bugs that only manifest in real iOS Safari — the ones desktop Playwright (`browser-microscope`), headless WebKit, and jsdom are structurally blind to because they don't model the iOS toolbar, dynamic viewport, or touch input. It boots a Simulator, drives Safari through safaridriver, injects JS to read exact geometry and viewport-unit resolution, screenshots the device, and scrolls via macOS CGEvent to trigger toolbar-collapse dynamics — turning "looks wrong on my iPad" into measured numbers. ## Prerequisites All environmental preconditions are checked by `./scripts/preflight.sh` (workflow step 1) — macOS, Xcode + a Simulator runtime, safaridriver, Node.js >= 18, a Swift toolchain, and Accessibility permission for the terminal. The script only checks and instructs; the human-only setup actions it may ask for are: - `safaridriver --enable` (one-time, prompts for auth). - Granting the terminal **Accessibility** permission (System Settings → Privacy & Security → Accessibility; applies to new processes without a restart). - Installing Xcode / an iOS Simulator runtime if absent. ## Variables DRIVER_PORT: 4444 # Port for `safaridriver -p`; scripts read SIM_DRIVER=http://localhost:4444 BRIDGE_PORT: 8899 # Port for `sim-bridge.mjs serve` (Mode B); scripts read SIM_BRIDGE_PORT DEVICE_MATCH: iPad Air 13 # Substring to pick the device from `simctl list` (match the bug report's device class) ## Pick a mode first **safaridriver and CGEvent gestures are mutually exclusive.** A synthetic drag is indistinguishable from a finger, so during a WebDriver session it raises the native "Safari is Running an Automated Test" alert, which swallows that gesture and every one after it. Choose before booting: - **Mode A — safaridriver** (`sim-session.mjs`): static measurement of a page as loaded. JS injection, geometry, viewport units, page screenshots. **No gestures.** - **Mode B — bridge** (`sim-bridge.mjs`): anything involving a real gesture — scrolling, toolbar collapse, momentum, rubber-band, touch focus. The bridge reverse-proxies the target and injects a measurement agent, so no WebDriver session exists and the lock never appears. Read `./references/automation-lock.md`. If the task says "scroll", "collapse the toolbar", or "after scrolling" → Mode B. ## Workflow 1. **Confirm the environment** - Run the preflight doctor, then pick the target device. - Tool: `bash ./scripts/preflight.sh` — one PASS/FAIL line per check. - IF: `RESULT: FAIL` → relay each `fix:` remediation to the user verbatim and STOP; those checks (Xcode install, `safaridriver --enable`) are human-only actions. On non-macOS the script exits immediately: this skill is macOS-only. - IF: `RESULT: DEGRADED` (only `swift`/`accessibility` failed) → relay the fix lines, then CONTINUE: those gate only the CGEvent helpers (`sim-scroll.swift`, `sim-tap.swift`). Static measurement, the bridge, and programmatic scrolling (`sim-bridge.mjs eval "window.scrollBy(0,600)"`) all work without them — enough to trigger lazy-loading, scroll-position logic, and IntersectionObservers. Only true toolbar-collapse / momentum / rubber-band dynamics require the real gesture. - IF: all PASS → `xcrun simctl list devices available | grep -i "<DEVICE_MATCH>"` to get a UDID. - Example: report says "iPad Air 13-inch M2" → pick an `iPad Air 13-inch` UDID on the closest iOS runtime; if the bug is version-sensitive, note two UDIDs (e.g. one on iOS 18.x, one on 26.x). 2. **Boot the Simulator and driver** - Boot the device, open the Simulator GUI (needed — CGEvent scroll drives the visible window), start safaridriver. - Tool: `xcrun simctl boot <udid>`; `open -a Simulator`; `safaridriver -p <DRIVER_PORT> &` then `curl -s localhost:<DRIVER_PORT>/status`. - IF: Mode B → skip safaridriver entirely; an active WebDriver *session* is what raises the automation lock, and the bridge needs none. - Example: `{"value":{"ready":true}}` → driver is up. If a prior run left a session, `xcrun simctl terminate <udid> com.apple.mobilesafari` clears a stuck Safari pairing. 3. **Open the page and read the baseline** - Navigate and capture the untouched viewport state. - Tool (Mode A): `node ./scripts/sim-session.mjs open <udid> "<url>"` then `node ./scripts/sim-session.mjs viewport`. - Tool (Mode B): `node ./scripts/sim-bridge.mjs serve <targetOrigin> --port <BRIDGE_PORT> &`; `xcrun simctl openurl booted "http://localhost:<BRIDGE_PORT>/<path>"`; `node ./scripts/sim-bridge.mjs probe` for the baseline. Re-testing after a code change or a previous bridge run? Add `--fresh` — it mints a random port, i.e. a brand-new localhost origin Safari has never cached (see Gotchas). - IF: instrumentation must run before the page's own scripts (message taps, early hooks) → `serve … --init-js hooks.js` inlines that file right after the agent, ahead of all page code. - Example: local Storybook story `http://localhost:6006/iframe.html?...&id=...` → JSON showing `visualViewport.height`, `units.dvh/svh/lvh`, `units.pct_of_ICB`. For basic-auth staging, embed creds in the URL (`https://:pass@host/…`); a corporate ZTNA / self-signed cert shows a native "Not Private" interstitial that **cannot be automated** (see Gotchas). 4. **Reproduce the state, then measure** - Drive the interaction that triggers the bug, then screenshot + read geometry. Scrolling to collapse the toolbar is the usual trigger — and it requires **Mode B**. - Tool (Mode B): `swift ./scripts/sim-scroll.swift 0.8 0.25 60` to scroll down; then `node ./scripts/sim-bridge.mjs probe` (latest frame) / `eval "<js>"` (element geometry) / `console 50` (page console + JS errors) / `log 40` (frame history across the gesture). - **`eval` contract:** a bare expression returns its value (`eval "window.scrollY"` — auto-wrapped in `return (...)`); a multi-statement script runs as a function body and needs an explicit `return`, else the answer is `result: null` with a hint. Every answer carries `answeredBy` (the URL of the page that ran it) — **check it**: with more than one tab connected, evals go to whichever client polls first. `clients` lists connected pages; `eval "<js>" --match <url-substring>` pins the eval to one of them. - Screenshot with `xcrun simctl io booted screenshot out.png` then `Read out.png` — it captures **Safari's chrome**, so you can see whether the toolbar actually collapsed. `sim-session.mjs shot` is page-only and will hide both the toolbar state and any native modal. - IF: a drag prints `SCROLL_DONE` but `scrollY` stays 0 and the page logs no `touchstart` → the automation lock is up, not a focus/permission fault. Read `./references/automation-lock.md`. - IF: you need to observe a transient mid-interaction state (open a dialog exactly while the toolbar is collapsed) → inject the diagnostic decorator so screenshots carry the numbers, and auto-trigger on viewport GROWTH. Read `./references/diagnostic-decorator.md`. In Mode B the bridge's HUD and `log` already do this. - IF: you need to tap web content → dispatch a JS click (`.click()` via `sim-session.mjs js` or `sim-bridge.mjs eval`), NOT `sim-tap.swift` (see Gotchas). - Example: after scroll, `getBoundingClientRect()` on the offending element vs `visualViewport.height` → the gap in pixels, matching or refuting the report. 5. **Interpret against the hardware-only test** - Decide whether the Simulator can even show this bug before chasing it. - IF: `units.pct_of_ICB === units.dvh` after the toolbar collapses → the Simulator resolves `height:100%` to the dynamic viewport and **cannot reproduce stale-ICB gaps**; the bug is real-hardware-only. Stop, report that, and — if a fix changes `100%`→`100dvh` — note the sim can't distinguish fix from current (see Gotchas). - Example: measured `pct_of_ICB 1323 === dvh 1323` on collapse, no gap → declare hardware-only, hand the physical-device confirmation back to the user rather than grinding. 6. **Tear down** - Close cleanly so the next run isn't blocked by a stuck pairing. - Tool: `node ./scripts/sim-session.mjs close`; `pkill -f "safaridriver -p <DRIVER_PORT>"`; `pkill -f "sim-bridge.mjs serve"` if Mode B ran; optionally `xcrun simctl shutdown <udid>`. - Example: session closed → re-running step 2 later starts fresh. If `open` errors with "already paired", `xcrun simctl terminate <udid> com.apple.mobilesafari` and retry. ## References ### The automation lock, and gesture-capable Mode B - IF: any gesture is involved (scroll, toolbar collapse, momentum, touch focus), or a drag reports `SCROLL_DONE` while the page doesn't move - THEN: Read and apply `./references/automation-lock.md` - EXAMPLES: - "scroll until the toolbar collapses, then measure the sheet" - "the drag says it worked but scrollY is still 0" - "a modal I can't see is blocking the Simulator" ### Painting measurements into device screenshots - IF: you must observe a transient or animated state, or Web Inspector numbers won't land in a screenshot - THEN: Read and apply `./references/diagnostic-decorator.md` - EXAMPLES: - "capture the viewport numbers at the exact frame the sheet opens" - "the gap only exists mid-scroll, screenshot won't tell me the height" - "auto-open the dialog when the toolbar collapses" ## Gotchas Environment facts learned the hard way — they defy reasonable assumptions: - **A CGEvent gesture during a WebDriver session is silently eaten by the automation lock** — see *Pick a mode first* and `./references/automation-lock.md` for detail + recovery. - **safaridriver interaction is broken on the Simulator.** WebDriver touch-pointer actions **hang** (request never returns); wheel actions return **501 not implemented**; discrete synthetic pointer clicks do **not** reliably convert to device touches. What works: `sim-scroll.swift` (CGEvent drag, Mode B only) for scrolling, and JS `.click()` dispatch for tapping web content. - **`sim-tap.swift` takes DEVICE POINTS, not fractions.** `0..1024 × 0..1366` on an iPad Air 13". Passing `0.5 0.635` taps the window's top-left corner and looks like "native UI ignored my click". Read real coordinates off a `simctl io booted screenshot` (screenshot px ÷ 2 at 2x), and sanity-check the window mapping with `swift ./scripts/sim-window.swift`. - **Native Safari UI is *partly* automatable — the distinction matters.** The automation-lock alert **does** respond to synthetic clicks (a mis-aimed tap on it will hit "Stop Test Session" and kill your session). Cert interstitials ("This Connection Is Not Private") genuinely do not. So a corporate **ZTNA / self-signed staging cert** still walls the sim even with correct basic-auth creds — you need a human tap or an enrolled device. Don't burn time scripting past *that* one. - **safaridriver screenshots are page-only.** They show a normal-looking page while a native modal covers the screen, and never show the toolbar — so they cannot tell you whether it collapsed. Use `xcrun simctl io booted screenshot` whenever chrome or native UI matters. - **The Simulator can't reproduce stale-ICB / dynamic-viewport bugs.** Its WebKit resolves `height:100%` to the *dynamic* viewport, so `pct_of_ICB === dvh` after the toolbar collapses and there is no gap. Real iPad hardware keeps `100%`/ICB pinned to the *small* viewport, which is the actual bug. Consequence: the sim can neither reproduce these nor distinguish a `100%`→`100dvh` fix from the status quo. Verify the fix is a no-op in the sim (zero regression) and hand real-device confirmation to the user. This is a *decision-saver* — check step 5 early. - **CGEvent needs a focused Simulator and Accessibility permission.** `check-accessibility.swift` must print `true`; the scroll/tap helpers click the Simulator title bar to focus it first (`NSWorkspace.activate` is unreliable). If drags don't scroll, the window isn't frontmost or permission is missing. - **Small-viewport height is version-specific.** iPad Air 13" reads ~1280 (iOS 26) vs ~1292 (iOS 18.4) with the toolbar shown. Auto-trigger logic must key off viewport **growth from a captured baseline**, never an absolute pixel threshold. - **`open` fails with "already paired" after a crash.** Clear it with `xcrun simctl terminate <udid> com.apple.mobilesafari`, then retry — don't recreate the driver. - **"Could not find any session hosts" hides two distinct causes.** (1) Remote automation not enabled — the simulator path reports it with this generic message, but a *host*-Safari probe names it: `curl -s -X POST localhost:<DRIVER_PORT>/session -H 'Content-Type: application/json' -d '{"capabilities":{"alwaysMatch":{"browserName":"Safari"}}}'`. An explicit "must enable 'Allow remote automation'" error → human runs `safaridriver --enable` (needs auth) or Safari → Settings → Developer → Allow remote automation. If the probe *succeeds*, DELETE its session, and the cause is (2): a runtime/Safari version gap — safaridriver won't pair with a much older simulator runtime (observed: safaridriver 26.x vs iOS 17.0 runtime). Fix: install a current iOS Simulator runtime (update Xcode, or `xcodebuild -downloadPlatform iOS` — multi-GB, ask the user first). - **`xcode-select` pointing at CommandLineTools breaks simctl AND safaridriver — and only sudo truly fixes it.** If `xcode-select -p` prints `/Library/Developer/CommandLineTools` while `/Applications/Xcode.app` exists: `DEVELOPER_DIR=/Applications/Xcode.app` rescues `xcrun simctl`, but safaridriver **ignores it** when locating Simulator.app — session creation fails with "Could not find Simulator.app or a devices:// URL handler" (and `lsregister`/relaunching Simulator does not help). The only fix is the human running `sudo xcode-select -s /Applications/Xcode.app`, then restarting safaridriver. - **Sim Safari's cache survives everything short of a new origin.** It serves stale HTML *and* stale JS chunks for a proxied localhost origin even across `simctl terminate` of Safari and cache-busting query strings — on hydrating apps this manifests as fresh server HTML silently "patched back" to old markup by stale-cached client JS (hydration mismatch), which reads exactly like your change didn't deploy. A new port is a new origin with an empty cache: re-serve with `--fresh` (or hand-pick an unused port) after every build you intend to verify. - **Evals go to whichever connected page polls first.** Background tabs are suspended by iOS but wake unpredictably and can answer an eval meant for the foreground page — measurements then interleave across tabs and fabricate phantom effects (scroll positions jumping, wrong DOM). Always check `answeredBy` in the response; run `clients` when anything looks impossible; use `--match <url-substring>` to pin evals to one page; `simctl terminate booted com.apple.mobilesafari` + reopen to guarantee a single tab. - **Reverse-proxying a child iframe's SPA origin usually kills the app.** Serving a second bridge for the iframe's origin injects the agent fine, but SPAs break under a swapped origin (base-href, absolute API URLs, service-worker scope) — the child may blank out, reload loops, or self-remove its own iframe. Parent-side message taps still work through it; just don't expect the child app to *function*. For looking *inside* a cross-origin iframe on desktop engines, plain Playwright reaches into OOPIFs natively (`page.frames()`) — exhaust that first, and burn a Simulator run only on the iOS-specific part. ## Works well with Optional collaborators — this skill runs standalone and these degrade gracefully if absent. - **`browser-microscope`** — the desktop-Playwright counterpart. Reproduce and measure there first (faster, no Simulator boot, reaches cross-origin iframes natively); escalate here only when the bug is iOS-Safari-only or involves toolbar/dynamic-viewport/touch dynamics Playwright can't model. - **`diagnose`** — this skill is the reproduce-and-instrument step for iOS-Safari-only bug reports inside a disciplined diagnosis loop; measurements here become the evidence for its hypothesise/fix/regression-test phases.