# vellum-client-qa
Cross-platform QA rig for `vellum-assistant` clients. Three skills that compose:
| Skill | What it does | Runs where |
|---|---|---|
| `qa-mock-backend` | Fake platform + daemon API so any client boots to the real chat UI with scripted state | Anywhere bun runs |
| `qa-web-ui` | Playwright + headless Chromium with mobile emulation, CDP touch gestures, video proof | Linux container or macOS |
| `qa-ios-simulator` | Real native iOS app (Capacitor/WKWebView/Swift) verified via XCUITest on a simulator | macOS with Xcode |
**Decision rule:** web-rendered behavior (CSS, layout, gestures, React) → `qa-web-ui`. Anything native (edit menus, plugins, permissions, push, WKWebView quirks, keyboard `visualViewport`) → `qa-ios-simulator`. Both boot against `qa-mock-backend`.
## Prerequisites
- `bun` everywhere; `playwright-core` + Chromium for web; Xcode + an iOS simulator runtime for iOS
- A checkout of `vellum-ai/vellum-assistant` (set `QA_REPO_DIR`, see skills)
- ffmpeg with libx264 for QuickTime-compatible video output
## Status
Extracted Jul 2026 from a working single-user rig (verified on real PR QA: gesture tests Jul 7, iOS Reply edit-menu Jul 9 / PR #37629). Self-dry-run passed Jul 9 in a clean environment (env vars only): qa-web-ui harness booted the app and passed positive+negative gesture assertions; qa-mock-backend standalone server passed all contract checks and booted the real app to a rendered transcript with zero route mocks. **Second-engineer dry-run passed Jul 9** (voice-mode feature test, all assertions green) — their feedback (feature-dependency discovery step, Playwright glob/transport limits, module-shadowing globs, pass-through defaults, macOS Chromium path, sandbox-vs-host networking) is incorporated as of v0.2.0. Keep reporting rough edges.
v0.2.1 folds in four lessons from a real worktree-diff QA run: (1) deps must match the branch under test — a stale/shared `node_modules` crashes the boot on missing packages or SDK members; (2) test the real booted app, not a synthetic component mount — runtime-injected styles (Tailwind v4 `adoptedStyleSheets`) leave ad-hoc mounts unstyled; (3) store seeding only works against the *same* store instance — a dynamic-import module-split silently no-ops the seed; (4) drive trigger/position-dependent UI with real gestures, not synthetic calls with hardcoded rects.
v0.2.2 folds in four more from a second engineer's real QA run: (1) for a multi-scenario proof video prefer one page + `reload()` over multiple pages — you get a single webm in guaranteed order, no concat (and `mtime` ties make `ls -t` a coin flip); (2) page-level media/pointer emulation (`Emulation.setEmulatedMedia`) does NOT survive `reload()` — re-apply after each navigation (context `isMobile`/`hasTouch` do persist); (3) don't assert animation state attributes (`[data-state="open"]`) for animated components — they settle late or live on a non-visible ancestor; assert on geometry or `elementFromPoint`; (4) probe the mock port before starting a server — one may already be running.
v0.2.4 folds in resource-hygiene lessons from a QA run that ballooned scratch to ~30GB / ~800k files and triggered file-descriptor pressure: (1) cleanup is failure-safe, not a trailing step — the web harness wraps its run in `try/finally` so `context.close()`→`browser.close()`→dev-server kill always fire even when an assertion throws, and both skills' teardown/complete-when checklists now require confirming no `vite|chrome|playwright|ffmpeg`/`mock-server|simctl` processes are left; (2) a resource budget up front — `df -h` before any clone/install, rethink at ~70% and hard-stop at ~80%, cap live browser/dev-server sessions at 2–3; (3) one clone + per-branch worktrees, never N fresh full clones (worktrees share git objects; the `bun install` cost is unavoidable either way), and remove any run-only clone/worktree when done; (4) delete this run's artifacts once delivered and never write tokens/secrets to scratch files.
v0.2.3 folds in three lessons from a native iOS safe-area QA run (mobile nav drawer): (1) synthetic XCUITest taps (coordinate *and* a11y) don't reliably fire React `onClick` on WKWebView content — drive web-rendered interactions from inside the web layer by injecting `element.click()` via the mock's `index.html` and timing capture off a beacon, reserving native taps for genuinely native controls; (2) `cap sync` / `xcodegen generate` regenerate the project on every re-sync, wiping `Info.plist` ATS (`NSAllowsArbitraryLoads`) and any hand-added UITests target — re-apply after each sync or bake into `project.yml`; (3) accept the notification permission prompt in-test via `addUIInterruptionMonitor` + a direct springboard **Allow** tap (so it's off-screen for captures) rather than relying on `simctl privacy grant notifications`, which can fail and re-fire. Plus a note: read the built `.app`'s actual `CFBundleIdentifier` rather than assuming it matches `capacitor.config.ts`.
v0.2.5 folds in one lesson from a native iOS safe-area color-band QA run: the notification permission prompt can also be cleared **outside** an XCUITest run — when driving the simulator manually or scripting `simctl` screenshots for a one-off visual check — via the **computer-use** skill (`computer_use_open_app "Simulator"` to bring the sim frontmost, `computer_use_observe`, then `computer_use_click` the alert's **Allow** by accessibility `element_id`). This unblocks scrim-free color/pixel captures when there's no test target. It also records that suppressing the prompt from the web bundle never works (the prompt is native, `@capacitor/push-notifications`), so accept/dismiss it at the native layer.
查看 README 源文件