pwa-device-testing

name

pwa-device-testing

description

This skill should be used when the user asks to "test on device", "test on emulator", "run emulator", "launch AVD", "test PWA", "test on Android", "test on mobile", "verify on real device", "check on phone", or discusses testing a feature on an actual device or emulator rather than headless Playwright. Also use when validating features that headless browsers cannot cover (biometric, PWA install, Chrome autofill, touch gestures, password managers). Use proactively when a feature has been implemented that touches any of these capabilities.

PWA Device Testing

Headless Playwright tests cover logic and layout but cannot validate:

• Chrome password manager / autofill behavior
• WebAuthn biometric (fingerprint, face)
• PWA install-to-homescreen and standalone mode
• Real touch gestures, virtual keyboard, IME
• Service worker update UX on actual Chrome
• CSS safe-area-inset rendering on notched devices

This skill provides the correct setup, known pitfalls, and ready-to-use templates for testing on real Chrome via Android emulator.

Quick Start

# First time only:
scripts/setup-avd.sh

# Every session (Appium tests -- primary):
scripts/run-appium-tests.sh                     # full suite (31 tests)
scripts/run-appium-tests.sh --suite my-label    # tagged archive directory

# Legacy (CDP emulator tests):
npm run test:emulator

IMPORTANT: ALWAYS run Appium/emulator tests via scripts/run-appium-tests.sh, never bare npx playwright test --config=playwright.appium.config.js. The script handles screen recording, ANR dialog dismissal, archival to test-history/, and ffprobe validation. Without it, test runs produce no video evidence for review.

Fast-gate tests (tsc, eslint, headless npx playwright test) are fine to run directly -- they don't touch the emulator and don't need recording.

npm run test:emulator (via scripts/run-emulator-tests.sh) handles the legacy CDP path: boots emulator if needed, enables Chrome debugging, sets up port forwarding, runs Playwright over CDP.

Architecture

Host machine                          Android Emulator (Pixel 7, API 35)
+-----------------------+             +---------------------------+
| MobiSSH server :8081  |<--adb rev-->| Chrome tab: localhost:8081|
| Playwright test runner|--CDP:9222-->| Chrome DevTools socket    |
+-----------------------+             +---------------------------+

• CDP connection: Playwright connectOverCDP() to real Chrome via ADB-forwarded DevTools port
• Port forwarding: adb reverse tcp:8081 tcp:8081 so emulator's localhost reaches host
• Single worker CDP: One CDP connection per test file, fresh tab per test

Interaction Design Principles (learned the hard way)

Emulator tests must be faithful proxies for human interaction

Every dialog, prompt, and overlay that appears during a test flow must be handled the way a real user would handle it: see it, understand what it's asking, and dismiss it appropriately. This includes both app-owned dialogs (host key accept, vault setup) and Chrome-native UI (save password bar, add username suggestion, notification prompt).

If a test can't click a button, a real user can't either. The test is surfacing a real UX bug.

Never use force: true to work around click interception

When Playwright reports <div class="vault-dialog">...</div> intercepts pointer events, the correct response is to fix the CSS/layout so the button is actually clickable, NOT to bypass Playwright's actionability checks with { force: true }. Using force: true:

• Hides real interaction bugs from the test suite
• Creates a test that passes while the actual user flow is broken
• Masks layout overflow on mobile viewports

Common causes of click interception on mobile:

• align-items: center on position: fixed; inset: 0 overlays -- when the dialog content is taller than the viewport, content overflows and sibling elements intercept clicks. Fix: align-items: flex-start + overflow-y: auto on the overlay, margin: auto 0 on the dialog for vertical centering that still allows scrolling.
• Chrome-native UI (password save bar, username suggestion) appearing as a layer between your app dialog and the click target. Fix: suppress Chrome autofill on test fields with autocomplete="off", pre-grant notifications, use --disable-fre flag.
• Keyboard pushing elements up so labels overlap buttons. Fix: ensure sufficient spacing, or scroll the button into view first.

Feature removal is a valid outcome

The selection overlay feature (#55) went through 6+ commits, was feature-flagged off, still caused interference with core scroll behavior (#143), and was ultimately removed entirely (600 lines deleted in 0ac4010). This validates the "know when to quit" principle: if every fix introduces a new bug, the abstraction is wrong. Strip it, ship the working core, and re-approach later with a cleaner design.

Always verify server version before asking user to test

scripts/verify-test-ready.sh [--url https://mobissh.tailbe5094.ts.net]

Checks server currency, HTTP endpoint hash match, and prints the ?reset=1 URL for SW cache busting. Exit 1 = not ready.

Critical Pitfalls (learned the hard way)

Chrome DevTools socket requires set-debug-app

Handled automatically by scripts/run-emulator-tests.sh and scripts/run-appium-tests.sh. If running manually: adb shell am set-debug-app --persistent com.android.chrome then force-stop and relaunch Chrome.

No browser.newContext() on Android Chrome

Android Chrome's CDP exposes a single default browser context. Calling browser.newContext() throws: Protocol error (Target.createBrowserContext): Failed to create browser context.

Correct pattern:

const context = browser.contexts()[0]; // use the default
const page = await context.newPage();  // new tab within it

Shared localStorage across tests

Since all tabs share the single default context, localStorage is shared. Every test fixture MUST clear localStorage AND reload before the test runs. The app reads localStorage on init (panel state, vault, profiles), so clearing alone doesn't help if the app already initialized with stale state:

await page.goto(BASE_URL, { waitUntil: 'domcontentloaded' });
await page.evaluate(() => localStorage.clear());
await page.reload({ waitUntil: 'domcontentloaded' }); // app re-inits with clean state

Vault snapshot fixture (skip per-test vault setup)

Creating the vault from scratch per test wastes ~5s (keyboard dismiss, modal wait, form fill). The vaultSnapshot worker-scoped fixture creates the vault once per test file and snapshots the localStorage keys. The emulatorPage fixture restores the snapshot and auto-unlocks:

// Worker-scoped: creates vault once, snapshots localStorage keys
vaultSnapshot: [async ({ cdpBrowser }, use) => {
  // ... create vault with password 'test', snapshot localStorage ...
  await use(snapshot); // { vaultMeta, vaultData, ... }
}, { scope: 'worker' }],

// Per-test: restores snapshot, unlocks vault via addInitScript hook
emulatorPage: async ({ cdpBrowser, vaultSnapshot }, use) => {
  // ... restore snapshot to localStorage ...
  await page.addInitScript(() => {
    // Hook into __appReady to unlock vault before promptVaultSetupOnStartup
    Object.defineProperty(window, '__appReady', { /* ... fill pw + click unlock */ });
  });
  await page.reload({ waitUntil: 'domcontentloaded' });
  await page.waitForFunction(() => window.__vaultUnlocked === true);
  // ...
},

Key insight: The vault unlock bar doesn't appear on startup — it appears later when ensureVaultKeyWithUI() is called during profile operations. Checking for the bar after reload will find needsUnlock=false (race condition). The addInitScript + __appReady hook approach intercepts the app boot to unlock the vault before any UI flow triggers it.

Use emulatorPage for tests that need a pre-configured vault. Use cleanPage for tests that exercise vault setup from scratch.

Playwright outputDir isolation (CRITICAL)

Playwright clears its outputDir on each run. The default is test-results/, which wipes emulator recordings, report.json, frames, and any other non-Playwright artifacts.

Every Playwright config MUST set a dedicated outputDir:

• playwright.config.js → test-results/headless
• playwright.emulator.config.js → test-results/playwright-emulator
• playwright.appium.config.js → test-results-appium
• playwright.browserstack.config.js → test-results/browserstack

Without this, run-emulator-tests.sh writes recording.mp4 and report.json to test-results/emulator/, then the next Playwright run wipes them.

workers: 1 is mandatory for CDP

Parallel Playwright workers each try to interact with the same single Chrome instance over CDP. This causes "Target page, context or browser has been closed" across all tests. Always set workers: 1 in the emulator config:

module.exports = defineConfig({
  workers: 1, // single Chrome instance via CDP
  // ...
});

Inject page state AFTER navigation, not before

Any page.evaluate() state injection (WS spies, test globals) done before page.goto() gets destroyed by the navigation. Always inject on the live, already-loaded page:

// WRONG: spy gets destroyed by goto()
await page.evaluate(() => { window.__spy = []; });
await page.goto(BASE_URL);

// RIGHT: inject after the page is loaded
await page.goto(BASE_URL, { waitUntil: 'domcontentloaded' });
await page.evaluate(() => { window.__spy = []; });

Use actionTimeout for fast selector failure

Without actionTimeout, a bad selector (e.g. #connectBtn that doesn't exist) waits the full test timeout (60s), then cleanup closes the page, producing a misleading "Target page closed" error. Set a short action timeout so bad selectors fail fast with the actual error:

use: {
  actionTimeout: 10_000, // fail fast on bad selectors
}

Elements may not have IDs

Don't assume HTML elements have IDs. Use semantic/structural selectors:

// BAD: #connectBtn doesn't exist
await page.locator('#connectBtn').click();

// GOOD: target by form context + type
await page.locator('#connectForm button[type="submit"]').click();

Page.screencastFrame CDP doesn't work on Android emulator

The Page.startScreencast / Page.screencastFrame CDP API returns 0 frames on Android emulator Chrome. It works on desktop Chrome but the emulator's GPU pipeline doesn't produce screencast frames. Don't rely on frame count assertions. Screenshots via page.screenshot() work fine as an alternative.

Worker-scoped CDP connection is mandatory

Creating a new connectOverCDP() per test destabilises the DevTools socket. After ~4-5 connect/disconnect cycles, the connection drops with "Target page, context or browser has been closed." Use a worker-scoped fixture:

cdpBrowser: [async ({}, use) => {
  const browser = await chromium.connectOverCDP(`http://127.0.0.1:${CDP_PORT}`);
  await use(browser);
  browser.close();
}, { scope: 'worker' }],

Chrome nag modals block test visibility on first launch

Handled by scripts/run-appium-tests.sh (pre-grants notifications, sets Chrome flags) and the Appium fixture's modal-dismiss step. If writing a new test runner, replicate the three-layer defense:

1. adb shell pm grant com.android.chrome android.permission.POST_NOTIFICATIONS
2. Chrome --disable-fre --no-first-run --no-default-browser-check flags
3. Fixture-level modal dismiss with 2s timeout fallback

KVM group membership requires session reload

After sudo usermod -aG kvm $USER, the current shell doesn't pick up the new group. Use sg kvm -c 'emulator ...' or start a new login session.

AVD config uses = (with spaces)

The config.ini generated by avdmanager uses key = value (space-equals-space), not key=value. Sed patterns without spaces silently fail and the fallback echo creates duplicate keys. The setup script uses a set_avd_prop helper that handles both formats.

WebAuthn biometric toggle in headless test browsers

prfAvailable() returns true in Playwright's headless Chromium (PublicKeyCredential API exists) but navigator.credentials.create() hangs forever (no authenticator). In headless tests, uncheck the biometric toggle via page.evaluate:

await page.evaluate(() => {
  const cb = document.getElementById('vaultEnableBio');
  if (cb) cb.checked = false;
});

The CSS toggle hides the checkbox with opacity:0; width:0; height:0, so Playwright's isVisible() returns false and uncheck() silently skips it. Always use page.evaluate for CSS-hidden form elements.

Real SSH Integration via Docker

For features that need a live SSH connection (gestures, terminal buffer, command execution), use the Docker test-sshd container instead of WebSocket mocks:

docker compose -f docker-compose.test.yml up -d test-sshd   # simple test container, no lifecycle script
ssh -p 2222 testuser@localhost  # password: testpass

The sshd-fixture.js helper starts the container automatically and exposes credentials to tests. The setupRealSSHConnection(page, sshServer) helper in fixtures.js handles: SSRF bypass for localhost, WS URL rewriting (localhost→10.0.2.2), connect form fill, host key acceptance, and waiting for connected state. Vault setup is handled by the vaultSnapshot + emulatorPage fixture (no per-test vault creation).

Touch Gesture Testing

Gesture helpers use CDP Input.dispatchTouchEvent which goes through Chrome's real input pipeline and fires DOM touch events faithfully. An in-page touch visualizer draws green dots/trails at finger positions so gestures are visible in screen recordings (Android's pointer_location overlay doesn't register CDP touches).

Helpers in tests/emulator/fixtures.js:

• swipe(page, selector, startX, startY, endX, endY, steps) -- single-finger swipe via CDP
• pinch(page, selector, startDist, endDist, steps) -- two-finger pinch via CDP
• sendCommand(page, cmd) -- type into IME input char-by-char

The touch visualizer is injected automatically by swipe() and pinch(). Green dots show current finger positions, small trail dots persist for 2s to show the gesture path.

Verify gesture effects through app state, not visual diffs:

// Scroll: check xterm buffer position
const vp = await page.evaluate(() => window.__testTerminal.buffer.active.viewportY);

// Swipe: check WS spy for tmux commands
const msgs = await page.evaluate(() => window.__mockWsSpy.filter(...));

// Pinch: check terminal font size
const font = await page.evaluate(() => window.__testTerminal.options.fontSize);

Exploratory Interaction Testing (script-first, then assert)

Before writing test assertions, use the emulator to explore what actually happens when a user performs a gesture. This inverts the typical TDD approach: instead of starting with expected behavior and checking if it matches, start by simulating the physical interaction, capturing what the device shows, and reviewing the visual evidence to discover the domain of possibilities.

The workflow

1. Script the interaction textually. Before writing any test code, describe the interaction as a sequence of physical actions:

- Navigate to Settings panel
- Place two fingers on screen, 200px apart
- Move fingers apart to 400px (pinch out / zoom in)
- Observe: does the bottom bar stay anchored? Does content scale?
- Release fingers
- Observe: does layout return to normal?

2. Translate to emulator script. Use the gesture helpers to replay the interaction on the emulator, with screen recording running but NO assertions:

test('explore: pinch zoom on settings panel', async ({ emulatorPage: page }) => {
  await page.locator('[data-panel="settings"]').click();
  await page.waitForSelector('#panel-settings.active');
  await screenshot(page, testInfo, '01-before-pinch');

  // Simulate pinch-out (zoom in)
  await pinch(page, '#panel-settings', 100, 300);
  await page.waitForTimeout(500);
  await screenshot(page, testInfo, '02-during-zoom');

  // Release and observe
  await page.waitForTimeout(1000);
  await screenshot(page, testInfo, '03-after-zoom');
});