visual-regression-testing

name

visual-regression-testing

description

Catches unintended UI pixel changes by snapshotting rendered output and diffing against approved baselines — make snapshots deterministic (disable CSS animations/transitions/caret, mask dynamic regions like dates/avatars/ads, freeze the clock and seed randomness, preload+wait for fonts, pin viewport + deviceScaleFactor, force reduced-motion and a fixed color-scheme), generate per-browser/per-OS baselines (never share a Linux baseline with a dev's macOS), tune the diff threshold (maxDiffPixelRatio / anti-alias mode) instead of inflating it to hide flake, run baselines in ONE pinned container so subpixel/font rendering is identical, and wire a human review/approve flow (Playwright --update-snapshots, Chromatic/Percy approve UI) — at component level (isolated, fast) and page level (integration). Effectively a pixel contract: a diff is a question for a human, not an auto-pass.

When to Use

Reach for this skill when the goal is detecting unintended pixel/visual changes against an approved baseline, not functional behavior or a11y conformance:

• "A CSS refactor / Tailwind upgrade / design-token change silently broke a layout somewhere"
• "Add visual regression / screenshot tests to this component library or these pages"
• "Set up Playwright toHaveScreenshot, Chromatic, Percy, or BackstopJS"
• "Snapshots flake — they pass on CI but fail on my Mac, or fail randomly"
• "Tune the diff threshold / mask the date+avatar regions / freeze animations"
• "Wire the baseline review-and-approve flow into PRs"

NOT this skill:

• Asserting a button click opens a modal, a form submits, navigation/DOM state, network mocking → write-playwright-e2e (functional E2E; this skill is the screenshot-diff layer that also runs on a stabilized page)
• WCAG conformance, contrast ratios, ARIA, keyboard/focus order, screen-reader semantics → audit-accessibility-wcag (correct semantics, not whether pixels match a baseline)
• A snapshot/screenshot test that's flaky for timing/ordering reasons → debug-flaky-tests (root-causing nondeterminism in general; this skill prescribes the visual-specific stabilizers)
• Structuring the test suite, fixtures, assertions for unit/integration tests → write-tests
• Driving a real browser to manually inspect/debug a rendering bug → debug-frontend-browser
• Catching LCP/CLS/perf regressions (layout shift as a metric, not a pixel diff) → optimize-core-web-vitals
• Defining the tokens (color/space/type scale) whose changes you're guarding → design-token-system

Steps

1. Pick the tier by what you own. Each is a screenshot + perceptual diff against a stored baseline; they differ in where baselines live and review happens.

   Tool	Baseline storage	Review/approve	Best for
   Playwright toHaveScreenshot	git (PNGs committed per project)	--update-snapshots + PR diff of .png	self-hosted, full control, free; you own the render env
   Chromatic	cloud (Storybook)	hosted UI, per-story approve, branch baselines	Storybook component libs; turbosnap diffs only changed stories
   Percy (BrowserStack)	cloud	hosted UI, approve per snapshot	cross-browser cloud render, framework-agnostic SDK
   BackstopLP / BackstopJS	git/local	approve CLI, HTML report	legacy/no-cloud, reference+test+report flow

   Default to Playwright toHaveScreenshot when you control the runner (commit baselines, run in a pinned container); reach for Chromatic/Percy when you can't pin a render env or want cross-browser cloud baselines without managing them.

2. Render env is the baseline — pin it or every diff is noise. Font hinting and subpixel antialiasing differ across OS/GPU, so a macOS-generated PNG will never match a Linux CI PNG. Generate and verify baselines in one environment:

   • Playwright: pin the Docker image to your exact version — mcr.microsoft.com/playwright:v1.50.0-noble — and run baseline generation and CI in the same image. Never commit a baseline produced on a dev's machine.
   • Snapshot filenames already encode browser/OS (button-chromium-linux.png). Keep that suffix; do not force a single platform name to "share" baselines across OSes — generate one baseline per (browser, platform) you actually test.
   • npx playwright test --update-snapshots locally only via docker run in that image, or with a dedicated CI "update baselines" job — so the bytes match CI.

3. A mid-transition frame is the #1 flake source.

Common Errors

• Baseline made on macOS, CI runs Linux. Font/subpixel rendering differs → every snapshot "fails." Fix: generate and run in one pinned container image (mcr.microsoft.com/playwright:vX.Y-noble); never commit a dev-machine baseline.
• Animations/transitions not disabled. Mid-flight frame captured → random diffs. Fix: animations:'disabled', caret:'hide', inject transition/animation:none!important, emulateMedia({reducedMotion:'reduce'}).
• Web font swaps after the shot (FOUT). Glyph metrics shift → text diffs. Fix: await document.fonts.ready + self-host/preload fonts.
• Live time/random/data. "2 min ago", uuids, live API → churns pixels. Fix: page.clock.setFixedTime, seed/stub Math.random/randomUUID, route APIs to fixtures.
• Raising maxDiffPixelRatio to stop flake. Hides real regressions across the whole frame. Fix: eliminate nondeterminism (steps 3–6) and mask dynamic regions; keep the threshold near zero.
• waitForTimeout/networkidle instead of a render signal. Flaky on slow CI, deprecated. Fix: wait on fonts.ready, specific image decode(), or rely on toHaveScreenshot's built-in retry-until-stable.
• Forcing one platform name to share baselines. A "shared" baseline matches no real env. Fix: one baseline per (browser, platform); keep the OS suffix in the filename.
• Auto-running --update-snapshots in CI. Silently re-baselines regressions → the test never fails on a real change. Fix: dedicated, reviewed update job; commit PNGs in the PR.
• Only the default/happy state snapshotted. Hover/error/empty/dark/RTL regressions slip through. Fix: a baseline per meaningful state.
• No DPR pin. HiDPI runner doubles pixels vs 1x → size mismatch. Fix: deviceScaleFactor:1 + scale:'css'.
• Giant full-page snapshots only. One header change fails 40 pages; slow, untriageable. Fix: component-level pyramid + a few critical page shots.
• Baselines committed as raw blobs. Binary churn bloats the repo. Fix: Git LFS; prune orphaned PNGs.

Verify

1. Determinism re-run: run the suite twice back-to-back with no code change in the pinned CI image → zero diffs. Any nonzero diff on a clean re-run is leftover nondeterminism — fix it before trusting the suite.
2. Env parity: generate a baseline in the container and run CI in the same container → match; confirm filenames carry the (browser, platform) suffix and no baseline was produced on a dev machine.
3. Real regression is caught: deliberately change a color/padding/font-size by a few px → the relevant snapshot fails and the report shows a highlighted diff.png; the build goes red.
4. Masking works, threshold is tight: a masked region (avatar/clock) churning its content produces no diff, while an unmasked 1% layout shift does fail — proving the threshold isn't swallowing real changes.
5. Stabilizers active: animations disabled, document.fonts.ready awaited, clock fixed, randomness seeded, APIs stubbed to fixtures — grep the config/setup for each; a snapshot taken mid-animation or with a live Date.now() would fail check 1.
6. Approve flow is manual: confirm no job runs --update-snapshots/auto-approve on the main path; an intentional change requires committing new PNGs (or clicking approve) in a reviewed PR, and that PR's diff shows the pixel change.
7. State coverage: the critical components have baselines for hover/focus/error/empty/dark/RTL, not just default; responsive breakpoints are separate named snapshots.

Done = snapshots are byte-stable on a clean re-run in one pinned render env, dynamic regions are masked (not threshold-inflated), per-(browser,platform) baselines live in version control via LFS, a real few-pixel change goes red with a visible diff, and every baseline update is a deliberate, reviewed human approval — never an automatic CI step.