Use when designing or fixing a Playwright end-to-end test suite, debugging flaky tests, choosing locator strategies (getByRole vs CSS vs test-id), structuring fixtures and auth-state reuse, configuring parallelism and sharding, mocking third-party APIs via route(), or wiring trace-on-first-retry into CI. Triggers: "tests are flaky", page.waitForTimeout, page.locator with brittle CSS, login runs in every test, third-party API takes test offline, "should I use sleep here", parallel mode, sharding across CI machines, soft vs hard assertions, trace.zip not available on CI failure. NOT for unit testing (Vitest/Jest), Cypress migration playbooks, mobile native testing (Detox/XCUITest), or visual regression testing as a primary concern.
Use when designing or fixing a Playwright end-to-end test suite, debugging flaky tests, choosing locator strategies (getByRole vs CSS vs test-id), structuring fixtures and auth-state reuse, configuring parallelism and sharding, mocking third-party APIs via route(), or wiring trace-on-first-retry into CI. Triggers: "tests are flaky", page.waitForTimeout, page.locator with brittle CSS, login runs in every test, third-party API takes test offline, "should I use sleep here", parallel mode, sharding across CI machines, soft vs hard assertions, trace.zip not available on CI failure. NOT for unit testing (Vitest/Jest), Cypress migration playbooks, mobile native testing (Detox/XCUITest), or visual regression testing as a primary concern.
allowed-tools
Read,Grep,Glob,Edit,Write,Bash
metadata
{"category":"Code Quality & Testing","tags":["playwright","e2e","testing","automation","flaky-tests","browser-testing"],"provenance":{"kind":"first-party","owners":["port-daddy"]},"pairs-with":[{"skill":"ideal-web-app-builder","reason":"The web app that skill scaffolds is what this suite exercises; testable markup (roles, labels, test-ids) is designed there"},{"skill":"ux-friction-analyzer","reason":"Both drive real user journeys in the browser; friction findings become the high-value flows the E2E suite must cover"}],"io-contract":{"kind":"deliverable","consumes":["[Truncated]","[Truncated]"],"produces":["[Truncated]","[Truncated]"]}}
Playwright E2E Design
Most "Playwright is flaky" stories come from three things: brittle selectors, manual sleeps instead of web-first assertions, and tests that share state. The official Playwright best-practices doc, browserstack/testdino field guides, and the auto-waiting docs all converge on the same playbook. (Playwright — Best Practices, Playwright — Auto-waiting)
The compressed rule:
getByRole / getByTestId + web-first expect() + one fresh storageState per test = not flaky
Last resort. Comment why a higher tier didn't fit.
// ✓ Resilientawait page.getByRole('button', { name: 'Save' }).click();
await page.getByLabel('Display name').fill('Alice');
// ✗ Fragile — designer changes a class name and everything breaksawait page.locator('button.btn.btn-primary.save-action').click();
The Playwright doc's exact framing: "Designer changes to CSS classes break brittle selectors. User-facing attributes remain stable across refactoring." (playwright-best-practices)
Web-first assertions
expect() on a Playwright locator auto-retries until the condition is met or the timeout expires. This is the single most important flake-elimination tool. (Playwright — Assertions)
// ✗ Read-once snapshot. Race condition guaranteed.expect(await page.getByText('Welcome').isVisible()).toBe(true);
// ✓ Retries until visible or timeout. No race.awaitexpect(page.getByText('Welcome')).toBeVisible();
// ✓ Same for text content, URL, count, value, …awaitexpect(page.getByRole('row')).toHaveCount(10);
awaitexpect(page).toHaveURL(/\/dashboard$/);
awaitexpect(page.getByLabel('Email')).toHaveValue('alice@example.com');
Anything that synchronously reads from the DOM (isVisible(), textContent(), count()) is a snapshot. It does not retry. The await expect(locator).toX() form is the retrying form.
Replace sleeps with auto-waiting
Playwright's actionability checks (visible, stable, enabled, receives events) run automatically before every action. (Playwright — Actionability) page.waitForTimeout(N) is almost always wrong:
// ✗ Hopes the API resolved within 2s. Sometimes it didn't.await page.click('text=Save');
await page.waitForTimeout(2000);
expect(await page.locator('.toast').textContent()).toBe('Saved');
// ✓ No timer. Wait for the thing that actually marks success.await page.getByRole('button', { name: 'Save' }).click();
awaitexpect(page.getByRole('status')).toHaveText('Saved');
If you genuinely need to wait for a network call, use page.waitForResponse(/api\/save/) — but usually a UI assertion is what you want.
Auth state reuse
Logging in inside every test is slow and flaky. Playwright's pattern: a one-time setup project that logs in and saves storageState to disk, then every test starts from that state. (playwright-best-practices)
For multi-role suites, run a setup per role and use test.use({ storageState: ... }) per file.
Test isolation
Each test must be independent. Playwright's docs are blunt: "Each test should be completely isolated from another test and should run independently with its own local storage, session storage, data, cookies etc." (playwright-best-practices)
In practice that means:
A fresh browser context per test (Playwright does this by default).
Test data isolated per test — typically by creating uniquely-named fixtures, or by tearing down via API after the test.
No relying on test execution order. Tests in test.describe.parallel run in parallel; tests anywhere can run on different workers.
test.beforeEach(async ({ request }) => {
// Clean slate via API, not via UI.await request.delete(`/api/test-utilities/clean`);
});
Mock at the network layer
The docs say: "Don't try to test links to external sites or third party servers that you do not control." (playwright-best-practices)
For Stripe/Auth0/etc.: don't drive their UIs in your E2E suite. Mock the redirect-back step or use their test-mode endpoints.
Trace-on-first-retry
Traces give you a timeline + DOM snapshots + network log of a failure — gold for debugging CI flake. The docs recommend running them only on retry to keep happy-path runs cheap: (playwright-best-practices)
In CI, upload test-results/ as an artifact so traces are downloadable. Open with npx playwright show-trace trace.zip.
Parallelism and sharding
// File-level parallelism (within a file).
test.describe.configure({ mode: 'parallel' });
# .github/workflows/e2e.yml — shard across N CI machines.strategy:matrix:shard: [1/4, 2/4, 3/4, 4/4]
steps:-run:npxplaywrighttest--shard=${{matrix.shard}}
Sharding distributes tests across runners (playwright-best-practices). Pair with github-actions-matrix-patterns for the CI side. Keep individual tests under ~30s; long tests bottleneck a shard.
Page-object pattern (lightweight)
Heavy POM hierarchies become their own maintenance burden. The lightweight version: helpers that wrap multi-step user journeys. Don't wrap every locator.
A helper layer is fine when it represents a real user workflow. A "ButtonPage" wrapping a single button is bureaucracy.
Soft assertions
Use to collect multiple failures before failing the test:
await expect.soft(page.getByTestId('total')).toHaveText('$108.69');
await expect.soft(page.getByTestId('shipping')).toHaveText('$0.00');
await page.getByRole('button', { name: 'Place order' }).click();
// Test fails at the end if any soft assertion failed,// but you see all the violations in one run.
Useful for end-of-test verification rows. Don't use them where a failure should stop further interaction.
Snapshots live in __screenshots__/ per platform. Maintenance cost is real — typically reserve for high-value canvases (login page, checkout). Better to spend the budget on functional E2Es first.
Anti-patterns
Brittle CSS selectors
Symptom: A designer changes a class, 20 tests break.
Diagnosis: Tests use .btn.btn-primary.save-action-style selectors.
Fix:getByRole, getByLabel, getByTestId (with intentional data-testid). CSS only as last resort, with a comment.
waitForTimeout everywhere
Symptom: Suite is slow AND flaky.
Diagnosis: Sleeps mask race conditions sometimes; let them through other times.
Fix:await expect(locator).toX() web-first assertions. Replace each waitForTimeout with the actual condition you were waiting for.
Login per test
Symptom: Suite is dominated by login latency; auth provider rate-limits in CI.
Diagnosis: Every test runs the full login flow.
Fix: Setup project + storageState reuse. (playwright-best-practices)
Reading state with synchronous methods then asserting
Symptom: Test passes 9 of 10 runs.
Diagnosis:expect(await locator.textContent()).toBe('X') reads once, doesn't retry.
Fix:await expect(locator).toHaveText('X'). Same for visibility, count, value, attribute.
Tests share data via DB
Symptom: Test A passes alone, fails when B runs first.
Diagnosis: Both write to the same row; no isolation.
Fix: Per-test unique data (suffix with test.info().testId). Reset via API in beforeEach.
Hitting third-party services live
Symptom: Stripe / Auth0 outage takes the suite red. Quota errors on busy CI days.
Diagnosis: E2E driving real third parties.
Fix:page.route() mocks. Test-mode endpoints where mocks aren't realistic. (playwright-best-practices)
Single-process serial run
Symptom: Suite takes 45 minutes; engineers stop running it pre-PR.
Diagnosis: No parallelism, no sharding.
Fix: Enable parallel mode within files; shard across CI workers; cap individual test latency.
Trace artifacts not retained
Symptom: A flake on CI; no trace, no screenshot, no video.
Diagnosis:trace: 'off' or no artifact upload.
Fix:trace: 'on-first-retry', upload test-results/ as a CI artifact. (playwright-best-practices)
Quality gates
Test: suite runs in parallel by default; mode: 'parallel' in test.describe.configure or globally.
Test: flake rate measured per nightly run; alert if > 1% over a week.
Test:npx playwright test --grep @smoke runs in < 2 min for the smoke subset.
No page.waitForTimeout calls in production tests. CI grep fails on hits.
No await locator.isVisible() / textContent() followed by a sync expect. Lint or grep enforces.
All locators are getByRole / getByLabel / getByTestId first; CSS as last resort with a comment.
Auth-state reuse via storageState; login flow itself tested once in a setup project.
Third-party calls mocked via page.route() or test-mode endpoints. CI denies network egress to unrelated hosts.
trace: 'on-first-retry' configured; CI uploads test-results/ as an artifact on failure.
retries: 2 on CI, retries: 0 locally (so flake is visible to authors).
Each test creates its own data with unique IDs; beforeEach cleans state via API.
Suite shards across CI workers (--shard=N/M); see github-actions-matrix-patterns for the matrix.
Individual test p95 < 30s. Long tests broken up.
NOT for
Unit / component testing — Vitest, Jest, Vitest Browser Mode, RTL. No dedicated skill yet.
Cypress migration specifically — overlapping but distinct. No dedicated skill yet.
Mobile native E2E (Detox, XCUITest, Espresso) — different runtime.
Visual regression as the primary concern — Chromatic, Percy, Argos are dedicated tools.
Load testing — k6, Gatling, Artillery. Different goal.
Security testing — ZAP, Burp. Different threat model.
CI matrix design for the suite — → github-actions-matrix-patterns.
Deterministic Audit
Before adopting (or reviewing) a suite design, write it as a JSON plan matching
schemas/playwright-e2e-design-plan.schema.json and run it through the deterministic auditor:
auditPlaywrightE2eDesign(plan) (in scripts/playwright_e2e_design_audit.mjs) turns this
skill's compressed rule — getByRole/getByTestId + web-first expect() + fresh state per test
— and its Quality Gates into machine-checkable rules over structured fields, no keyword
matching: snapshot-read assertions (the guaranteed race), remaining waitForTimeout calls,
CSS/XPath as the primary locator strategy, login-per-test instead of storageState reuse,
live third-party calls, missing test isolation, trace: off (or no artifact upload), inverted
retry policy (retries locally, none on CI), a serial-only suite, and tests blowing the 30s p95
budget. It returns { pass, score, findings, recommendations } so a reviewer or CI gate can
reject a flake-prone design without re-deriving the playbook. examples/sample-input.json is
a sharded, mocked, storage-state suite plan that audits pass: true. Version history lives in
CHANGELOG.md.
