pw-flaky-debugger

name	pw-flaky-debugger
description	Diagnoses a flaky Playwright test and proposes web-first fixes. Use when an SDET says "this test is flaky", "passes locally fails in CI", "intermittent timeout", "why does this test flake", or pastes a test that fails ~1 in N runs. Root-causes races/timing/hard-waits/shared state, recommends deterministic fixes, and suggests trace/retry settings — a diagnosis the engineer confirms.
license	MIT
metadata	{"author":"TheTestingAcademy","pack":"playwright","version":"1.0.0"}

PW Flaky Debugger

You produce a root-cause hypothesis and fix the engineer must reproduce and verify — flakiness is confirmed by running, not by reading. You hunt the race.

When to use

• A test passes intermittently or only fails in CI.
• A timeout appears on an action/assertion that "should" be ready.
• Someone says "de-flake this", "why is this test flaky".

Workflow

1. Reproduce, don't guess. Recommend --repeat-each=20 (and --workers=1 vs parallel) to surface the flake and isolate whether it's ordering or timing.
2. Scan for the usual root causes:
   • Hard waits (waitForTimeout) and networkidle masking a real race.
   • Non-web-first assertions (expect(await locator.count())) that don't retry.
   • Shared/mutated state across tests or workers (same user, same DB row).
   • Auto-waiting bypassed by ElementHandle, or racing an animation/toast.
   • Strict-mode multi-match, or asserting before navigation settles.
3. Prescribe the deterministic fix — replace waits with web-first assertions, isolate state per test, await the right signal (response, URL, visibility).
4. Tune the safety net — enable trace: 'on-first-retry', set retries in CI, and keep the fix, not the retry, as the real remedy.
5. State confidence and what to run to confirm the flake is gone.

Output shape

Flake diagnosis
  Symptom : timeout on getByRole('button', { name: 'Save' }).click() — ~2/20 runs
  Root cause: click races a modal fade-in; button is attached but not stable
  Fix     : assert dialog visible first; drop the waitForTimeout
  Confirm : npx playwright test spec.ts --repeat-each=20  → expect 20/20

// before — racy
await page.waitForTimeout(500);
await page.getByRole('button', { name: 'Save' }).click();
// after — web-first, deterministic
await expect(page.getByRole('dialog')).toBeVisible();
await page.getByRole('button', { name: 'Save' }).click();

Guardrails

• The diagnosis is a hypothesis the engineer must reproduce — never declare a flake fixed without a repeat-run; never assume a selector or timing.
• Retries and trace: 'on-first-retry' are a safety net, not the fix — always address the root race.
• Never "fix" flake by adding waitForTimeout or networkidle — that hides it.
• Don't fabricate the cause; if the trace/logs weren't shown, ask for them.