| name | flaky-test-triage |
| description | Structured procedure for detecting, classifying, and eliminating flaky tests — covers quarantine mechanics, root-cause patterns (timing, state, network, ordering), and the fix or delete decision. |
Flaky Test Triage
When to Use This
A test passes sometimes and fails sometimes with no code change. Use this skill to move from "just re-run it" (which normalizes the flakiness) to a quarantined, owned, time-bounded remediation.
Step 1 — Confirm It Is Truly Flaky
Run the test 10× in isolation (no parallelism, clean state each time):
# Playwright example
npx playwright test <test-file> --repeat-each=10
# Jest example
npx jest <test-file> --testNamePattern "<test-name>" --forceExit
If it fails in isolation, it's not a flake — it's a real regression. Treat it as a bug.
If it only fails when run with the full suite, it is an ordering/state pollution flake (see classification below).
Step 2 — Classify the Root Cause
| Pattern | Observable | Typical cause |
|---|
| Timing | Fails with "element not found" / "timeout" / "assertion on async value" | Fixed sleep; polling without a condition; race between UI render and assertion |
| State pollution | Fails after specific other tests; passes in isolation | Shared database rows; global singletons; missing teardown |
| Network/external | Fails with connection errors; highly environment-dependent | Real HTTP call to staging/prod; uncontrolled third-party API |
| Order-dependent | Fails only in full-suite run, order varies | Test A leaves dirty state; Test B depends on Test A's setup |
| Resource contention | Fails only in parallel runs; passes serially | Port conflicts; file-system races; shared fixtures with mutation |
| Environment-sensitive | Fails only in CI, passes locally | Different timezone/locale; different clock skew; different file-system case sensitivity |
Step 3 — Quarantine Immediately
While investigating, quarantine so the failing test doesn't pollute the suite's signal:
test.fixme('should submit order', async ({ page }) => { ... });
test.skip('should submit order', () => { ... });
@pytest.mark.skip(reason="flaky: timing issue, owner: @alice, deadline: 2026-06-19, issue: #1234")
Quarantine rules:
- One named owner per flaky test.
- A deadline: fix or delete within 2 sprint cycles (≈ 4 weeks).
- Linked to a tracking issue.
- Never leave it quarantined without a date — the graveyard grows.
Step 4 — Fix by Root Cause
Timing:
await page.click('#submit');
await page.waitForTimeout(2000);
await page.click('#submit');
await page.waitForSelector('[data-testid="confirmation"]', { state: 'visible' });
State pollution:
- Add
beforeEach/afterEach teardown that resets all shared state.
- Use test-scoped database transactions that roll back after each test (if your ORM supports it).
- Inject unique identifiers (UUIDs) per test run to namespace shared resources.
Network/external:
- Replace real HTTP calls with a test double (MSW for browsers/Node; WireMock for Java;
responses for Python requests).
- If you must hit real endpoints, run those tests in a separate suite tagged
[integration] and exclude them from the fast gate.
Order-dependent:
- Run the suite with
--randomize to surface ordering dependencies.
- Identify the "dirty" test with bisect:
pytest --randomly-seed=<seed>.
- Add explicit teardown to the dirty test; never rely on test ordering.
Step 5 — Fix or Delete Decision
| Situation | Decision |
|---|
| Test covers real behavior; fix is clear | Fix |
| Test covers real behavior; fix takes > 1 sprint | Quarantine with owner + deadline, keep in backlog |
| Test duplicates coverage of a more stable test | Delete |
| Test covers behavior no longer in the product | Delete |
| Test has been quarantined > 2 sprint cycles with no owner | Delete (the behavior it tested should be covered elsewhere or added back properly) |
Never leave a quarantined test without a deletion date.
Pitfalls
- Re-running in CI without tracking — every re-run that passes trains the team to ignore red and hides real regressions behind "it'll probably pass next run."
- Adding
sleep to fix a timing flake — it makes the test slower and often still flaky under load; wait on observable state instead.
- Fixing the flake without removing the
fixme annotation — the test runs but is forever marked broken.
- Quarantining without an owner — the test accumulates in the skip list and coverage silently drops.
See Also
