flaky-test-detective

name

flaky-test-detective

description

Drive a flaky (intermittently failing) test to its root cause and proven stability through the fixed sequence CLASSIFY (ordering, shared state, timing/race, environment, infrastructure, or real intermittent product bug) → REPRODUCE deterministically (repeat runs, order shuffling, parallel stress, seed/timezone control) → FIX ONE CAUSE → PROVE stability with repeated runs and real counts. Never fixes by adding retries, sleeps, or looser assertions; distinguishes test bugs from product bugs (a race the test exposes may ship to users). Manages quarantine only with owner, ticket, and expiry. Use when a test fails intermittently, passes on retry, fails only in CI or only in parallel, or a suite's trust is eroding from random red. Do NOT use for tests that fail consistently (systematic-debugger for the product bug, or fix the test), for setting suite-wide retry/flake policy (qa-automation-architect), or for deciding suite membership (regression-suite-curator).

Flaky Test Detective

Purpose

Convert "it fails sometimes" into a named root cause, one targeted fix, and statistical evidence of stability — instead of the usual retries, sleeps, and eroding trust. The deliverable is a case report: classification with evidence, a deterministic reproduction (or an honest account of why not), the single cause fixed, before/after stability counts from real repeated runs, and a prevention note. Flakes are also triaged honestly: some "flaky tests" are real intermittent product bugs wearing a test's name.

Use When

• Use when: a test fails intermittently — locally, in CI, on retry, only in parallel, only at certain times.
• Use when: CI passes-on-retry are accumulating (retry telemetry from the automation blueprint's policy) and cases need working.
• Use when: a quarantined test's expiry arrives and it needs diagnosis or release.
• Do NOT use when: the test fails EVERY run — that's a plain failure: product bug → systematic-debugger; test bug → fix at its layer's engineer skill.
• Do NOT use when: setting suite-wide retry/quarantine POLICY — qa-automation-architect; this skill works individual cases under that policy.
• Do NOT use when: deciding whether a test belongs in the suite at all — regression-suite-curator (a chronic flake may be a curation candidate, flagged as this skill's output).

Inputs to Inspect

1. Failure history: CI runs, frequency, pass-on-retry counts, first appearance (what merged around then), failure message VARIANTS (multiple distinct errors = possibly multiple causes).
2. The test code and its fixtures: shared state, module-level setup, data it assumes, timers/awaits, network/mocks.
3. Execution context of failures vs passes: parallel workers, order/seed, CI runner class vs local, timezone/locale, resource pressure.
4. The code under test: async paths, caches, background effects — candidate real races.
5. The flake policy in force (quarantine rules, retry settings) so case handling conforms.

Workflow

1. Classify first against the taxonomy in references/flake-taxonomy.md: ordering dependence, shared state (data/global/module), timing/race (test-side or product-side), environment (TZ/locale/CI-runner), infrastructure (network/container), or intermittent product bug. Evidence: failure variants, context diffs between passing and failing runs. State the leading hypothesis and what observation would refute it.
2. Reproduce deterministically. Escalate: repeat the single test N times (runner repeat flags / loop); shuffle order + fixed seeds to expose ordering; run the suite in parallel locally to expose shared state; pin TZ/locale/env to CI's values; add targeted delay/load at the suspected race point to make it reliable. Record exact commands + counts (e.g., "fails 7/50 shuffled, 0/50 sequential" is a classification result in itself).
3. Isolate the ONE cause the reproduction demonstrates. Multiple suspected causes → separate cases, fixed one at a time (per systematic-debugger discipline — one change per attempt).
4. Triage: test bug or product bug? If the race lives in product code, the flake is a FINDING about production behavior — route the product fix (systematic-debugger/owner) and keep the test; do not "harden" the test into hiding a real race.
5. Fix that one cause at the test layer when it is a test bug: real isolation (own data per test-data-architect patterns), await the actual condition (web-first/waitFor semantics per the layer's engineer skill), control time/seed, remove order dependence. FORBIDDEN as fixes: retries, sleeps, widened timeouts without a named reason, weakened assertions, deletion-to-green.
6. Prove stability with counts: re-run the reproduction protocol — before (fails k/N under condition X) vs after (0/N under the same X, N ≥ the count that reliably reproduced). One green run proves nothing.
7. Close the case: report; if quarantined, release or extend WITH owner/ ticket/expiry per policy; add the prevention note (pattern to lint/avoid); flag chronic offenders to regression-suite-curator.

Output Format

FLAKE CASE — <test id/name>
History: <frequency, first seen, pass-on-retry counts, failure variants>
Classification: <category + evidence + refutable hypothesis>
Reproduction: <exact commands/conditions → counts (fails k/N under X)>
Cause: <the ONE demonstrated cause>
Triage: <test bug | product bug (routed with evidence) | infra (routed)>
Fix: <the one change made; forbidden-fix check passed>
Stability proof: <before k/N vs after 0/N under identical conditions, N=<n>>
Quarantine action: <none | released | extended (owner, ticket, expiry)>
Prevention: <pattern note; chronic-offender flag → regression-suite-curator>

Validation Checklist

• Classification stated with evidence before any fix.
• Reproduction attempted with escalating techniques; commands + counts recorded (or the honest why-not).
• Exactly ONE cause fixed in this case.
• Product-side races routed as product bugs, test kept honest.
• No retries/sleeps/loosened assertions/deletions as "fixes".
• Stability proven by repeated runs under the reproducing condition, with counts, not one green run.
• Quarantine actions carry owner + ticket + expiry.

Gotchas

• Pass-on-retry in CI is a flake SIGNAL being suppressed — treat retry telemetry as the case intake queue, not as the fix.
• "Fails only in CI" usually means resource pressure, TZ/locale, or worker parallelism — pin those locally before blaming infrastructure.
• Multiple distinct failure messages from one test = likely multiple causes; fixing one and seeing "still flaky" is expected — file separate cases.
• A sleep that "fixes" it confirms a race and fixes nothing — the condition the code awaits is the fix.
• Time bombs: tests failing near midnight, month-end, or DST transitions are environment flakes with a schedule — check failure timestamps for clustering.
• Deleting the flaky test deletes the coverage AND the evidence of a possible product race — curation is a decision for regression-suite-curator with the case report attached.

Stop Conditions

• Cannot reproduce after the escalation ladder → report techniques tried with counts, park with enhanced logging/telemetry on next CI occurrences — do NOT guess-fix an unreproduced flake.
• Reproduction demonstrates a product-side race with user impact → stop test-side work; route immediately as a product bug with the reproduction.
• The "fix" would be a retry/sleep/weakened assertion because the real fix needs product refactoring → surface the tradeoff for a scoped decision instead of silently degrading the test.
• Flake source is shared CI infrastructure beyond the repo → route to the infra owner with evidence; don't mutate tests to mask infra.

Supporting Files

• references/flake-taxonomy.md — the classification taxonomy with signatures, the reproduction escalation ladder with runner-specific repeat/shuffle commands, and the forbidden-fix list with allowed alternatives.
• evals/evals.json — trigger + behavior cases.
• evals/trigger-evals.json — discrimination within the unit/build/flake/data cluster and against systematic-debugger.