Detect, quarantine, and systematically fix flaky tests with automated retry analysis, root cause categorization, and CI pipeline integration for test reliability
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Detect, quarantine, and systematically fix flaky tests with automated retry analysis, root cause categorization, and CI pipeline integration for test reliability
Flaky tests are tests that produce different outcomes (pass or fail) when run against the same code under the same conditions. They erode confidence in the test suite, train developers to ignore failures, and slow down CI pipelines with unnecessary retries. A single flaky test in a suite of 500 can cause the entire pipeline to require re-runs, wasting developer time and compute resources. This skill provides a systematic approach to detecting, quarantining, categorizing, and fixing flaky tests while maintaining CI pipeline stability.
Core Principles
Detection Before Quarantine: A test must be proven flaky through repeated execution before it is quarantined. A single failure does not make a test flaky; it might indicate a real bug. Multi-run analysis with statistical tracking separates genuine flakiness from legitimate failures.
Quarantine Is Not Deletion: Quarantined tests must remain in the codebase and continue running in a separate pipeline. Quarantine is a temporary holding pattern that prevents flaky tests from blocking the main pipeline while preserving the test and tracking its behavior.
Root Cause Categorization Drives Fixes: Different types of flakiness require different fixing strategies. Timing issues need explicit waits, state leakage needs proper cleanup, external dependencies need mocking, and race conditions need synchronization. Categorizing the root cause directs the fix.
Flakiness Is Measurable: Track the flakiness rate (failures per total runs) for every test over time. This metric determines quarantine decisions, fix prioritization, and verifies that fixes actually resolved the issue.
Zero-Tolerance Pipeline: The main CI pipeline must be green to merge. Allowing occasional failures or manual re-runs normalizes flakiness and makes it impossible to distinguish real failures from known flaky ones.
Fix the Root Cause, Not the Symptom: Adding retries or increasing timeouts masks flakiness without fixing it. These approaches hide real issues and increase overall test execution time. Address the underlying non-determinism.
Isolation Verification: After fixing a flaky test, verify the fix by running the test in isolation and in the full suite multiple times. Some flakiness only manifests under specific ordering or parallel execution conditions.
The most reliable way to detect flaky tests is to run the test suite multiple times and compare results. A test that fails even once across multiple runs of identical code is flaky.
// scripts/detect-flaky.tsimport { execSync } from'child_process';
import * as fs from'fs';
interfaceTestResult {
testName: string;
file: string;
runs: number;
passes: number;
failures: number;
flakinessRate: number;
failureMessages: string[];
}
interfaceDetectionConfig {
runs: number;
testCommand: string;
resultPattern: string;
flakinessThreshold: number;
}
constconfig: DetectionConfig = {
runs: 10,
testCommand: 'npx playwright test --reporter=json',
resultPattern: 'test-results/results.json',
flakinessThreshold: 0.1, // 10% failure rate = flaky
};
asyncfunctiondetectFlakyTests(): Promise<TestResult[]> {
const resultsByTest = newMap<string, TestResult>();
console.log(`Running test suite ${config.runs} times to detect flaky tests...`);
for (let run = 1; run <= config.runs; run++) {
console.log(`\n--- Run ${run}/${config.runs} ---`);
try {
execSync(config.testCommand, {
stdio: 'pipe',
env: {
...process.env,
PLAYWRIGHT_JSON_OUTPUT_NAME: `test-results/run-${run}.json`,
},
});
} catch {
// Test failures are expected; we continue regardless
}
const resultsFile = `test-results/run-${run}.json`;
if (!fs.existsSync(resultsFile)) continue;
const results = JSON.parse(fs.readFileSync(resultsFile, 'utf-8'));
for (const suite of results.suites || []) {
for (const spec of suite.specs || []) {
const key = `${suite.file}::${spec.title}`;
if (!resultsByTest.has(key)) {
resultsByTest.set(key, {
testName: spec.title,
file: suite.file,
runs: 0,
passes: 0,
failures: 0,
flakinessRate: 0,
failureMessages: [],
});
}
const entry = resultsByTest.get(key)!;
entry.runs++;
const passed = spec.tests.every((t: any) => t.status === 'passed');
if (passed) {
entry.passes++;
} else {
entry.failures++;
const failMsg = spec.tests
.filter((t: any) => t.status === 'failed')
.map((t: any) => t.results?.[0]?.error?.message || 'Unknown error')
.join('; ');
entry.failureMessages.push(failMsg);
}
}
}
}
// Calculate flakiness ratesconstallResults: TestResult[] = [];
for (const result of resultsByTest.values()) {
result.flakinessRate = result.failures / result.runs;
allResults.push(result);
}
// Filter to only flaky tests (failed some but not all runs)const flakyTests = allResults.filter(
(r) => r.flakinessRate > 0 && r.flakinessRate < 1 && r.flakinessRate >= config.flakinessThreshold
);
console.log(`\nDetected ${flakyTests.length} flaky tests out of ${allResults.length} total tests`);
// Write report
fs.writeFileSync(
'test-results/flaky-detection-report.json',
JSON.stringify(
{
timestamp: newDate().toISOString(),
totalRuns: config.runs,
totalTests: allResults.length,
flakyTests: flakyTests.sort((a, b) => b.flakinessRate - a.flakinessRate),
},
null,
2
)
);
return flakyTests;
}
detectFlakyTests().catch(console.error);
// tests/e2e/checkout.spec.tsimport { test, expect } from'./fixtures/quarantine-fixture';
test.describe('Checkout Flow', () => {
test('should complete checkout with credit card', async ({ page }) => {
// This test is stable - runs normallyawait page.goto('/products/1');
await page.click('[data-testid="add-to-cart"]');
await page.goto('/checkout');
await page.fill('[data-testid="card-number"]', '4242424242424242');
await page.click('[data-testid="place-order"]');
awaitexpect(page.locator('[data-testid="order-confirmation"]')).toBeVisible();
});
test('should complete checkout with PayPal', async ({ page, quarantine }) => {
// This test is in the quarantine registry - will be skipped in main pipelineawait page.goto('/products/1');
await page.click('[data-testid="add-to-cart"]');
await page.goto('/checkout');
await page.click('[data-testid="pay-with-paypal"]');
// PayPal iframe interaction...awaitexpect(page.locator('[data-testid="order-confirmation"]')).toBeVisible();
});
});
Root Cause Categories and Fixing Strategies
Timing Issues
Timing flakiness occurs when tests assume operations complete within a fixed time. The fix is to replace arbitrary timeouts with explicit condition waits.
// FLAKY: Uses fixed timeouttest('should show notification after save', async ({ page }) => {
await page.click('[data-testid="save-button"]');
await page.waitForTimeout(2000); // Arbitrary wait - FLAKYconst notification = page.locator('[data-testid="notification"]');
awaitexpect(notification).toBeVisible();
});
// FIXED: Waits for specific conditiontest('should show notification after save', async ({ page }) => {
await page.click('[data-testid="save-button"]');
const notification = page.locator('[data-testid="notification"]');
awaitexpect(notification).toBeVisible({ timeout: 10000 }); // Explicit condition wait
});
State Leakage Between Tests
State leakage happens when one test modifies shared state (database, browser storage, global variables) that another test depends on.
// FLAKY: Tests share state
test.describe('User Settings', () => {
test('should enable dark mode', async ({ page }) => {
await page.goto('/settings');
await page.click('[data-testid="dark-mode-toggle"]');
// Leaves dark mode enabled for next test
});
test('should show default light theme', async ({ page }) => {
await page.goto('/settings');
// FAILS if previous test ran first and enabled dark modeawaitexpect(page.locator('body')).toHaveClass(/light-theme/);
});
});
// FIXED: Each test manages its own state
test.describe('User Settings', () => {
test.beforeEach(async ({ page }) => {
// Reset user preferences before each testawait page.evaluate(() =>localStorage.clear());
await page.request.post('/api/test/reset-user-preferences');
});
test('should enable dark mode', async ({ page }) => {
await page.goto('/settings');
await page.click('[data-testid="dark-mode-toggle"]');
awaitexpect(page.locator('body')).toHaveClass(/dark-theme/);
});
test('should show default light theme', async ({ page }) => {
await page.goto('/settings');
awaitexpect(page.locator('body')).toHaveClass(/light-theme/);
});
});
// FLAKY: Race condition between navigation and assertiontest('should load dashboard data', async ({ page }) => {
await page.goto('/dashboard');
// Data might not have loaded yetconst count = await page.locator('[data-testid="item-count"]').textContent();
expect(parseInt(count!)).toBeGreaterThan(0);
});
// FIXED: Wait for the data to be in the expected statetest('should load dashboard data', async ({ page }) => {
await page.goto('/dashboard');
// Wait for the loading state to completeawait page.waitForResponse((response) =>
response.url().includes('/api/dashboard') && response.status() === 200
);
// Now assert on the dataconst countLocator = page.locator('[data-testid="item-count"]');
awaitexpect(countLocator).not.toHaveText('0');
const count = await countLocator.textContent();
expect(parseInt(count!)).toBeGreaterThan(0);
});
# .github/workflows/quarantine-ci.ymlname:QuarantinePipelineon:push:branches: [main]
schedule:-cron:'0 */4 * * *'# Every 4 hoursjobs:quarantine-tests:runs-on:ubuntu-lateststeps:-uses:actions/checkout@v4-uses:actions/setup-node@v4with:node-version:20-run:npmci-run:npxplaywrightinstall--with-deps# Run only quarantined tests with retries-name:Runquarantinedtestsrun:npxplaywrighttest--retries=3continue-on-error:true# Don't fail the pipelineenv:QUARANTINE_RUN:'true'-name:Updateflakinesstrackingrun:npxts-nodescripts/flaky-report.tsenv:CI_RUN_ID:${{github.run_id}}-name:Checkforrecoveredtestsrun:|
npx ts-node scripts/check-recovered.ts
# Outputs tests that passed all retries, candidates for unquarantine
Test Isolation Verification
// scripts/verify-fix.tsimport { execSync } from'child_process';
interfaceVerificationResult {
testName: string;
isolatedRuns: { passed: number; failed: number };
suiteRuns: { passed: number; failed: number };
verdict: 'fixed' | 'still-flaky' | 'order-dependent';
}
asyncfunctionverifyFix(testFile: string, testName: string): Promise<VerificationResult> {
constRUNS = 20;
constresult: VerificationResult = {
testName,
isolatedRuns: { passed: 0, failed: 0 },
suiteRuns: { passed: 0, failed: 0 },
verdict: 'still-flaky',
};
console.log(`Verifying fix for "${testName}" with ${RUNS} runs...`);
// Phase 1: Run the test in isolationconsole.log('\nPhase 1: Isolated runs');
for (let i = 0; i < RUNS; i++) {
try {
execSync(
`npx playwright test "${testFile}" --grep "${testName}" --retries=0`,
{ stdio: 'pipe' }
);
result.isolatedRuns.passed++;
} catch {
result.isolatedRuns.failed++;
}
}
console.log(` Isolated: ${result.isolatedRuns.passed}/${RUNS} passed`);
// Phase 2: Run the test within the full suiteconsole.log('\nPhase 2: Full suite runs');
for (let i = 0; i < RUNS; i++) {
try {
execSync(`npx playwright test --retries=0`, { stdio: 'pipe' });
result.suiteRuns.passed++;
} catch {
result.suiteRuns.failed++;
}
}
console.log(` Suite: ${result.suiteRuns.passed}/${RUNS} passed`);
// Determine verdictif (result.isolatedRuns.failed === 0 && result.suiteRuns.failed === 0) {
result.verdict = 'fixed';
} elseif (result.isolatedRuns.failed === 0 && result.suiteRuns.failed > 0) {
result.verdict = 'order-dependent';
} else {
result.verdict = 'still-flaky';
}
console.log(`\nVerdict: ${result.verdict}`);
return result;
}
Establish a flakiness budget. Set a target for overall suite flakiness (e.g., less than 2% of tests are flaky at any time). Track this metric and treat exceeding the budget as a team priority.
Automate quarantine decisions. When a test exceeds a configurable flakiness threshold (e.g., 15% failure rate over 50 runs), automatically quarantine it and create a tracking ticket.
Run quarantined tests in a separate CI job. This keeps the main pipeline reliable while still executing quarantined tests to track their behavior and detect if a code change inadvertently fixes them.
Assign ownership for quarantined tests. Every quarantined test should have an assignee and a tracking ticket. Unowned quarantined tests accumulate indefinitely.
Set time limits on quarantine. A test quarantined for more than 30 days without progress should be escalated. Indefinite quarantine is equivalent to deletion.
Fix by root cause category. Maintain a playbook for each root cause type. Timing issues require wait-for-condition patterns, state leakage requires setup/teardown improvements, and external dependencies require mocking.
Verify fixes with multi-run confirmation. A fix is not verified by a single passing run. Run the previously flaky test at least 20 times in both isolated and full-suite modes to confirm stability.
Use test annotations to communicate quarantine status. Annotate quarantined tests in the code so developers reviewing test files understand which tests are quarantined and why.
Track flakiness trends over time. Monitor whether the overall flakiness rate is improving or degrading. This reveals systemic issues and measures the effectiveness of reliability efforts.
Review test infrastructure alongside test code. Flakiness often originates from CI runner resource constraints, Docker networking issues, or shared test databases. Investigate infrastructure when flakiness patterns span unrelated tests.
Anti-Patterns to Avoid
Adding retries to the main pipeline as a permanent solution. Retries mask flakiness and increase pipeline duration. They should only be used temporarily in quarantine pipelines while fixes are in progress.
Deleting flaky tests instead of fixing them. Flaky tests often cover real functionality. Deleting them trades test reliability for reduced test coverage. Always fix first; only delete if the test provides no value.
Increasing timeouts globally to address timing flakiness. Raising the global timeout from 30 seconds to 60 seconds slows down the entire suite and does not fix the underlying issue. Use targeted explicit waits.
Blaming the test framework for flakiness. While framework bugs exist, the vast majority of flakiness is caused by test design issues. Investigate test code before concluding the framework is at fault.
Quarantining tests without creating tracking tickets. Quarantine without accountability leads to a growing graveyard of disabled tests. Every quarantine action must create a tracked work item.
Running flaky detection only once. Flakiness evolves as the codebase changes. Schedule weekly flaky detection runs to catch newly flaky tests before they accumulate.
Fixing flakiness by adding sleep statements. Fixed sleeps are inherently unreliable because system performance varies. Replace sleeps with condition-based waits that react to actual state changes.
Debugging Tips
Enable trace recording on retry. Configure Playwright to capture traces on first retry (trace: 'on-first-retry'). Traces provide a complete timeline of actions, network requests, and DOM snapshots for diagnosing intermittent failures.
Compare passing and failing run logs side by side. The difference between a passing and failing run often reveals the root cause: a missing API response, a slower DOM update, or a different data state.
Check test execution order. Run the failing test after specific other tests to identify order-dependent flakiness. Use --shard or randomized ordering to expose hidden dependencies.
Examine CI runner resource utilization. High CPU or memory usage on CI runners causes timing-related flakiness. Check runner metrics during test execution to identify resource contention.
Look for time-zone and locale sensitivity. Tests that pass in one time zone but fail in another often involve date formatting or comparison. CI runners may use different locales than development machines.
Inspect network timing in traces. Network-related flakiness shows up as variable response times or timeout errors in traces. Consider mocking slow or unreliable external services.
Run the test in a loop locally. Use for i in $(seq 1 50); do npx playwright test "test-name" || echo "FAILED on run $i"; done to reproduce flakiness locally before attempting a fix.
Check for shared mutable state. Global variables, module-level caches, and singleton patterns can leak state between tests when running in parallel. Review test setup and teardown for completeness.
Verify database cleanup completeness. If tests share a database, ensure that beforeEach/afterEach hooks clean up all relevant tables. Missing cleanup of junction tables or audit logs is a common cause of state leakage.
Use deterministic test data. Random data generation without fixed seeds can cause tests to pass or fail depending on the generated values. Use seeded random generators or fixed test data when determinism matters.
