flaky-test-detector

name	flaky-test-detector
description	Identify and fix intermittently failing tests. Use when the user mentions flaky tests, random failures, or unreliable CI.

Flaky Test Detector

Identify, analyze, and fix flaky tests that undermine CI/CD reliability.

Description

This skill helps QA teams identify flaky tests (tests that pass and fail intermittently without code changes), analyze their root causes, and generate fixes. Flaky tests erode team confidence in test suites and slow down development.

Instructions

When the user provides test results, test code, or describes intermittent failures:

1. Identify Flakiness: Analyze test results across multiple runs to find inconsistent tests
2. Classify Root Cause: Determine the type of flakiness
3. Analyze Code: Review the test code for common flaky patterns
4. Generate Fix: Provide specific code changes to eliminate flakiness
5. Prevention Guide: Recommend practices to prevent new flaky tests

Common Flakiness Patterns

Timing & Async Issues

• Symptom: Test passes locally, fails in CI (or vice versa)
• Causes: Hard-coded sleep/delays, race conditions, async operations not properly awaited
• Fix Pattern: Replace sleep() with explicit waits, use polling with timeouts, await promises/futures

Test Order Dependency

• Symptom: Test fails when run alone but passes in full suite (or vice versa)
• Causes: Shared state between tests, missing setup/teardown, global variable mutation
• Fix Pattern: Isolate test state, use fresh fixtures, reset globals in beforeEach/afterEach

Environment Dependency

• Symptom: Test fails on specific machines or CI environments
• Causes: Hardcoded paths, timezone assumptions, locale-dependent formatting, OS-specific behavior
• Fix Pattern: Use relative paths, explicit timezones, locale-agnostic assertions

Data Dependency

• Symptom: Test fails after database changes or with specific data states
• Causes: Shared test database, non-deterministic query ordering, auto-increment ID assumptions
• Fix Pattern: Use isolated test data, explicit ordering, avoid ID-dependent assertions

Network & External Services

• Symptom: Test fails intermittently with timeout or connection errors
• Causes: Real API calls in tests, DNS resolution, network latency
• Fix Pattern: Mock external services, use recorded responses, add retry with backoff

Resource Contention

• Symptom: Test fails under parallel execution
• Causes: Shared files, ports, database rows, global state
• Fix Pattern: Use unique resources per test, randomize ports, isolate file operations

Analysis Checklist

## Flaky Test Analysis: [test_name]

### Identification
- [ ] Failure rate: X out of Y runs
- [ ] First observed: [date]
- [ ] Affected environments: [list]
- [ ] Passes in isolation: yes/no

### Root Cause Category
- [ ] Timing/Async
- [ ] Test Order Dependency
- [ ] Environment Dependency
- [ ] Data Dependency
- [ ] Network/External Service
- [ ] Resource Contention
- [ ] Non-deterministic Logic

### Evidence
- [Log snippets, error messages, timing data]

### Recommended Fix
- [Specific code changes]

### Prevention
- [What practice would have prevented this]

Code Review Patterns

When reviewing test code, flag these anti-patterns:

FLAKY RISK: sleep()/setTimeout() without condition check
FLAKY RISK: Shared database without transaction isolation
FLAKY RISK: Assertions on timestamps without tolerance
FLAKY RISK: Unordered collection compared with strict equality
FLAKY RISK: File system operations without cleanup
FLAKY RISK: Port hardcoding without availability check
FLAKY RISK: Date.now() or new Date() in assertions
FLAKY RISK: Missing await/async handling

Example Usage

These tests are flaky in our CI pipeline. Here are results from 5 runs:
Run 1: test_checkout PASS, test_payment FAIL
Run 2: test_checkout FAIL, test_payment PASS
...
Here's the test code: [paste code]

Review this test file and identify potential flaky test risks:
[paste test code]