dyaddeflake-e2e

name	dyad:deflake-e2e
description	Identify and fix flaky E2E tests by running them repeatedly and investigating failures.

Deflake E2E Tests

Identify and fix flaky E2E tests by running them repeatedly and investigating failures.

Arguments

• $ARGUMENTS: (Optional) Specific E2E test file(s) to deflake (e.g., main.spec.ts or e2e-tests/main.spec.ts). If not provided, will prompt to deflake the entire test suite.

Instructions

1. Check if specific tests are provided:

   If $ARGUMENTS is empty or not provided, ask the user:

   "No specific tests provided. Do you want to deflake the entire E2E test suite? This can take a very long time as each test will be run 10 times."

   Wait for user confirmation before proceeding. If they decline, ask them to provide specific test files.

2. Install dependencies:

   npm install
   

3. Build the app binary:

   npm run build
   

   IMPORTANT: This step is required before running E2E tests. E2E tests run against the built binary. If you make any changes to application code (anything outside of e2e-tests/), you MUST re-run npm run build before running E2E tests again, otherwise you'll be testing the old version.

4. Run tests repeatedly to detect flakiness:

   For each test file, run it 10 times:

   PLAYWRIGHT_RETRIES=0 PLAYWRIGHT_HTML_OPEN=never npm run e2e -- e2e-tests/<testfile>.spec.ts --repeat-each=10
   

   IMPORTANT: PLAYWRIGHT_RETRIES=0 is required to disable automatic retries. Without it, CI environments (where CI=true) default to 2 retries, causing flaky tests to pass on retry and be incorrectly skipped as "not flaky."

   Notes:

   • If $ARGUMENTS is provided without the e2e-tests/ prefix, add it
   • If $ARGUMENTS is provided without the .spec.ts suffix, add it
   • A test is considered flaky if it fails at least once out of 10 runs

5. For each flaky test, investigate with debug logs:

   Run the failing test with Playwright browser debugging enabled:

   DEBUG=pw:browser PLAYWRIGHT_RETRIES=0 PLAYWRIGHT_HTML_OPEN=never npm run e2e -- e2e-tests/<testfile>.spec.ts
   

   Analyze the debug output to understand:

   • Timing issues (race conditions, elements not ready)
   • Animation/transition interference
   • Network timing variability
   • State leaking between tests
   • Snapshot comparison differences

6. Fix the flaky test:

   Common fixes following Playwright best practices:

   • Use await expect(locator).toBeVisible() before interacting with elements
   • Use await page.waitForLoadState('networkidle') for network-dependent tests
   • Use stable selectors (data-testid, role, text) instead of fragile CSS selectors
   • Add explicit waits for animations: await page.waitForTimeout(300) (use sparingly)
   • Use await expect(locator).toHaveScreenshot() options like maxDiffPixelRatio for visual tests
   • Ensure proper test isolation (clean state before/after tests)

   IMPORTANT: Do NOT change any application code. Assume the application code is correct. Only modify test files and snapshot baselines.

7. Update snapshot baselines if needed:

   If the flakiness is due to legitimate visual differences:

   PLAYWRIGHT_RETRIES=0 PLAYWRIGHT_HTML_OPEN=never npm run e2e -- e2e-tests/<testfile>.spec.ts --update-snapshots
   

8. Verify the fix:

   Re-run the test 10 times to confirm it's no longer flaky:

   PLAYWRIGHT_RETRIES=0 PLAYWRIGHT_HTML_OPEN=never npm run e2e -- e2e-tests/<testfile>.spec.ts --repeat-each=10
   

   The test should pass all 10 runs consistently.

9. Summarize results:

   Report to the user:

   • Which tests were identified as flaky
   • What was causing the flakiness
   • What fixes were applied
   • Verification results (all 10 runs passing)
   • Any tests that could not be fixed and need further investigation