triage-ci-flake

# Triage CI Failure ## Overview Systematic workflow designed specifically for triaging and fixing end-to-end (e2e) test failures in CI, especially flaky Playwright tests that pass locally but fail in CI. E2E tests that made it to `main` are usually flaky due to timing, bundling, or environment differences. **CRITICAL RULE: You MUST run the reproduction workflow before proposing any fixes. No exceptions.** ## When to Use - CI test fails on `main` branch after PR was merged - Test passes locally but fails in CI - Test failure labeled as "flaky" or intermittent - E2E test timing out in CI only - User provides a PR URL/number with failing checks ## PR-Based Workflow (When PR URL/Number Provided) When the user provides a PR URL (e.g., `https://github.com/payloadcms/payload/pull/16701`) or PR number: ### Step 1: Fetch PR Status Checks First, use `tool_search` with query "github pull request status checks" to load the GitHub tools. Then use the `github-pull-request_pullRequestStatusChecks` tool to get all failing checks: ``` Tool: github-pull-request_pullRequestStatusChecks Parameters: pullRequestNumber: <extracted from URL or provided> repo: { owner: "payloadcms", name: "payload" } ``` ### Step 2: Aggregate Failing Tests Parse the status checks response and create a summary table: | Suite | Check Status | Target URL | | --------------------- | ------------ | ---------- | | admin**e2e**list-view | failed | [link] | | plugin-import-export | failed | [link] | For each failing check, the `context` field contains the test suite name (e.g., `admin__e2e__list-view`). ### Step 3: Extract Failure Details from Each Check For each failing check: 1. Visit the `targetUrl` to get detailed failure logs 2. Extract the specific test name and error message 3. Add to the aggregated failure list Present a consolidated summary: ```markdown ## Failing Tests Summary ### 1. admin**e2e**list-view (3/4) - **Test**: "should use custom pagination limit" - **Error**: Locator `.per-page__button` not found - **File**: test/admin/e2e/list-view/e2e.spec.ts:1495 ### 2. plugin-import-export - **Test**: "should inherit limit from list view URL" - **Error**: Locator `.per-page__button` not found - **File**: test/plugin-import-export/e2e.spec.ts:150 ``` ### Step 4: Identify Common Patterns Look for patterns across failures: - Same selector errors → likely a component change needing test updates - Same test file → localized issue - Different errors in same suite → may be test pollution ### Step 5: Proceed with Triage For each unique failure, follow the standard reproduction workflow below. ## MANDATORY First Steps **YOU MUST EXECUTE THESE COMMANDS. Reading code or analyzing logs does NOT count as reproduction.** 1. **Extract** suite name, test name, and error from CI logs 2. **EXECUTE**: Kill port 3000 to avoid conflicts 3. **EXECUTE**: `pnpm dev $SUITE_NAME` (use run_in_background=true) 4. **EXECUTE**: Wait for server to be ready (check with curl or sleep) 5. **EXECUTE**: Run the specific failing test with Playwright directly (npx playwright test test/TEST_SUITE_NAME/e2e.spec.ts:31:3 --headed -g "TEST_DESCRIPTION_TARGET_GOES_HERE") 6. **If test passes**, **EXECUTE**: `pnpm prepare-run-test-against-prod` 7. **EXECUTE**: `pnpm dev:prod $SUITE_NAME` and run test again **Only after EXECUTING these commands and seeing their output** can you proceed to analysis and fixes. **"Analysis from logs" is NOT reproduction. You must RUN the commands.** ## Core Workflow ```dot digraph triage_ci { "CI failure reported" [shape=box]; "Extract details from CI logs" [shape=box]; "Identify suite and test name" [shape=box]; "Run dev server: pnpm dev $SUITE" [shape=box]; "Run specific test by name" [shape=box]; "Did test fail?" [shape=diamond]; "Debug with dev code" [shape=box]; "Run prepare-run-test-against-prod" [shape=box]; "Run: pnpm dev:prod $SUITE" [shape=box]; "Run specific test again" [shape=box]; "Did test fail now?" [shape=diamond]; "Debug bundling issue" [shape=box]; "Unable to reproduce - check logs" [shape=box]; "Fix and verify" [shape=box]; "CI failure reported" -> "Extract details from CI logs"; "Extract details from CI logs" -> "Identify suite and test name"; "Identify suite and test name" -> "Run dev server: pnpm dev $SUITE"; "Run dev server: pnpm dev $SUITE" -> "Run specific test by name"; "Run specific test by name" -> "Did test fail?"; "Did test fail?" -> "Debug with dev code" [label="yes"]; "Did test fail?" -> "Run prepare-run-test-against-prod" [label="no"]; "Run prepare-run-test-against-prod" -> "Run: pnpm dev:prod $SUITE"; "Run: pnpm dev:prod $SUITE" -> "Run specific test again"; "Run specific test again" -> "Did test fail now?"; "Did test fail now?" -> "Debug bundling issue" [label="yes"]; "Did test fail now?" -> "Unable to reproduce - check logs" [label="no"]; "Debug with dev code" -> "Fix and verify"; "Debug bundling issue" -> "Fix and verify"; } ``` ## Step-by-Step Process ### 1. Extract CI Details From CI logs or GitHub Actions URL, identify: - **Suite name**: Directory name (e.g., `i18n`, `fields`, `lexical`) - **Test file**: Full path (e.g., `test/i18n/e2e.spec.ts`) - **Test name**: Exact test description - **Error message**: Full stack trace - **Test type**: E2E (Playwright) ### 2. Reproduce with Dev Code **CRITICAL: Always run the specific test by name, not the full suite.** **SERVER MANAGEMENT RULES:** 1. **ALWAYS kill all servers before starting a new one** 2. **NEVER assume ports are free** 3. **ALWAYS wait for server ready confirmation before running tests** ```bash # ======================================== # STEP 2A: STOP ALL SERVERS # ======================================== lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear" # ======================================== # STEP 2B: START DEV SERVER # ======================================== # Start dev server with the suite (in background with run_in_background=true) pnpm dev $SUITE_NAME # ======================================== # STEP 2C: WAIT FOR SERVER READY # ======================================== # Wait for server to be ready (REQUIRED - do not skip) until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready" # ======================================== # STEP 2D: RUN SPECIFIC TEST # ======================================== # Run ONLY the specific failing test using Playwright directly # For E2E tests (DO NOT use pnpm test:e2e as it spawns its own server): pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name" ``` **Did the test fail?** - ✅ **YES**: You reproduced it! Proceed to debug with dev code. - ❌ **NO**: Continue to step 3 (bundled code test). ### 3. Reproduce with Bundled Code If test passed with dev code, the issue is likely in bundled/production code. **IMPORTANT: You MUST stop the dev server before starting prod server.** ```bash # ======================================== # STEP 3A: STOP ALL SERVERS (INCLUDING DEV SERVER FROM STEP 2) # ======================================== lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear" # ======================================== # STEP 3B: BUILD AND PACK FOR PROD # ======================================== # Build all packages and pack them (this takes time - be patient) pnpm prepare-run-test-against-prod # ======================================== # STEP 3C: START PROD SERVER # ======================================== # Start prod dev server (in background with run_in_background=true) pnpm dev:prod $SUITE_NAME # ======================================== # STEP 3D: WAIT FOR SERVER READY # ======================================== # Wait for server to be ready (REQUIRED - do not skip) until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready" # ======================================== # STEP 3E: RUN SPECIFIC TEST # ======================================== # Run the specific test again using Playwright directly pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name" ``` **Did the test fail now?** - ✅ **YES**: Bundling or production build issue. Look for: - Missing exports in package.json - Build configuration problems - Code that behaves differently when bundled - ❌ **NO**: Unable to reproduce locally. Proceed to step 4. ### 4. Unable to Reproduce If you cannot reproduce locally after both attempts: - Review CI logs more carefully for environment differences - Check for race conditions (run test multiple times: `for i in {1..10}; do pnpm test:e2e...; done`) - Look for CI-specific constraints (memory, CPU, timing) - Consider if it's a true race condition that's highly timing-dependent ## Common Flaky Test Patterns ### Race Conditions - Page navigating while assertions run - Network requests not settled before assertions - State updates not completed