菜单
Use when CI tests fail on main branch after PR merge, when investigating flaky test failures, or when user provides a PR URL/number to aggregate all failing tests
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/payloadcms/payload --skill triage-ci-flake命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
基于 SOC 职业分类
正在显示 SKILL.md
| name | triage-ci-flake |
| description | Use when CI tests fail on main branch after PR merge, when investigating flaky test failures, or when user provides a PR URL/number to aggregate all failing tests |
| allowed-tools | Write, Bash(date:*), Bash(mkdir -p *) |
Systematic workflow designed specifically for triaging and fixing end-to-end (e2e) test failures in CI, especially flaky Playwright tests that pass locally but fail in CI. E2E tests that made it to main are usually flaky due to timing, bundling, or environment differences.
CRITICAL RULE: You MUST run the reproduction workflow before proposing any fixes. No exceptions.
main branch after PR was mergedWhen the user provides a PR URL (e.g., https://github.com/payloadcms/payload/pull/16701) or PR number:
First, use tool_search with query "github pull request status checks" to load the GitHub tools.
Then use the github-pull-request_pullRequestStatusChecks tool to get all failing checks:
Tool: github-pull-request_pullRequestStatusChecks
Parameters:
pullRequestNumber: <extracted from URL or provided>
repo: { owner: "payloadcms", name: "payload" }
Parse the status checks response and create a summary table:
| Suite | Check Status | Target URL |
|---|---|---|
| admine2elist-view | failed | [link] |
| plugin-import-export | failed | [link] |
For each failing check, the context field contains the test suite name (e.g., admin__e2e__list-view).
For each failing check:
targetUrl to get detailed failure logsPresent a consolidated summary:
## Failing Tests Summary
### 1. admin**e2e**list-view (3/4)
- **Test**: "should use custom pagination limit"
- **Error**: Locator `.per-page__button` not found
- **File**: test/admin/e2e/list-view/e2e.spec.ts:1495
### 2. plugin-import-export
- **Test**: "should inherit limit from list view URL"
- **Error**: Locator `.per-page__button` not found
- **File**: test/plugin-import-export/e2e.spec.ts:150
Look for patterns across failures:
For each unique failure, follow the standard reproduction workflow below.
YOU MUST EXECUTE THESE COMMANDS. Reading code or analyzing logs does NOT count as reproduction.
pnpm dev $SUITE_NAME (use run_in_background=true)pnpm prepare-run-test-against-prodpnpm dev:prod $SUITE_NAME and run test againOnly after EXECUTING these commands and seeing their output can you proceed to analysis and fixes.
"Analysis from logs" is NOT reproduction. You must RUN the commands.
digraph triage_ci {
"CI failure reported" [shape=box];
"Extract details from CI logs" [shape=box];
"Identify suite and test name" [shape=box];
"Run dev server: pnpm dev $SUITE" [shape=box];
"Run specific test by name" [shape=box];
"Did test fail?" [shape=diamond];
"Debug with dev code" [shape=box];
"Run prepare-run-test-against-prod" [shape=box];
"Run: pnpm dev:prod $SUITE" [shape=box];
"Run specific test again" [shape=box];
"Did test fail now?" [shape=diamond];
"Debug bundling issue" [shape=box];
"Unable to reproduce - check logs" [shape=box];
"Fix and verify" [shape=box];
"CI failure reported" -> "Extract details from CI logs";
"Extract details from CI logs" -> "Identify suite and test name";
"Identify suite and test name" -> "Run dev server: pnpm dev $SUITE";
"Run dev server: pnpm dev $SUITE" -> "Run specific test by name";
"Run specific test by name" -> "Did test fail?";
"Did test fail?" -> "Debug with dev code" [label="yes"];
"Did test fail?" -> "Run prepare-run-test-against-prod" [label="no"];
"Run prepare-run-test-against-prod" -> "Run: pnpm dev:prod $SUITE";
"Run: pnpm dev:prod $SUITE" -> "Run specific test again";
"Run specific test again" -> "Did test fail now?";
"Did test fail now?" -> "Debug bundling issue" [label="yes"];
"Did test fail now?" -> "Unable to reproduce - check logs" [label="no"];
"Debug with dev code" -> "Fix and verify";
"Debug bundling issue" -> "Fix and verify";
}
From CI logs or GitHub Actions URL, identify:
i18n, fields, lexical)test/i18n/e2e.spec.ts)CRITICAL: Always run the specific test by name, not the full suite.
SERVER MANAGEMENT RULES:
# ========================================
# STEP 2A: STOP ALL SERVERS
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"
# ========================================
# STEP 2B: START DEV SERVER
# ========================================
# Start dev server with the suite (in background with run_in_background=true)
pnpm dev $SUITE_NAME
# ========================================
# STEP 2C: WAIT FOR SERVER READY
# ========================================
# Wait for server to be ready (REQUIRED - do not skip)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"
# ========================================
# STEP 2D: RUN SPECIFIC TEST
# ========================================
# Run ONLY the specific failing test using Playwright directly
# For E2E tests (DO NOT use pnpm test:e2e as it spawns its own server):
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name"
Did the test fail?
If test passed with dev code, the issue is likely in bundled/production code.
IMPORTANT: You MUST stop the dev server before starting prod server.
# ========================================
# STEP 3A: STOP ALL SERVERS (INCLUDING DEV SERVER FROM STEP 2)
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"
# ========================================
# STEP 3B: BUILD AND PACK FOR PROD
# ========================================
# Build all packages and pack them (this takes time - be patient)
pnpm prepare-run-test-against-prod
# ========================================
# STEP 3C: START PROD SERVER
# ========================================
# Start prod dev server (in background with run_in_background=true)
pnpm dev:prod $SUITE_NAME
# ========================================
# STEP 3D: WAIT FOR SERVER READY
# ========================================
# Wait for server to be ready (REQUIRED - do not skip)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"
# ========================================
# STEP 3E: RUN SPECIFIC TEST
# ========================================
# Run the specific test again using Playwright directly
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name"
Did the test fail now?
If you cannot reproduce locally after both attempts:
for i in {1..10}; do pnpm test:e2e...; done)Fix patterns:
toBeVisible(), toHaveText())waitForFunction() with condition checksafterEachFix patterns:
afterEachdeleteAll that affects other testssetTimeout/sleep instead of condition-based waitingFix patterns:
waitForPageStability() helperWhen fixing e2e tests, be aware of these eslint rules:
playwright/no-networkidle - Avoid waitForLoadState('networkidle') (use condition-based waiting instead)payload/no-wait-function - Avoid custom wait() functions (use Playwright's built-in waits)payload/no-flaky-assertions - Avoid non-retryable assertionsplaywright/prefer-web-first-assertions - Use built-in Playwright assertionsExisting code may violate these rules - when adding new code, follow the rules even if existing code doesn't.
After fixing:
# Ensure dev server is running on port 3000
# Run test multiple times to confirm stability
for i in {1..10}; do
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name" || break
done
# Run full suite
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts
# If you modified bundled code, test with prod build
lsof -ti:3000 | xargs kill -9 2>/dev/null
pnpm prepare-run-test-against-prod
pnpm dev:prod $SUITE_NAME
until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts
NO FIX WITHOUT REPRODUCTION FIRST
If you propose a fix before completing steps 1-3 of the workflow, you've violated this skill.
This applies even when:
No exceptions. Run the reproduction workflow first.
Every excuse for skipping reproduction, and why it's wrong:
| Rationalization | Reality |
|---|---|
| "The logs show the exact error" | Logs show symptoms, not root cause. Reproduce. |
| "I can see the problem in the code" | You're guessing. Reproduce to confirm. |
| "This is obviously a race condition" | Maybe. Reproduce to be sure. |
| "I've seen this error before" | This might be different. Reproduce. |
| "The stack trace is clear" | Stack trace shows where, not why. Reproduce. |
| "Time pressure - need to fix fast" | Reproducing IS fast. Guessing wastes time. |
| "The test file shows the issue" | Reading ≠ running. Execute the commands. |
| "I'll analyze the code first" | Code analysis comes AFTER reproduction. |
| "Let me investigate the root cause" | Reproduction IS the investigation. |
| "I need to understand the error" | Understanding comes from seeing it fail. |
These thoughts mean you're about to violate the skill:
All of these mean: Stop immediately. Execute the reproduction commands. No analysis without execution.
CORRECT approach - executes commands first:
User: CI failed on i18n suite - https://github.com/.../actions/runs/XXX
Agent:
1. Extracted details from CI logs:
- Suite: i18n
- Test: "ensure translations update correctly when switching language"
- File: test/i18n/e2e.spec.ts:124
- Error: "Error loading language: en" TypeError: network error
2. Executing reproduction workflow:
[Uses Bash tool to kill ports]
$ lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"
[Uses Bash tool with run_in_background=true]
$ pnpm dev i18n
[Waits for server to start with curl check]
$ until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done && echo "Server ready"
[Uses Bash tool to run Playwright directly]
$ pnpm exec playwright test test/i18n/e2e.spec.ts -g "ensure translations update correctly when switching language"
Output shows: Test PASSED locally
3. Test passed with dev code, trying bundled code:
[Kills ports again]
$ lsof -ti:3000 | xargs kill -9 2>/dev/null
[Uses Bash tool]
$ pnpm prepare-run-test-against-prod
[Uses Bash tool with run_in_background=true]
$ pnpm dev:prod i18n
[Waits for server]
$ until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done
[Uses Bash tool]
$ pnpm exec playwright test test/i18n/e2e.spec.ts -g "ensure translations update correctly when switching language"
Output shows: Test FAILED - reproduced!
4. Now that I've reproduced it, analyzing the failure...
[Code analysis, root cause identification]
5. Implementing fix: [specific changes]
6. Verification:
[Uses Bash tool]
$ for i in {1..10}; do pnpm test:e2e i18n -g "exact test"; done
All 10 runs passed!
INCORRECT approach - analyzes without executing:
User: CI failed on i18n suite
Agent: Looking at the error logs, I can see this is a race condition...
[Analyzes code, proposes fix without running any commands]
❌ WRONG - This violates the skill. Must execute reproduction commands first.
| Mistake | Fix |
|---|---|
| Running full test suite first | Run specific test by name |
| Skipping dev code reproduction | Always try dev code first |
| Not testing with bundled code | If dev passes, test with prepare-run-test-against-prod |
| Proposing fix without reproducing | Follow the workflow - reproduce first |
Using networkidle in new code | Use condition-based waiting with waitForFunction() |
Adding arbitrary wait() calls | Use Playwright's built-in assertions and waits |
After you have:
You MUST prompt the user to create a PR:
The fix has been verified and is ready for review. Would you like me to create a PR with these changes?
Summary of changes:
- [List files modified]
- [Brief description of the fix]
- [Verification results]
IMPORTANT:
This ensures the user has visibility and control over what gets submitted for review.