| name | pr-ci-failure-triage |
| description | Analyze all CI failures in a Pull Request and categorize them as regressions, pre-existing flaky tests, infrastructure issues, or cascade failures. |
Skill: PR CI Failure Triage
Purpose
Analyze all CI failures in a Pull Request and categorize them:
- Failures caused by the PR (regressions)
- Pre-existing flaky tests (unrelated to PR)
- Infrastructure issues (transient, unrelated)
- Cascade failures (caused by earlier test crashing the server)
This skill provides a high-level triage of PR CI status and creates summary comments.
Identifying CI Failures
CI Report URL
PR runs:
https://s3.amazonaws.com/altinity-build-artifacts/PRs/<PR>/<SHA>/<RUN_ID>/ci_run_report.html
Branch/REF runs (e.g., antalya-25.8, v25.8.16.20001.altinityantalya):
https://s3.amazonaws.com/altinity-build-artifacts/REFs/<BRANCH>/<SHA>/<RUN_ID>/ci_run_report.html
JSON Log Browser
PR runs:
https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=<PR>&sha=<SHA>&name_0=PR&name_1=<URL_ENCODED_CHECK_NAME>
Branch/REF runs — note name_0=MasterCI and REF= instead of PR=:
https://altinity-build-artifacts.s3.amazonaws.com/json.html?REF=<BRANCH>&sha=<SHA>&name_0=MasterCI&name_1=<URL_ENCODED_CHECK_NAME>
Step 1: Get All Failing Tests
Query the CI database to get all failures for the PR:
SELECT check_name, test_name, test_status, check_start_time
FROM `gh-data`.checks
WHERE pull_request_number = <PR_NUMBER>
AND test_status = 'FAIL'
ORDER BY check_name, test_name
Step 2: Identify Cascade Failures
When a test crashes the server (SIGABRT), subsequent tests fail as cascade effects.
Cascade Failure Indicators
These test names indicate cascade failures, not root causes:
| Test Name Pattern | Meaning |
|---|
Fatal messages (in clickhouse-server.log...) | Server crash detected |
Killed by signal (in clickhouse-server.log...) | SIGABRT/SIGSEGV detected |
Exception in test runner | Test runner failed |
Server died | Server process terminated |
Sanitizer assert (in stderr.log) | Sanitizer detected issue |
Identifying Root Cause
When cascade failures are present:
- Find the earliest failing test in that job (by timestamp or test order)
- Check if it's a test with actual assertions, not a meta-check
- The root cause test typically runs before the cascade failures
Step 3: Categorize Each Failure
For each unique failing test (excluding cascade indicators), determine category:
Category 1: Pre-existing Flaky Test
Query historical failures:
SELECT test_name, pull_request_number, check_start_time
FROM `gh-data`.checks
WHERE test_name = '<TEST_NAME>'
AND test_status = 'FAIL'
ORDER BY check_start_time DESC
LIMIT 20
Indicators:
- Failures exist across multiple unrelated PRs
- Failures predate the current PR by weeks/months
- Failures occur on PR=0 (master branch)
Category 2: Infrastructure Issue
Indicators:
- Error messages about Docker, networking, timeouts
Connection refused, Container failed to start- Passes on rerun
- Affects unrelated test types
Category 3: PR-Caused Regression
Indicators:
- Test was passing before this PR
- Failure pattern matches PR changes (e.g., JSON tests fail after JSON-related change)
- Consistent failure on specific build types
Step 4: Debug vs Release Build Differences
Different from sanitizer builds:
| Build Type | Behavior |
|---|
Debug (amd_debug) | Assertions enabled, LOGICAL_ERROR causes SIGABRT crash |
Release (amd_binary, arm_binary) | No assertions, errors may be silent or return incorrect data |
Pattern Recognition
| Observation | Likely Cause |
|---|
| Fails on debug, passes on release | Assertion catches bug that release ignores |
| Fails on release, passes on debug | Performance/timing issue |
| Fails on both | Fundamental bug |
Step 5: Check GitHub Job Status vs Database
Jobs may show different status on GitHub vs database due to reruns:
gh api repos/Altinity/ClickHouse/actions/runs/<RUN_ID>/jobs \
--jq '.jobs[] | select(.name | contains("<JOB_NAME>")) | {name, conclusion}'
Discrepancy causes:
- Partial rerun only ran sanity checks (not actual tests)
- GitHub shows latest run status, database has all runs
Step 6: Investigate Root Cause Tests
For each test identified as potentially PR-caused, use the appropriate skill:
| Test Type | Skill to Use |
|---|
Stateless tests (NNNNN_test_name) | upstream-test-investigation |
Integration tests (test_*/test.py::test_*) | upstream-test-investigation |
Regression tests (/suite/subsuite/test) | regression-test-database-investigation |
Then:
- Check if test is related to PR changes
- Verify local reproduction if needed
Step 7: Create PR Summary Comment
After categorization, create a structured comment:
## CI Failures Analysis
### Related to this PR
- `<test_name>`: <brief description of failure>
- ...
All other failures in same job (`Fatal messages`, `Killed by signal`, etc.)
are cascade failures from server crash.
### Pre-existing Flaky Tests (Unrelated)
- `<test_name>` - flaky since <date>
- ...
### Infrastructure Issues (Unrelated)
- `<description>` - passed on rerun
### Issue/Fix References
- **Issue created:** #<NUMBER>
- **Fix PR:** #<NUMBER>
Standard Output Format
## PR #<NUMBER> CI Triage
### Summary
| Category | Count | Tests |
|----------|-------|-------|
| PR-caused regression | N | test1, test2 |
| Cascade failures | N | (from server crash) |
| Pre-existing flaky | N | test3, test4 |
| Infrastructure | N | description |
### Root Cause Analysis
<For each PR-caused regression, brief analysis>
### Recommendations
1. <action item>
2. <action item>
Related Skills
upstream-test-investigation - Deep investigation of upstream test (Stateless/Integration/Fuzzer/Stress)upstream-ci-database-queries - CI database query reference (Altinity + upstream)regression-test-database-investigation - Altinity regression test database investigationgithub-issue-template - Templates for writing GitHub issues after investigation
Notes
- Always check if failures predate the PR before attributing them to PR changes
- Server crashes (SIGABRT) on debug builds often indicate assertion failures
- A single root cause test can cause dozens of cascade failures
- Rerun data may mask original failures in GitHub UI
