investigate-ci

# Investigate CI Failure Skill A read-only first pass over a CI failure: turn a single URL into a per-test verdict (flaky vs real) plus a root-cause hypothesis, with no copy-paste and no manual `wget`. ## Arguments - `$0` (required): one of - a GitHub PR URL (`https://github.com/ClickHouse/ClickHouse/pull/NNNNN`), - a direct S3/CI report URL (`https://s3.amazonaws.com/.../json.html?PR=...&sha=...`), or - a GitHub **issue** URL (`https://github.com/ClickHouse/ClickHouse/issues/NNNNN`) — typically a bot-generated `flaky test` issue. Resolved to its report URL in step 0. - `$1` (optional): master-history window in days for the flaky-vs-real verdict. Default `14`. ## Hard rules - **Read-only.** Never `git commit`, `git push`, edit source, switch branches, close issues, or comment on the PR. This skill diagnoses; the human decides what to do. Surface findings, do not act on them. To read source at the report's commit, use `git show <sha>:<path>` rather than checking it out; only fetch/switch after asking the user (see step 4). - Use `tmp/investigate/<sha>/` for all working files, never `/tmp` (per CLAUDE.md). The `<sha>` is the first 7 characters of the report commit SHA — one subdirectory per investigation so artifacts from different PRs or branches never collide and you can return to an earlier investigation without re-downloading. - **Run every `gh` read through `.claude/tools/gh-ro.sh` (same args as `gh`).** It drops a poisoned `GH_CONFIG_DIR` — some agent/CI runners set it to a config dir with no working auth, which makes raw `gh` fail — and refuses any non-read-only subcommand, so it can never create/close/edit/merge/ comment. The examples below use it for this reason. (`fetch_ci_report.js` clears `GH_CONFIG_DIR` internally, so drive it as `node .claude/tools/fetch_ci_report.js …` directly.) - Wrap test names, identifiers, and log excerpts in backticks per the project style rule. - Say "exception", not "crash", for logical errors. ## Steps ### 0. Resolve an issue URL to a report URL `fetch_ci_report.js` accepts only PR, S3 `json.html`, and direct `result_*.json` URLs — **not** issue links. If `$0` is `.../issues/NNNNN`, read the issue body and extract the report URL first: ```bash .claude/tools/gh-ro.sh issue view <NNNNN> --repo ClickHouse/ClickHouse --json title,body ``` Read the issue from the command output — do **not** redirect to a file. A `.claude/tools/gh-ro.sh issue view … > tmp/investigate/$SHA/issue.json` redirect is a file write that rides the wildcard `Bash(.claude/tools/gh-ro.sh:*)` allow (not the hook), so a symlinked `tmp`/`tmp/investigate` could land it outside the scratch dir without a prompt. (Step 1 creates `tmp/investigate/$SHA`.) Bot-generated `flaky test` issues use this body format: ``` Test name: <test_name> Failure reason: <reason> CI report: <S3 json.html url> ← use this as the report URL for the steps below Failing test history: <play.clickhouse.com link> ``` Extract the `CI report:` URL and use it as `$0` for the rest of the skill. Also keep the `Test name:` value — for a flaky-test issue the test to classify is already named, so step 3 can run even if the S3 artifacts have expired (CI report links are retained only for a limited time; if the report 404s, fall back to the named test plus the issue's `Failing test history` link). If the issue is **not** a flaky-test report (no `CI report:` line), ask the user for the report or PR URL rather than guessing. **When the input is a flaky-test issue, step 3 is the decisive signal — run it first.** The issue already names the test and the bot only files it after a failure, so master history settles flaky-vs-real directly. Treat step 1 (the S3 report) and the optional step-4 artifact download as best-effort enrichment: bot-filed issues are often weeks old and their artifacts have already expired, so do not block on them. ### 1. Set up and fetch the failed tests `fetch_ci_report.js` needs `node` on `PATH`. Run these as **separate** commands (not one compound block) so each matches an allowed shape under the investigate profile — a combined `mkdir … ; if … node …` string matches neither the exact `mkdir` allow nor the node-fetch hook and would prompt. Primary inputs (PR/S3) skip step 0, so create the parent working dir first: ```bash mkdir -p tmp/investigate ``` Probe for `node` (`command -v` is allowed; do not wrap the fetch in an `if`/`;` one-liner, which would hide the `node` command from the hook): ```bash command -v node ``` If `node` is present, fetch the failed tests and their output: ```bash node .claude/tools/fetch_ci_report.js "$0" --failed --cidb 2>&1 ``` For a **single HTML report URL** (including `sha=latest`, which resolves to the actual build commit) the tool prints a `SHA: <40-hex-sha>` line — read it and set `$SHA` to the first 7 characters. For a **PR URL** the tool prints a multi-report summary without a `SHA:` line; extract `$SHA` from the `?sha=<hex>` query-string parameter in any `🔗 Report:` URL printed in the summary, or re-run with `--report N` on the relevant job to get single-report output that does print `SHA:`. For a **direct `result_*.json` S3 URL** (e.g. `https://s3.amazonaws.com/clickhouse-test-reports/PRs/111528/<sha>/result_fast_test_arm_darwin.json`) the tool also prints a `SHA:` line extracted from the URL path. Either way, `$SHA` is always a concrete commit hash, never a PR-number fallback. Then create the working directory: ```bash mkdir -p tmp/investigate/$SHA ``` This prints the failed tests **and their output** straight from the praktika `result_*.json` (no copy-paste), with a CIDB link per failed test. Read it from the command output — do **not** add a `> tmp/investigate/$SHA/…` redirect (a redirect is a file write the hook won't auto-approve, since it can't be made symlink-safe, so it would prompt); the harness persists large output to a file you can re-read or `grep`. **If `node` is absent, the fallback depends on the input type:** - **Issue URL** (step 0 already gave you `Test name:` and the `Failing test history` link) → proceed without the report: run step 3 on that named test; the S3 report and step-4 artifacts are best-effort enrichment. A missing `node` is not fatal here. - **PR or S3 report URL** → `node` is **required**. Without the report you have no failed test names, job names, or labels, so steps 2–3 (issue/fix search and the `test_name IN (...)` history query) cannot run. Do **not** limp on with a partial investigation — stop and tell the user to install `node` (or re-run where `node` is on `PATH`). Per failure the tool prints a `🏷️ labels:` line (CI's non-CIDB labels — the `issue` match link and flags like `retry_ok`; see step 2a), the CIDB link, and the **failure reason section** extracted from `result.info` as follows: the bash debug-trace section (`.debuglog:` path header or `+ [timestamp]` xtrace lines) is stripped entirely as pure noise; then from what remains, up to 40 lines are shown as **head + tail** (first 20, `--- (N lines omitted) ---`, last 20), so the `Reason: ...` at the top of stateless-test output and the `ninja: build stopped` / compiler errors at the bottom of build logs are both visible. If the meaningful section is ≤ 40 lines, it appears in full. CI's matchers test `Failure reason` against the **whole** `result.info`; when the full output matters (e.g. an issue with a `Failure reason:` field deep in the trace), drill via the CIDB link or the full artifact (step 4). - If `$0` is a PR URL with many reports and the noise is high, narrow with `--report <n>` after listing reports (run the tool with no `--failed` to see the index). - Record the PR number and the exact failed **test names** as they appear — the names must match `checks.test_name` for step 3 (and feed the issue search in step 2). - **Job-level failures** are printed as `⚙️ JOB: <job name>` (instead of `❌ FAIL:`). These are synthetic entries — the job name is **not** a real `checks.test_name` value. Skip the `checks.test_name` history lookup in step 3 for them; go directly to step 4 (artifact download) to find the root cause from logs and harness output. - Record the **count** of failed tests. The cheap steps (2–3) always run over all of them, but a large count changes how step 3 scopes the expensive deep-dive — see "Scope the deep-dive". ### 2. Search for an existing tracking issue and an existing fix For **every** failed test, run two searches and record a per-test answer to two questions that go into the final report: - **Issue:** is the failure already tracked, or does an issue still need to be created? - **Fix:** does a fix already exist, and what is its status (WIP / merged / already in this branch or not)? Do both before the deep-dive — a tracked failure with a merged fix often short-circuits the rest of the investigation. #### 2a. Existing tracking issue + "does an issue need to be created?" Search issues (open **and** closed) by test name — a hit often names the tracking flaky-test issue, and its comments may already carry the root cause, a fix PR, or a "known flaky" note. ```bash .claude/tools/gh-ro.sh issue list --repo ClickHouse/ClickHouse --state all --limit 10 \ --search "<distinctive test-name fragment> in:title,body" \ --json number,title,state,stateReason,url,labels,closedAt ``` Search on a distinctive fragment (the function or `test_*` name **without** the parametrization suffix), not the full parametrized string — GitHub search tokenizes on punctuation and the full name rarely matches. If a candidate looks relevant, read it **with its comments** — they often already provide the diagnosis: ```bash .claude/tools/gh-ro.sh issue view <NNNNN> --repo ClickHouse/ClickHouse \ --json number,title,state,stateReason,body,comments,labels,closedAt ``` **How CI decides a failure is already tracked** (so you can answer "needs an issue?" the same way CI's matcher does — see `ci/praktika/issue.py`, `Issue.check_result`): CI builds a catalog from issues labeled **`testing`** (`IssueLabels.CI_ISSUE`) that are **open, or were closed within the last ~8 hours**, and routes each by whether it also carries the **`infrastructure`** label: - **Flaky-test issues** (no `infrastructure` label) → `Issue._check_flaky_test_match`. Matches when the issue's `Test name:` body field is a **suffix** of the failing test's name (`result.name.endswith(test_name)`; pytest parametrization/module rules apply), **and** if the issue sets a `Failure reason:`, that text is a **substring** of the failure output. - **Infrastructure issues** (`testing` **+** `infrastructure` label) → `Issue._check_infrastructure_match`. These do **not** match by `Test name:`. They match a failure when **all** of the present fields hold: `Failure reason:` is a substring of the output; every `Failure flags:` value (e.g. `retry_ok`) is a label on the result; `Test pattern:` matches the test name; and `Job pattern:` matches the **job** name. Note the pattern matching is **not** true SQL `LIKE`: the pattern is split on `%`, empty fragments are dropped, and it matches if **any** remaining fragment is a plain substring (so it is OR-across-fragments, order is not enforced, and a bare `%` — like an empty field — is treated as no constraint). Examples: [#87123](https://github.com/ClickHouse/ClickHouse/issues/87123) (`Job pattern: Unit%`), [#91410](https://github.com/ClickHouse/ClickHouse/issues/91410), [#92089](https://github.com/ClickHouse/ClickHouse/issues/92089) (`Job pattern: Stateless tests (amd_msan%`, `Failure reason: DB::Exception: Timeout exceeded`). **Fast path — read the labels `fetch_ci_report.js` prints.** The tool surfaces each failure's non-CIDB labels on a `🏷️ labels:` line. Two are decisive: - An **`issue`** label gives you the matched issue number **for free** — the printed link is the issue CI matched **at run time** (e.g. `Server died` → `issue (…/issues/107487)`, `Hung check …` → `…/107941`). It saves the *search*, but it is **not** by itself `tracked #N`: the label reflects the catalog when the report was produced, not the issue's state **now**. Always `.claude/tools/gh-ro.sh issue view` the linked issue and classify from its **current** `state`/`closedAt` — an issue closed after the run and aged past the ~8 h window is `stale #N` (reopen candidate), not `tracked`. The label shortcuts the lookup; it does not replace the tracked-vs-stale decision below. - **Failure flags** (e.g. `retry_ok`) appear here too — these are exactly the labels an infrastructure issue matches on via `Failure flags:` (below), so this line is how you verify that constraint. And **absence** of an `issue` label is **not** proof of "untracked": CI stamps it from the catalog *as it was at that run* (open + closed-within-8h then), so a tracking issue filed or reopened **after** the run won't show. So: an `issue` label → look up that issue and classify by current state; no label → still run the GitHub search below before concluding `needs issue`/`untracked`. **For an `INFRA/BUILD` or timeout/harness-level failure, also run the infrastructure path.** A test-name search alone will miss these, so a pre-existing, already-tracked infra failure would be mis-reported as `needs issue`. List the infra issues — **`--state all`**, because CI's catalog includes not just open ones but every `testing` issue closed in the last ~8 h (`TestCaseIssueCatalog.from_gh`), and a just-closed infra issue is still auto-matched — and match by `Job pattern` / `Failure reason` against the failing job and its output: ```bash .claude/tools/gh-ro.sh issue list --repo ClickHouse/ClickHouse --state all --label testing --label infrastructure \ --limit 100 --json number,title,url,body,state,closedAt ``` Compare each candidate's fields the way `_check_infrastructure_match` does: `Failure reason:` is a substring of the output; `Job pattern:`/`Test pattern:` match the failing job/test name; and every `Failure flags:` value must appear on the failure's `🏷️ labels:` line from step 1 (that line is the only place these flags are visible — e.g. `test_dns_cache … → retry_ok`). All present fields must hold. A match on an open issue — or one closed within ~8 h (check `closedAt`) — is `tracked #N` exactly as CI would attribute it; a match on an issue closed longer ago is a `stale #N` reopen candidate (same same-failure/recurrence check as the flaky case). **Always run the issue search for every failed name** — it is one cheap `gh issue list` and is the only way to mirror CI's attribution. Generic, harness-level names get tracked too: `Server died`, `Hung check failed, possible deadlock found`, and the upgrade `Error message in clickhouse-server.log` check frequently *do* have a `testing` issue (e.g. `Server died` → [#107487](https://github.com/ClickHouse/ClickHouse/issues/107487), `Test name: Server died`), and `Issue._check_flaky_test_match` will mark such a result with the `issue` label. So do **not** skip the search for them. The `generic failure / untracked` value is **only** for a generic bucket or anonymized error *class* (e.g. `Logical error: Bad cast from type A to B`, where the harness replaces concrete types with `A`/`B` and groups by stack hash `STID`) **after** the search finds no matching `testing` issue — because filing a *new* per-failure issue for such a shifting bucket makes no sense. It is a "searched, nothing matched, and not worth filing" verdict, never a "didn't look" one. Determine, per test: - **Tracked** — a `testing` issue matches (by the rule above, or reached via the report's `issue` label) and is **currently** open or closed within ~8 h (`.claude/tools/gh-ro.sh issue view` → `state`/`closedAt`). No new issue needed; CI will keep auto-matching it. Applies to generic-bucket names too when such an issue exists (e.g. `Server died` → #107487). If the linked/ matched issue is closed longer ago, it is `stale #N`, not `tracked`. - **Needs an issue** — the failure is a pre-existing **FLAKY** or **INFRA/BUILD** problem (per