check-ci

# Check CI Monitor GitLab CI and GitHub Actions until all jobs finish, then investigate any failures and report results. When `--commit` is used (or defaulting to HEAD), both GitLab pipelines and GitHub Actions workflow runs are monitored. GitHub monitoring requires `ddtool auth github login --org DataDog`; if unavailable, a warning is printed and only GitLab is monitored. `--pipeline <id>` is GitLab-only and skips GitHub. ## Input `$ARGUMENTS` may contain any combination of: - `--commit <ref>` — git ref to resolve (default: HEAD); monitors GitLab + GitHub - `--pipeline <id>` — specific GitLab pipeline ID (skips GitHub monitoring) - `--jobs <pat1,pat2>` — comma-separated substring patterns (case-insensitive); monitor ONLY the jobs whose name contains any pattern. In monitoring mode the run finishes as soon as the matched subset is terminal (rather than waiting for the whole pipeline); with `--list-jobs` the snapshot shows only matched jobs. Applies to both GitLab jobs and GitHub Actions workflow jobs. Omit for whole-pipeline monitoring (the default). - `--list-jobs` — quick snapshot mode (no monitoring) If no `--commit` or `--pipeline` is given, default to `--commit HEAD`. ## Quick mode — `--list-jobs` Run synchronously and exit immediately: ```bash .claude/ci/check-ci --commit <ref> --list-jobs ``` Prints all jobs grouped by pipeline (GitLab) and workflow run (GitHub Actions) with their status. Print the table to the user and stop. Do not continue to the monitoring steps. ## Full monitoring mode ### Step 1 — Start check-ci in the background ```bash PYTHONUNBUFFERED=1 .claude/ci/check-ci [OPTIONS] ``` - Use `run_in_background: true` in Bash tool invocation. Do NOT append `&` or redirect output. - The Bash tool returns immediately with an output file path like `/path/to/tasks/<id>.output` ("Output is being written to ..." in the tool invocation output). Note this path — it is required in Step 2. This file path will be referred to as `OUTPUT_FILE` henceforth. - Default options if the user provided none: `--commit HEAD`. - You may also pass `--max-failures 50` (default) and `--timeout 7200` (default, 2 h). - To watch only a named subset of jobs, add `--jobs "<pat1>,<pat2>,..."`. The run then finishes as soon as those matched jobs are terminal, instead of waiting for the whole pipeline. Example — watch the three Windows `test_c` version variants: ```bash PYTHONUNBUFFERED=1 .claude/ci/check-ci \ --jobs "windows test_c: [7.2],windows test_c: [7.3],windows test_c: [7.4]" ``` ### Step 2 — Start ci-watch in the background ```bash .claude/ci/ci-watch [--start-offset N] OUTPUT_FILE ``` - `OUTPUT_FILE` must be the output file from the check-ci task above. - Use `run_in_background: true`. - You are notified when ci-watch exits through a task notification. While it runs, you may do other work. - On exit, ci-watch always prints `RESUME_OFFSET: <N>`. Record it for re-runs. ci-watch exit codes: | Code | Meaning | |------|---------| | 0 | All pipelines completed — no failures | | 1 | One or more `FAILED:` lines detected | | 2 | Stale — no new output for 5 minutes | | 3 | check-ci timed out | ### Step 3 — Speak and act on the result **Immediately after ci-watch exits**, call `mcp__speak_when_done__speak(message="...")` (the first time, you'll need to do invoke `ToolSearch("select:mcp__speak_when_done__speak")`: - Exit 0 → "All CI jobs passed" - Exit 1 → "<N> CI jobs failed" (count with `grep "^FAILED:" OUTPUT_FILE | wc -l`) - Exit 2 or 3 → "CI monitor timed out" Then choose the appropriate action: #### All jobs passed (exit 0) Report success to the user and stop. #### Failures detected (exit 1) 1. List the failed jobs: ```bash grep "^FAILED:" OUTPUT_FILE ``` The output directory is `/tmp/gitlab_<pipeline_id>/`. Logs are at: - `fail_logs/<job_id>.log` — GitLab job traces - `gh_fail_logs/gh_<job_id>.log` — GitHub Actions job logs GitHub entries in `failure.txt` are prefixed `[GH]`. 2. Read each failure log and diagnose the root cause. Look for: - Compile errors or linker failures - Test assertion failures (include the failing test name and diff) - Infrastructure/flakiness signals (timeout, network, Docker pull failures, OOM) — mark these as flaky rather than real failures. Except you don't need to go through of them if it becomes evident it's unnecessary. 3. Report findings grouped by root cause. 4. **Fix and push only when all three conditions hold:** a. The user explicitly asked you to fix CI failures. b. You have made changes to address the failures. c. The current branch has an upstream remote branch. If any condition is missing, stop and report instead. When all three hold: commit the fix, push, then go back to Step 1 to re-monitor. If possible, before attempting a fix, try to reproduce the failure locally. Check @.claude/ci/index.md for instructions. Then attempt your fix and rerun to confirm the fix resolves the problem. #### Stale or timed out (exit 2 or 3) Re-run ci-watch with `--start-offset <RESUME_OFFSET>` (Step 2) to resume watching from where you left off. If check-ci itself has also exited, restart from Step 1. #### Keep watching (user wants to continue after investigation) Re-run ci-watch with `--start-offset <RESUME_OFFSET>` (back to Step 2). ## Downloading artifacts Use `tooling/bin/download-artifacts` to fetch build outputs from CI jobs (e.g., compiled extensions, SSI loader, datadog-setup.php). Useful when investigating a failure that produced an artifact worth inspecting locally. ## Rules - Never push unless the user explicitly asked for it. See the global instruction "Do not push to git remotes unless explicitly asked to." - Flaky jobs (known to be intermittent, unrelated to the current changes) should be noted but not treated as real failures requiring a fix. However, to confirm that a test is failure you should look for similar failures in the merge base. - `GITLAB_PERSONAL_ACCESS_TOKEN` is already set in the environment — do not re-export it. - Raw job logs can also be fetched directly. For Gitlab: ```bash curl -s -H "PRIVATE-TOKEN: $GITLAB_PERSONAL_ACCESS_TOKEN" \ "https://gitlab.ddbuild.io/api/v4/projects/355/jobs/<JOB_ID>/trace" ```