mz-debug-ci

name	mz-debug-ci
description	Investigate CI failures on PR via gh + bk CLI. Trigger: failing checks, Buildkite failures, CI issues — "why is CI red", "build broken", "checks failing", "what went wrong in CI", "nightly broke", "tests failing on this PR", or pasted Buildkite URL. Also PR number + why failing.
argument-hint	<PR number or GitHub PR URL>

Investigate CI failures for a Materialize PR.

Prerequisites

This skill requires both gh (GitHub CLI) and bk (Buildkite CLI) to be installed and authenticated. Before doing anything else, verify both:

which gh && gh auth status
which bk && bk auth status

If either tool is missing or unauthenticated, stop immediately and tell the user what to fix (bk configure or bk auth login for Buildkite). Do not attempt to use the REST API directly or any other workaround — this workflow only works with these CLI tools.

Both gh and bk make network requests that are blocked by the default sandbox. All Bash commands in this workflow must use dangerouslyDisableSandbox: true.

Search existing failures by pattern

Search for already recorded failures in CI with a short, stable error substring:

bin/ci-failures 'foobar'

Use --search for the global search. Narrow results with --branch main, --version, --issue, --test, --build, --start-date, or --end-date. The command explains how to create a token when needed.

Step 1: Extract PR number

Parse $ARGUMENTS to get the PR number. Handle both formats:

• Plain number: 35192
• Full URL: https://github.com/MaterializeInc/materialize/pull/35192

Step 2: Find the build

Use gh to get the PR's branch name and then find the Buildkite build:

# Get the branch name for the PR
gh pr view <PR_NUMBER> --json headRefName --jq .headRefName

Alternatively, list failing checks directly:

gh pr checks <PR_NUMBER> 2>&1

Lines containing fail have tab-separated fields:

name	fail	0	https://buildkite.com/materialize/<PIPELINE>/builds/<BUILD>#<JOB_ID>	description

Extract from the URL:

• Pipeline: path segment after materialize/ (usually test)
• Build number: the number after builds/
• Job ID: the UUID after #

Step 3: Check annotations first

Before diving into logs, fetch the build annotations. They contain pre-extracted error messages, stack traces, and links to known flaky test issues — this saves significant time compared to grepping through raw logs.

bk api /pipelines/<PIPELINE>/builds/<BUILD_NUMBER>/annotations --no-pager 2>/dev/null

Note that bk can't be piped through 2>&1.

The response is JSON. Each annotation has:

• style: "error" for failures
• body_html: HTML containing the error summary, including:
  • The specific test/job that failed
  • The actual error message or stack trace in <pre><code> blocks
  • Links to known flaky test issues (look for GitHub issue links like database-issues/#NNNN)
  • Main branch history showing if this test passes on main (flaky test indicator)

Parse the error annotations to get a quick overview of all failures before fetching any logs.

Step 4: Fetch logs when needed

Only fetch full logs when annotations don't provide enough detail. Triage in this order:

1. clippy — compilation/lint errors that often explain everything
2. lint-and-rustfmt — formatting and lint-check failures
3. cargo-test — unit/integration test failures
4. fast-sql-logic-tests — SLT failures
5. testdrive — integration test failures (often cascading)
6. Everything else (checks-parallel, cluster-tests, dbt, etc.)

To fetch a job's log:

bk job log <JOB_ID> -p <PIPELINE> -b <BUILD_NUMBER> --no-timestamps --no-pager 2>/dev/null | tail -100

For large logs, first grep for errors to find the relevant section:

bk job log <JOB_ID> -p <PIPELINE> -b <BUILD_NUMBER> --no-timestamps --no-pager 2>/dev/null | grep -B2 -A5 'error\|FAIL\|panicked'

Fetch multiple job logs in parallel when they are independent (e.g., clippy + lint at the same time).

Artifacts

Jobs upload artifacts (junit XML, service logs, coredumps, ...). Use the bk artifacts subcommands:

# All artifacts of a build (each entry has id, job_id, path)
bk artifacts list <BUILD_NUMBER> -p <PIPELINE> --no-pager 2>/dev/null
# Artifacts of a single job
bk artifacts list <BUILD_NUMBER> -p <PIPELINE> --job-uuid <JOB_ID> --no-pager 2>/dev/null
# Download one artifact into the current directory
bk artifacts download <ARTIFACT_ID> --build <BUILD_NUMBER> -p <PIPELINE>

Do not use bk api for artifacts. The plural /pipelines/<PIPELINE>/builds/<BUILD_NUMBER>/artifacts endpoint works but silently returns only the first 30 entries, and its .../download endpoint fails to parse.

Shard contents

Sharded jobs (SLT, testdrive, platform checks, feature benchmark, ...) record the workflows/files/scenarios they ran in build meta-data. bin/ci-shards shows it:

# Mapping of every sharded job to what it ran, with a link per job whose
# `#` fragment is the `<JOB_ID>` for `bk job log` (omit links with --no-url)
bin/ci-shards https://buildkite.com/materialize/<PIPELINE>/builds/<BUILD_NUMBER>

# What a single job ran (job link as copied from the Buildkite UI)
bin/ci-shards 'https://buildkite.com/materialize/<PIPELINE>/builds/<BUILD_NUMBER>#<JOB_ID>'

# Which job(s) ran a specific item (test file, scenario, or workflow,
# `workflow_foo_bar` function names are matched as `foo-bar`)
bin/ci-shards https://buildkite.com/materialize/<PIPELINE>/builds/<BUILD_NUMBER> numeric.td

Step 5: Categorize failures

Use these Materialize-specific patterns to diagnose:

Clippy errors

Code lint issues in changed files. Common ones: as_conversions, needless_borrow, clone_on_ref_ptr. Fix the code, not the lint config.

check-test-flags lint failure

A new configuration flag was introduced but not registered in the required places:

• misc/python/materialize/parallel_workload/action.py (FlipFlagsAction)
• misc/python/materialize/mzcompose/__init__.py (get_variable_system_parameters / get_minimal_system_parameters / UNINTERESTING_SYSTEM_PARAMETERS)

Cargo test failures

Read the panic message or assertion diff. Common patterns:

• unwrap_err() on Ok → test expected an error but the code now succeeds
• assertion left == right failed → behavioral change in output

Testdrive cascades

After one test crashes environmentd, all subsequent tests in that shard fail with Name or service not known or connection closed. Only the first failure in a shard matters — everything after it is a cascade. Look for the first error: or FAIL in the log.

Testdrive shards with the same number (e.g., testdrive-10 and testdrive-with-alloydb-10) run the same tests — if both fail, it's likely to be the same root cause.

SLT failures

Check whether it's wrong output (behavioral change) vs. connection error (crash/timeout). Wrong output means the query semantics changed.

Step 6: Summarize

Group failures by root cause, not by job name. Typically many failing jobs share just 1-2 root causes. Present the summary as:

1. Root cause A — description, which jobs it affects, what to fix
2. Root cause B — description, which jobs it affects, what to fix

Distinguish between issues that are clearly caused by the PR's changes vs. pre-existing flaky tests. The annotations often link to known flaky test issues (GitHub database-issues links) — use these to identify pre-existing flakes vs. regressions introduced by the PR.