with one click
bisect
Bisect a ClickHouse regression using pre-built master binaries from CI. Use when the user wants to find the commit that introduced a bug.
Menu
Bisect a ClickHouse regression using pre-built master binaries from CI. Use when the user wants to find the commit that introduced a bug.
Investigate a ClickHouse CI failure end-to-end from a PR or S3 report URL. Fetches the failed tests and their output, searches for an existing tracking GitHub issue, classifies each as flaky vs a real regression using play.clickhouse.com master history, downloads and reads the harness artifacts only for failures that history does not explain, and reports a root-cause hypothesis. Read-only first pass — never commits, pushes, or edits.
Extract the inner ELF from a ClickHouse self-extracting `clickhouse` binary, including when its architecture differs from the host (e.g. to load an aarch64 CI core dump on an x86 workstation). Use when gdb/lldb needs real symbols from a downloaded CI/release binary, or when self-extraction by running the binary is not possible because of an architecture mismatch.
Create a ClickHouse git worktree with submodules hardlinked from the main repo. Use when the user wants to create a new worktree for ClickHouse development.
Fix the "CH Inc sync" job in a pull request. Diagnoses the failure as one of three types — conflicts found, build failed, or tests failed — and handles each accordingly.
Review a ClickHouse Pull Request for correctness, safety, performance, and compliance. Use when the user wants to review a PR or diff.
Continue work on an existing PR - resolve conflicts, fix CI failures, address reviewer feedback, and push updates. Use when the user wants to pick up and advance a pull request.
| name | bisect |
| description | Bisect a ClickHouse regression using pre-built master binaries from CI. Use when the user wants to find the commit that introduced a bug. |
| argument-hint | <repro.sql> [good-ref] [bad-ref] |
| disable-model-invocation | false |
| allowed-tools | Task, Bash(git:*), Bash(curl:*), Bash(chmod:*), Bash(mkdir:*), Bash(ls:*), Bash(/tmp/*), Bash(cat:*), Bash(utils/auto-bisect/*), Read, Write, Glob, Grep, WebFetch, LSP |
Bisect a ClickHouse regression using utils/auto-bisect/bisect.sh, which downloads pre-built CI binaries and manages the bisect loop automatically.
$0 (required): Path to a SQL repro file (e.g., /tmp/repro.sql)$1 (optional): Good ref — a git tag, branch, or SHA known to be good (default: latest v*.1.1-* tag)$2 (optional): Bad ref — a git tag, branch, or SHA known to be bad (default: HEAD)utils/auto-bisect/bisect.sh orchestrates the full bisect:
helpers/download.sh fetches the release binary from CI S3env/<option>.sh installs configs and starts a ClickHouse server (configs go to utils/auto-bisect/data/etc/, never to /etc/)$COMMIT_SHA, $CH_PATH (binary), and $GIT_WORK_TREE in the environmentThe working tree is never checked out — BISECT_HEAD is written for each step so git history is untouched.
The test script receives these environment variables:
$COMMIT_SHA — full SHA of the commit being tested$CH_PATH — path to the downloaded binary (e.g., utils/auto-bisect/data/clickhouse)$GIT_WORK_TREE — path to the ClickHouse repositoryExit codes:
0 — good (bug not present)1 — bad (bug present)125 — skip (handled upstream by download.sh, not typically needed in the test itself)git rev-parse <ref>git tag -l 'v*' --sort=-creatordate | grep '\.1\.1-' | head -5
Pick the nearest v<MAJOR>.<MINOR>.1.1-* tag before the bad ref.Create /tmp/bisect_test.sh. Use $CH_PATH for the binary — do not download it yourself, the framework handles that.
For clickhouse local repros (no server needed — use --env nothing):
#!/bin/bash
set -e
OUTPUT=$("$CH_PATH" local --multiquery 2>&1 <<'SQL'
-- paste repro SQL here
SQL
)
if echo "$OUTPUT" | grep -q "FAILURE_PATTERN"; then
exit 1 # bad
else
exit 0 # good
fi
For server-based repros (use --env single, the default):
#!/bin/bash
# Server is already running when this script is called.
# Use clickhouse-client on the default port (9000).
set -e
OUTPUT=$("$CH_PATH" client --multiquery 2>&1 <<'SQL'
-- paste repro SQL here
SQL
)
if echo "$OUTPUT" | grep -q "FAILURE_PATTERN"; then
exit 1
else
exit 0
fi
Failure pattern examples:
grep -q "LOGICAL_ERROR\|Code: [0-9]"OUTPUT to expected string[ "${PIPESTATUS[0]}" = "139" ]Before running the full bisect, verify the repro:
GOOD_SHA=$(git rev-parse <good-ref>)
BAD_SHA=$(git rev-parse <bad-ref>)
Optionally do a quick walker sanity check over just the two endpoints:
CH_PATH=/usr/local/bin/clickhouse /tmp/bisect_test.sh # should exit 0 on good
cd utils/auto-bisect
./bisect.sh \
--good <good-sha> \
--bad <bad-sha> \
--path /path/to/ClickHouse \
--test /tmp/bisect_test.sh \
--env nothing # or: single (default), replicateddb, sharedcatalog
Use --env nothing when the test uses clickhouse local (no server).
Use --env single (default) when the test needs a running server.
Walker mode — walk every commit linearly instead of bisecting (useful when you want to see the progression or the range is small):
./bisect.sh --good <sha> --bad <sha> --path ... --test ... --walker
# Or provide explicit commits:
./bisect.sh --path ... --test ... --walker --walker-commits "sha1 sha2 sha3"
# Or limit to N evenly-spaced steps:
./bisect.sh --good <sha> --bad <sha> --path ... --test ... --walker --walker-steps 10
Set a generous timeout (up to 10 min) — each step downloads ~200 MB.
After bisect completes, the output contains <sha> is the first bad commit.
git show --stat <bad-sha>
Report:
https://github.com/ClickHouse/ClickHouse/pull/<N>git show --statutils/auto-bisect/data/clickhouse and overwritten each step (not cached between steps — use walker if you want to re-test)bisect.sh uses --first-parent automaticallyutils/auto-bisect/data/etc/clickhouse-server/ — the host's /etc/clickhouse-* is never touchedBISECT_HEAD is used instead of git checkoutCH_CI_USER / CH_CI_PASSWORD and pass --private