bisect

# ClickHouse Bisect Skill Bisect a ClickHouse regression using `utils/auto-bisect/bisect.sh`, which downloads pre-built CI binaries and manages the bisect loop automatically. ## Arguments - `$0` (required): Path to a SQL repro file (e.g., `/tmp/repro.sql`) - `$1` (optional): Good ref — a git tag, branch, or SHA known to be good (default: latest `v*.1.1-*` tag) - `$2` (optional): Bad ref — a git tag, branch, or SHA known to be bad (default: `HEAD`) ## How the auto-bisect script works `utils/auto-bisect/bisect.sh` orchestrates the full bisect: 1. For each commit, `helpers/download.sh` fetches the release binary from CI S3 2. `env/<option>.sh` installs configs and starts a ClickHouse server (configs go to `utils/auto-bisect/data/etc/`, never to `/etc/`) 3. Your test script is called with `$COMMIT_SHA`, `$CH_PATH` (binary), and `$GIT_WORK_TREE` in the environment 4. Exit code from your test determines good/bad/skip The working tree is **never checked out** — `BISECT_HEAD` is written for each step so git history is untouched. ## Test script contract The test script receives these environment variables: - `$COMMIT_SHA` — full SHA of the commit being tested - `$CH_PATH` — path to the downloaded binary (e.g., `utils/auto-bisect/data/clickhouse`) - `$GIT_WORK_TREE` — path to the ClickHouse repository Exit codes: - `0` — good (bug not present) - `1` — bad (bug present) - `125` — skip (handled upstream by download.sh, not typically needed in the test itself) ## Process ### 1. Validate inputs - Read and understand the repro SQL - Resolve good/bad refs to full SHAs: `git rev-parse <ref>` - If no good ref given, find the previous release branch point: ```bash git tag -l 'v*' --sort=-creatordate | grep '\.1\.1-' | head -5 ``` Pick the nearest `v<MAJOR>.<MINOR>.1.1-*` tag before the bad ref. ### 2. Write the test script Create `/tmp/bisect_test.sh`. Use `$CH_PATH` for the binary — do **not** download it yourself, the framework handles that. **For `clickhouse local` repros (no server needed — use `--env nothing`):** ```bash #!/bin/bash set -e OUTPUT=$("$CH_PATH" local --multiquery 2>&1 <<'SQL' -- paste repro SQL here SQL ) if echo "$OUTPUT" | grep -q "FAILURE_PATTERN"; then exit 1 # bad else exit 0 # good fi ``` **For server-based repros (use `--env single`, the default):** ```bash #!/bin/bash # Server is already running when this script is called. # Use clickhouse-client on the default port (9000). set -e OUTPUT=$("$CH_PATH" client --multiquery 2>&1 <<'SQL' -- paste repro SQL here SQL ) if echo "$OUTPUT" | grep -q "FAILURE_PATTERN"; then exit 1 else exit 0 fi ``` Failure pattern examples: - Exception: `grep -q "LOGICAL_ERROR\|Code: [0-9]"` - Wrong result: compare `OUTPUT` to expected string - Segfault: `[ "${PIPESTATUS[0]}" = "139" ]` ### 3. Confirm the bug range Before running the full bisect, verify the repro: ```bash GOOD_SHA=$(git rev-parse <good-ref>) BAD_SHA=$(git rev-parse <bad-ref>) ``` Optionally do a quick walker sanity check over just the two endpoints: ```bash CH_PATH=/usr/local/bin/clickhouse /tmp/bisect_test.sh # should exit 0 on good ``` ### 4. Run bisect ```bash cd utils/auto-bisect ./bisect.sh \ --good <good-sha> \ --bad <bad-sha> \ --path /path/to/ClickHouse \ --test /tmp/bisect_test.sh \ --env nothing # or: single (default), replicateddb, sharedcatalog ``` Use `--env nothing` when the test uses `clickhouse local` (no server). Use `--env single` (default) when the test needs a running server. **Walker mode** — walk every commit linearly instead of bisecting (useful when you want to see the progression or the range is small): ```bash ./bisect.sh --good <sha> --bad <sha> --path ... --test ... --walker # Or provide explicit commits: ./bisect.sh --path ... --test ... --walker --walker-commits "sha1 sha2 sha3" # Or limit to N evenly-spaced steps: ./bisect.sh --good <sha> --bad <sha> --path ... --test ... --walker --walker-steps 10 ``` Set a generous timeout (up to 10 min) — each step downloads ~200 MB. ### 5. Report findings After bisect completes, the output contains `<sha> is the first bad commit`. ```bash git show --stat <bad-sha> ``` Report: - First bad commit SHA and message - PR number from message (e.g., "Merge pull request #NNNNN") → link to `https://github.com/ClickHouse/ClickHouse/pull/<N>` - Brief description from `git show --stat` - Any skipped commits (no CI binary) that fall in the range ## Notes - Binaries are downloaded to `utils/auto-bisect/data/clickhouse` and overwritten each step (not cached between steps — use walker if you want to re-test) - Only first-parent merge commits on master have CI binaries; `bisect.sh` uses `--first-parent` automatically - Configs are installed to `utils/auto-bisect/data/etc/clickhouse-server/` — the host's `/etc/clickhouse-*` is never touched - The working tree is never modified during bisect or walker runs; `BISECT_HEAD` is used instead of `git checkout` - For private CI builds, set `CH_CI_USER` / `CH_CI_PASSWORD` and pass `--private`