cross-revision-benchmark

name	cross-revision-benchmark
description	Use whenever a performance claim must be made or checked: a change is motivated by speed, a reviewer questions a speedup or regression, a PR needs before/after numbers, or two implementations/revisions must be timed against each other. Never assert performance from reasoning or a scratch timing loop — run this skill's benchmark procedure instead.

Cross-revision benchmarking

Compare two columns by default: the branch tip and its base (git merge-base HEAD master).

The benchmark comes first

Benchmark only a committed Google Benchmark file (benchmarks/*.cpp, registered in benchmarks/CMakeLists.txt, built as benchmarks_main). Commit the benchmark as the branch's first commit, before the change it measures — then it exists at every revision being compared. Pick the base column accordingly: the commit where the benchmark landed, or the base branch tip directly if the benchmark is already there.

Running a column (no rebuild churn)

Use a release preset (openblas-cpu or similar) — debug/ASan timings are not representative. build_preset.sh builds are uninstrumented: RUN_TESTS=ON would add --coverage to the cytnx library and skew the timings of anything linking it, so the script wires in strip-coverage-launcher.sh as the compiler/linker launcher automatically (see that script for details) — a build dir it made for tests is safe to reuse for benchmarks.

Switch revisions in place in the main tree (git checkout <rev>) — never a fresh worktree, which starts with an empty build dir and forces a full rebuild; in-place checkout keeps build/<preset> incremental. Stash uncommitted work first and return to the original ref at the end. Per revision:

S=.claude/skills/build-test-workflow/scripts/build_preset.sh
"$S" openblas-cpu --target benchmarks_main --test \
  --benchmark_repetitions=5 --benchmark_report_aggregates_only=true \
  --benchmark_filter='<pattern>' > /tmp/bm-<label>.txt

Strictly sequential: never run a benchmark concurrently with another build or benchmark run — CPU contention skews results.

Report

• Every column: a readable label and its commit SHA — e.g. "min-heap fix (5d4ca56)". All columns from the same machine and session; state the aggregation ("mean of 5 repetitions") and the unit.
• Compare against a correct baseline: if an old revision is buggy in a way that changes how much work it does (e.g. out-of-bounds reads causing skipped work), its numbers are not a fair baseline — say so explicitly rather than letting the table imply a fair race.
• Append the deciding numbers to the PR description so the performance choice is auditable.