| name | performance-benchmarking |
| description | Build credible benchmarks and make defensible performance claims. Use when a project claims to be fast, when adding a benchmark suite, when comparing against competing projects, when setting up performance regression detection in CI, or when the user wants to put a speed number in the README. Covers measurement methodology, statistical rigor, fair competitor comparisons, profiling before optimizing, microbenchmark pitfalls, and publishing results honestly. Also use when reviewing someone else's benchmark claims. |
Performance and Benchmarking
"Blazing fast" in a README is worth nothing. A reproducible benchmark with a stated
methodology is worth a great deal — and a benchmark that gets caught being unfair
costs more than never publishing one.
Profile before you optimize
Non-negotiable order: measure → find the actual hot spot → change one thing →
measure again. Programmer intuition about hot paths is wrong most of the time, and
optimizing cold code adds complexity for zero benefit.
py-spy record -o profile.svg -- python app.py
node --cpu-prof app.js
cargo flamegraph --bench my_bench
go test -cpuprofile cpu.out -bench . && go tool pprof -http=: cpu.out
perf record -g ./binary && perf report
Read the flame graph for width, not height. Deep stacks are normal; wide frames
are where time goes. And check allocation profiles too — in managed languages, GC
pressure is frequently the real answer and never appears as an obvious hot function.
Benchmark methodology
A benchmark nobody can reproduce is marketing. Requirements:
- Warm up. JIT compilation, caches, connection pools. Discard the first N runs.
- Repeat and report distribution. Minimum, median, p95, p99 — and the shape.
A single number hides bimodality, and the mean is the least useful statistic on a
long-tailed distribution.
- Report variance. ±2% and ±40% are completely different claims.
- Control the environment. Pin CPU frequency where possible, disable turbo, close
everything else, use a quiet machine. Never benchmark on a shared CI runner and
present the numbers as authoritative.
- Publish the hardware and versions. CPU model, RAM, OS, kernel, runtime version,
library versions, dataset. Without these the number is meaningless.
- Realistic workload. Benchmarking a parser on a 12-byte input measures function
call overhead, not parsing.
- Verify correctness in the benchmark. A fast wrong answer is easy. Assert on the
output — this also prevents dead-code elimination silently removing your workload.
Use a harness that handles the statistics for you rather than hand-rolled timing loops:
hyperfine --warmup 3 --runs 50 'old-bin input.json' 'new-bin input.json'
cargo bench
pytest-benchmark / asv
node --expose-gc + tinybench / mitata
go test -bench . -benchmem -count=10 | benchstat
benchstat and criterion do the thing most hand-written benchmarks skip: telling you
whether the difference is statistically significant at all.
Microbenchmark traps
Microbenchmarks lie more often than they inform.
- Dead code elimination. The compiler notices you discard the result and removes
the work. Use
std::hint::black_box, benchmark.Keep, or assert on the output.
- Constant folding. Benchmarking with a literal input computes at compile time.
- Branch predictor training. The same input 10,000 times is unrealistically
predictable; real workloads are not.
- Everything in L1 cache. A 1 KB dataset benchmark tells you nothing about the
100 MB case, and cache behavior is often the entire story.
- Measuring the harness. If the operation takes 40 ns and your loop overhead is
30 ns, you are benchmarking the loop.
- Alignment and layout luck. Unrelated code changes can shift performance several
percent by moving code across cache lines. Treat sub-5% microbenchmark deltas as
noise unless you can explain the mechanism.
When in doubt, benchmark at a level users can perceive: a whole request, a whole file
parse, a whole build.
Comparing against competitors
The highest-risk content you can publish. Do it fairly or not at all — the maintainers
of the project you benchmarked will read it, and being publicly corrected on an
unfair benchmark is a reputational event that outlives the blog post.
Rules:
- Use their recommended configuration. Benchmarking a competitor in debug mode,
without their cache enabled, or with default settings they explicitly tell users to
change, is the single most common form of benchmark dishonesty.
- Ask them to review it. Open an issue on their repo with your methodology before
publishing. This is unusual, cheap, and enormously credibility-enhancing.
- Publish the full harness — scripts, datasets, versions, raw output — in a repo
anyone can run.
- Report where you lose. A benchmark table with no losses reads as fabricated and
is treated as such by exactly the readers you are trying to convince.
- Compare like for like. If you skip a feature they provide (validation, retries,
cross-platform support), say so. The honest framing is "X% faster, and here is the
tradeoff", not a bare number.
- Date and version everything. They will ship a fix; your table will not.
State the workload in the claim itself: not "3× faster" but "parses a 50 MB NDJSON
file 3.1× faster than X v4.2 on an M3 Pro (median of 50 runs, ±3%)."
Regression detection in CI
Continuous benchmarking is valuable, but shared CI runners have 10–30% variance —
naive thresholds produce constant false alarms and get muted within a month.
Approaches that survive contact with reality:
- Track counters, not wall time. Instruction counts, allocations, syscalls, bytes
allocated. These are nearly deterministic on shared hardware.
cachegrind /
iai-callgrind / valgrind --tool=callgrind make this practical, and a 2% change
in instruction count is real signal.
- Compare within the same job, base vs PR, on the same runner, interleaved. Never
compare against a number recorded on a different machine last week.
- Alert on trend, not on a single run. Publish to a dashboard
(
github-action-benchmark) and investigate sustained slopes.
- Gate only on large regressions (>10%), and always allow an override label. A
performance gate that blocks a correctness fix is a broken process.
- Run the real benchmark suite on dedicated hardware, nightly, not per-PR.
Publishing results
In the README, keep it to one line with a link. In docs/benchmarks.md, give the full
picture: methodology, hardware, versions, dataset, raw numbers, the harness link, and
the caveats. Include a "when this project is slower" section — it is the most credible
paragraph you will write.
Re-run before every major release. A benchmark from three versions ago is a claim you
are no longer making truthfully.
Reviewing someone else's benchmark
Questions that expose most bad benchmarks in under a minute:
- What hardware, what versions, what dataset size?
- How many runs, and what is the variance?
- Is the harness public and runnable?
- Was the competitor configured as its own docs recommend?
- Is the workload representative, or chosen to favor one implementation?
- Does the benchmark verify correctness of the output?
- Who paid for it, and does the conclusion match their interest?
If a benchmark cannot answer 1–3, it is not evidence. Say so plainly and without
hostility.
Anti-patterns
- "Blazing fast" with no number.
- A number with no methodology.
- Benchmarking on a laptop on battery, or on a shared CI runner, and presenting it
as authoritative.
- Comparing your optimized build to their debug build.
- Cherry-picking the one workload where you win.
- Optimizing without profiling. You will make the code worse and no faster.
- Trading correctness or safety for speed silently. If you skip validation to win
a benchmark, that belongs in the claim.
- Never re-running published benchmarks. They become false over time on their own.
