performance-benchmarking

name

performance-benchmarking

description

Build credible benchmarks and make defensible performance claims. Use when a project claims to be fast, when adding a benchmark suite, when comparing against competing projects, when setting up performance regression detection in CI, or when the user wants to put a speed number in the README. Covers measurement methodology, statistical rigor, fair competitor comparisons, profiling before optimizing, microbenchmark pitfalls, and publishing results honestly. Also use when reviewing someone else's benchmark claims.

Performance and Benchmarking

"Blazing fast" in a README is worth nothing. A reproducible benchmark with a stated methodology is worth a great deal — and a benchmark that gets caught being unfair costs more than never publishing one.

Profile before you optimize

Non-negotiable order: measure → find the actual hot spot → change one thing → measure again. Programmer intuition about hot paths is wrong most of the time, and optimizing cold code adds complexity for zero benefit.

# Sampling profilers — start here
py-spy record -o profile.svg -- python app.py       # Python, no code changes
node --cpu-prof app.js                               # Node
cargo flamegraph --bench my_bench                    # Rust
go test -cpuprofile cpu.out -bench . && go tool pprof -http=: cpu.out
perf record -g ./binary && perf report               # anything on Linux

Read the flame graph for width, not height. Deep stacks are normal; wide frames are where time goes. And check allocation profiles too — in managed languages, GC pressure is frequently the real answer and never appears as an obvious hot function.

Benchmark methodology

A benchmark nobody can reproduce is marketing. Requirements:

1. Warm up. JIT compilation, caches, connection pools. Discard the first N runs.
2. Repeat and report distribution. Minimum, median, p95, p99 — and the shape. A single number hides bimodality, and the mean is the least useful statistic on a long-tailed distribution.
3. Report variance. ±2% and ±40% are completely different claims.
4. Control the environment. Pin CPU frequency where possible, disable turbo, close everything else, use a quiet machine. Never benchmark on a shared CI runner and present the numbers as authoritative.
5. Publish the hardware and versions. CPU model, RAM, OS, kernel, runtime version, library versions, dataset. Without these the number is meaningless.
6. Realistic workload. Benchmarking a parser on a 12-byte input measures function call overhead, not parsing.
7. Verify correctness in the benchmark. A fast wrong answer is easy. Assert on the output — this also prevents dead-code elimination silently removing your workload.

Use a harness that handles the statistics for you rather than hand-rolled timing loops:

hyperfine --warmup 3 --runs 50 'old-bin input.json' 'new-bin input.json'   # CLI tools
cargo bench                        # criterion: statistics + regression detection
pytest-benchmark / asv             # Python
node --expose-gc + tinybench / mitata
go test -bench . -benchmem -count=10 | benchstat   # benchstat gives significance

benchstat and criterion do the thing most hand-written benchmarks skip: telling you whether the difference is statistically significant at all.

Microbenchmark traps

Microbenchmarks lie more often than they inform.

• Dead code elimination. The compiler notices you discard the result and removes the work. Use std::hint::black_box, benchmark.Keep, or assert on the output.
• Constant folding. Benchmarking with a literal input computes at compile time.
• Branch predictor training. The same input 10,000 times is unrealistically predictable; real workloads are not.
• Everything in L1 cache. A 1 KB dataset benchmark tells you nothing about the 100 MB case, and cache behavior is often the entire story.
• Measuring the harness. If the operation takes 40 ns and your loop overhead is 30 ns, you are benchmarking the loop.
• Alignment and layout luck. Unrelated code changes can shift performance several percent by moving code across cache lines. Treat sub-5% microbenchmark deltas as noise unless you can explain the mechanism.

When in doubt, benchmark at a level users can perceive: a whole request, a whole file parse, a whole build.

Comparing against competitors

The highest-risk content you can publish. Do it fairly or not at all — the maintainers of the project you benchmarked will read it, and being publicly corrected on an unfair benchmark is a reputational event that outlives the blog post.

Rules:

• Use their recommended configuration. Benchmarking a competitor in debug mode, without their cache enabled, or with default settings they explicitly tell users to change, is the single most common form of benchmark dishonesty.
• Ask them to review it. Open an issue on their repo with your methodology before publishing. This is unusual, cheap, and enormously credibility-enhancing.
• Publish the full harness — scripts, datasets, versions, raw output — in a repo anyone can run.
• Report where you lose. A benchmark table with no losses reads as fabricated and is treated as such by exactly the readers you are trying to convince.
• Compare like for like. If you skip a feature they provide (validation, retries, cross-platform support), say so. The honest framing is "X% faster, and here is the tradeoff", not a bare number.
• Date and version everything. They will ship a fix; your table will not.

State the workload in the claim itself: not "3× faster" but "parses a 50 MB NDJSON file 3.1× faster than X v4.2 on an M3 Pro (median of 50 runs, ±3%)."

Regression detection in CI

Continuous benchmarking is valuable, but shared CI runners have 10–30% variance — naive thresholds produce constant false alarms and get muted within a month.

Approaches that survive contact with reality:

• Track counters, not wall time. Instruction counts, allocations, syscalls, bytes allocated. These are nearly deterministic on shared hardware. cachegrind / iai-callgrind / valgrind --tool=callgrind make this practical, and a 2% change in instruction count is real signal.
• Compare within the same job, base vs PR, on the same runner, interleaved. Never compare against a number recorded on a different machine last week.
• Alert on trend, not on a single run. Publish to a dashboard (github-action-benchmark) and investigate sustained slopes.
• Gate only on large regressions (>10%), and always allow an override label. A performance gate that blocks a correctness fix is a broken process.
• Run the real benchmark suite on dedicated hardware, nightly, not per-PR.

Publishing results

In the README, keep it to one line with a link. In docs/benchmarks.md, give the full picture: methodology, hardware, versions, dataset, raw numbers, the harness link, and the caveats. Include a "when this project is slower" section — it is the most credible paragraph you will write.

Re-run before every major release. A benchmark from three versions ago is a claim you are no longer making truthfully.

Reviewing someone else's benchmark

Questions that expose most bad benchmarks in under a minute:

1. What hardware, what versions, what dataset size?
2. How many runs, and what is the variance?
3. Is the harness public and runnable?
4. Was the competitor configured as its own docs recommend?
5. Is the workload representative, or chosen to favor one implementation?
6. Does the benchmark verify correctness of the output?
7. Who paid for it, and does the conclusion match their interest?

If a benchmark cannot answer 1–3, it is not evidence. Say so plainly and without hostility.

Anti-patterns

• "Blazing fast" with no number.
• A number with no methodology.
• Benchmarking on a laptop on battery, or on a shared CI runner, and presenting it as authoritative.
• Comparing your optimized build to their debug build.
• Cherry-picking the one workload where you win.
• Optimizing without profiling. You will make the code worse and no faster.
• Trading correctness or safety for speed silently. If you skip validation to win a benchmark, that belongs in the claim.
• Never re-running published benchmarks. They become false over time on their own.