create-benchmark

name

create-benchmark

description

Use when creating, editing, or reviewing a Go benchmark in this repo — any change under benchmarks/{slug}/ (*_test.go, a-consts.go, _meta.yml), or when the user asks to add/improve/fix a benchmark or compare Go implementations. Covers the BenchmarkName_behavior naming convention the cmd/ parser depends on, keeping _meta.yml implementation names in sync, writing correct benchmarks (b.N, ResetTimer, dead-code sinks, RunParallel), and how run/generate turn *_test.go into _bench.json. See reference.md for the full _meta.yml schema and parsed JSON shape. Not for frontend/chart work (that's plain repo work) — only for the benchmark subsystem.

Create Benchmark

Creating, modifying, and reviewing Go benchmarks in benchmarks/{slug}/.

Quick reference

• Benchmarks live in benchmarks/{slug}/ (slug is lowercase-with-hyphens).
• Each needs: one or more *_test.go + _meta.yml. Package name = slug with hyphens replaced by underscores (map-vs-switch → package map_vs_switch).
• Generated files (_bench.out, _bench.json) come from the cmd/ CLI — do not edit by hand.
• Shared constants go in a-consts.go (the a- prefix sorts first; the parser emits it as Constants in the JSON).

Full _meta.yml schema, field table, and parsed JSON structure: see reference.md.

Naming convention (critical)

Function format: BenchmarkImplementationName_behavior. The cmd/ parser splits it:

1. Implementation name: CamelCase after Benchmark, split at uppercase boundaries — BenchmarkStringBuilder → "String Builder", BenchmarkAtomicPointerCounter → "Atomic Pointer Counter".
2. Behavior suffix: lowercase after the underscore — _run for single-behavior; _read/_write/etc. for multi-behavior.

The implementation field in _meta.yml must exactly match the parsed CamelCase-to-space form. Every implementation in a group must define the same set of behavior suffixes — the UI auto-detects multiple behaviors and renders synced tabs.

Writing correct benchmarks

1. Drive the hot loop with b.N — never a fixed count.

   for i := 0; i < b.N; i++ { /* code under test */ }
   

2. b.ResetTimer() after expensive setup that shouldn't be measured.
3. Prevent dead-code elimination — consume results into a package-level sink:

   var sink int
   func BenchmarkFoo_run(b *testing.B) {
       for i := 0; i < b.N; i++ { sink = compute() }
   }
   

4. Prevent loop hoisting — vary inputs per iteration so invariant work isn't lifted out (e.g. _ = m[keys[i%len(keys)]]).
5. b.RunParallel for concurrent benchmarks, not raw goroutines.
6. Keep setup fair — all implementations in a group operate on equivalent data sizes and conditions.
7. b.StopTimer() / b.StartTimer() only for unavoidable per-iteration cleanup; prefer restructuring to avoid them.
8. Allocations are automatic — the CLI runs with -benchmem; no b.ReportAllocs().
9. Comment sparingly — short comments for complex steps only.

Common mistakes

• Value receiver on mutable state (func (c IntCounter) increment() mutates a copy — use a pointer receiver).
• Measuring setup instead of work (missing b.ResetTimer()).
• Inconsistent behavior suffixes across implementations.
• Creating fresh data inside the b.N loop (data-dependent timing).
• Side effects leaking between iterations.

Running / validating

• NEVER run task bench — it re-runs every benchmark (minutes of CPU). The user runs the full run+generate step. To verify a benchmark you touched, use plain Go tooling in its directory, e.g. go test -bench . -benchmem -benchtime 100x.
• The full pipeline (for context): task bench runs go run . run --all then go run . generate in cmd/, executing each benchmark across iteration counts 1000x–10000x and CPU counts 1,2,4,8,… with -bench . -benchmem, 10 repetitions, then writes the median into _bench.json.

Workflows

New benchmark: create benchmarks/{slug}/ → write *_test.go with BenchmarkName_behavior funcs → add a-consts.go if needed → write _meta.yml (verify implementation names match the parsed CamelCase form) → the user runs task bench to generate data.

Improving one: check b.N usage and b.ResetTimer(), fairness across implementations, dead-code sinks, receiver types (pointer vs value for mutation), b.RunParallel where concurrency matters, and that _meta.yml still matches the functions. Always update _meta.yml when renaming/adding/removing functions.