Use this skill when users ask about benchmarking Clojure code, measuring performance, profiling execution time, or using the criterium library. Covers the 0.5.x API including bench macro, bench plans, viewers, domain analysis, and argument generation.
Use this skill when users ask about benchmarking Clojure code, measuring performance, profiling execution time, or using the criterium library. Covers the 0.5.x API including bench macro, bench plans, viewers, domain analysis, and argument generation.
Criterium
Statistically rigorous benchmarking for Clojure that accounts for JVM warmup, garbage collection, and measurement overhead.
Overview
Criterium is the standard benchmarking library for Clojure. Unlike naive timing approaches, it provides:
JVM-aware measurement - Handles JIT warmup and GC interference
The bench macro wraps your expression in a measured - a benchmarkable unit that:
Prevents constant folding by hoisting arguments
Supports batched evaluation for fast expressions
Provides zero-allocation measurement
You rarely interact with measured directly, but it enables advanced patterns like argument generation. See Argument Generation for explicit usage with test.check generators.
Basic Benchmarking
The bench Macro
(bench/bench expr & options)
Returns the expression's value. Benchmark data available via (bench/last-bench).
Common Options
;; Change output format
(bench/bench (sort data) :viewer :pprint)
;; Use a specific bench plan
(bench/bench (sort data) :bench-plan criterium.bench-plans/distribution-analysis)
;; Limit benchmark duration
(bench/bench (sort data) :limit-time-s 5)
;; Collect allocation data (requires native agent)
(bench/bench (sort data) :with-allocation-trace true)
Using Local Bindings
The bench macro captures local bindings from the enclosing scope:
The domain-expr macro defines axes (parameter ranges) and expressions to benchmark. The bench function runs benchmarks at each coordinate and analyzes results.
Comparing Implementations
Use a map body in domain-expr to compare implementations:
Bindings are processed left-to-right, with earlier bindings available to later generators. This enables dependent generation where one value determines another:
(arg-gen/measured
[n (gen/choose 10 100) ; n bound first
coll (gen/vector gen/small-integer n)] ; n used to size the vector
(reduce + coll))
Options
;; Control generator size (affects sized generators like gen/vector)
(arg-gen/measured {:size 50}
[coll (gen/vector gen/small-integer)]
(sort coll))
;; Reproducible generation with seed
(arg-gen/measured {:seed 12345}
[n gen/small-integer]
(* n n))
The JIT compiler optimizes code during execution. Criterium handles warmup automatically, but be aware:
First benchmark in a session may be slower (class loading, JIT)
Run benchmarks multiple times if results seem inconsistent
The default plan includes warmup phases
Avoiding Measurement Pitfalls
Dead code elimination: The JVM may optimize away computations with unused results. Criterium prevents this by consuming return values, but avoid:
;; BAD - side-effect only, result discarded
(bench/bench (do (sort data) nil))
;; GOOD - return the result
(bench/bench (sort data))
Side effects: Benchmarks with side effects (I/O, mutation) may not measure what you intend:
;; BAD - file I/O dominates timing
(bench/bench (spit "test.txt" (str data)))
;; GOOD - separate I/O from computation
(bench/bench (str data))
Constant folding: The compiler may evaluate constant expressions at compile time:
;; BAD - may be optimized to constant
(bench/bench (+ 1 2))
;; BETTER - use local bindings
(let [a 1 b 2]
(bench/bench (+ a b)))
Interpreting Results
Outliers: Some outliers are normal (GC, OS scheduling). Concern when:
High-severe outliers exceed 5% of samples
Results vary significantly between runs
Minimum time much lower than mean
Confidence intervals: The 3σ bounds show where 99.7% of values fall. Wide bounds suggest high variance—consider longer benchmarks or investigating causes.
