# Criterium
## Alpha version
Master is now on 0.5.x, which should be considered ALPHA, subject to
breaking changes on new features.
See [ALPHA Readme](README.ALPHA.md)
If you try it, please give feedback on any breaking changes with 0.4.6.
## Stable version 0.4.6
Criterium measures the computation time of an expression. It is
designed to address some of the pitfalls of benchmarking, and benchmarking on
the JVM in particular.
This includes:
* statistical processing of multiple evaluations
* inclusion of a warm-up period, designed to allow the JIT compiler to
optimise its code
* purging of gc before testing, to isolate timings from GC state prior
to testing
* a final forced GC after testing to estimate impact of cleanup on the
timing results
* prevention of JIT optimisations that remove the measured work (see
below)
### What the harness stops, and its cost
This section describes current 0.5.x development behaviour, including the
[Unreleased array fix](CHANGELOG.md#unreleased), not stable 0.4.6 guarantees.
The harness stops constant folding and dead-code elimination of the
measured expression:
* Each argument of a `measured/expr` expression is stored once, before
timing starts, in an atomic cell. The cell is published once to the
blackhole, so escape analysis can not prove that the cell is local to
the thread. The measured loop reads the cell
with an opaque read (`getOpaque`) on every iteration. The JIT can not
prove the input is stable, so it can not move the expression out of
the loop or fold a `final` field read (for example a record field) to
a constant.
* The result of each iteration goes to a blackhole, so the JIT can not
remove the expression as dead code.
The opaque read has a small cost. For very cheap expressions the
reported time includes about 0.07 ns per primitive argument and about
0.2-0.35 ns per reference argument (the value changes by about 25-30%
between JVM runs). Criterium does not subtract this cost.
Every primitive argument (including `boolean`, `byte`, `short` and
`char`) reads through the opaque cell. Array arguments use
`AtomicReference` cells and opaque reads. Inferred array hints are exact
JVM-name strings, which preserve the component type and number of
dimensions. Transport preserves array identity and contents without array
copies or element boxing; it does not prevent mutations by the measured
expression. Separately, `measured/callable` arguments use plain bindings
without opaque reads.
## Installation
### Leiningen
Add the following to your `:dependencies`:
```clj
[criterium "0.4.6"]
```
### Maven
```xml
<dependency>
<groupId>criterium</groupId>
<artifactId>criterium</artifactId>
<version>0.4.6</version>
</dependency>
```
## Usage
The top level interface is in `criterium.core`.
(use 'criterium.core)
Use `bench` to run a benchmark in a simple manner.
```
(bench (Thread/sleep 1000))
=>
Execution time mean : 1.000803 sec
Execution time std-deviation : 328.501853 us
Execution time lower quantile : 1.000068 sec ( 2.5%)
Execution time upper quantile : 1.001186 sec (97.5%)
```
By default bench is quiet about its progress. Run `with-progress-reporting` to
get progress information on `*out*`.
```clj
(with-progress-reporting (bench (Thread/sleep 1000) :verbose))
(with-progress-reporting (quick-bench (Thread/sleep 1000) :verbose))
```
Lower level functions are available, that separate benchmark statistic
generation and reporting.
```clj
(report-result (benchmark (Thread/sleep 1000) {:verbose true}))
(report-result (quick-benchmark (Thread/sleep 1000)))
```
Note that results are returned to the user to prevent JIT from recognising that
the results are not used.
## Measurement Overhead Estimation
Criterium will automatically estimate a time for its measurement
overhead. The estimate is normally made once per session, and is
available in the `criterium.core/estimated-overhead-cache` var.
If the estimation is made while there is a lot of other processing
going on, then benchmarking quick functions may report small negative
times. You can force a recalculation of the overhead by calling
`criterium.core/estimated-overhead!`.
If you want consistency across JVM processes, it might be prudent to
explicitly set `criterium.core/estimated-overhead!` to a constant
value.
## References
[API Documentation](http://hugoduncan.github.io/criterium/0.4/api/)
[Annotated Source](http://hugoduncan.github.io/criterium/0.4/uberdoc.html)
See [Elliptic Group](http://www.ellipticgroup.com/html/benchmarkingArticle.html)
for a Java benchmarking library. The accompanying article describes many of the
JVM benchmarking pitfalls.
See [Criterion](http://hackage.haskell.org/package/criterion) for a Haskell
benchmarking library that applies many of the same statistical techniques.
## Todo
Serial correlation detection.
Multimodal distribution detection.
Use kernel density estimators?
## Non-Goals
Expressions are evaluated inside a function call. We deliberately do
not seek ultimate accuracy for very quick expressions where the
function overhead may be significant.
## Releasing
To release, run the `release.sh` script. This requires that you have
git-flow enabled your git repository with `git flow init`, and that
you have configured your
[credentials for clojars](https://github.com/technomancy/leiningen/blob/stable/doc/DEPLOY.md).
## YourKit
YourKit is kindly supporting open source projects with its full-featured Java
Profiler.
YourKit, LLC is the creator of innovative and intelligent tools for profiling
Java and .NET applications. Take a look at YourKit's leading software products:
* <a href="http://www.yourkit.com/java/profiler/index.jsp">YourKit Java Profiler</a> and
* <a href="http://www.yourkit.com/.net/profiler/index.jsp">YourKit .NET Profiler</a>.
## License
Licensed under [EPL](http://www.eclipse.org/legal/epl-v10.html)