testo-benchmarks

name	testo-benchmarks
description	Write or tune Testo performance benchmarks with

Benchmarks in Testo (#[Bench])

#[Bench] runs a method many times, collects timing statistics, and reports Mean, Median, RStDev, with outlier rejection. The stability target is RStDev < 2% — above that, the result is noisy and shouldn't be used to draw conclusions.

Requires the BenchmarkPlugin to be enabled for the suite. Fetch https://php-testo.github.io/llms.txt for the current #[Bench] parameter list — they evolve.

Canonical shape

<?php
declare(strict_types=1);

namespace App\Bench;

use Testo\Bench;

final class SerializationBench
{
    #[Bench(
        callables: [
            'json_encode'   => 'json_encode',
            'native_serial' => 'serialize',
            'custom'        => [self::class, 'customEncode'],
        ],
        arguments: [['a' => 1, 'b' => [2, 3, 4], 'c' => 'x']],
        calls: 1000,
        iterations: 5,
    )]
    public static function encode(): void
    {
        // Body is unused when `callables` is set — the listed callables are compared head-to-head.
    }

    public static function customEncode(array $data): string
    {
        return MyEncoder::encode($data);
    }
}

Key parameters:

• callables — map of label => callable to compare. Use it for A/B comparisons.
• arguments — array of positional arguments, applied to every callable.
• calls — invocations per iteration (the inner loop). Increase until per-iteration time is well above timer resolution.
• iterations — number of iterations (the outer loop). Drives the statistics (Mean/Median/RStDev).

Selecting or excluding benches

Benches are slow and noise-sensitive, so they shouldn't run on every testo invocation. Filter by type:

vendor/bin/testo --type=!bench   # everything except benches
vendor/bin/testo --type=bench    # only benches

--type is repeatable and !-prefixed values exclude (exclusion wins). Putting benches in their own SuiteConfig (with BenchmarkPlugin, enabled per suite) and running --suite=Bench works too.

Writing a clean benchmark

1. Isolate the work. Move setup outside the benched callable — building inputs every call inflates the result.
2. Warm up. The first iteration is often slower (cold cache). Trust the median over the mean for that reason.
3. Stabilise. Iterate until RStDev < 2%. If you can't get there, the system is noisy (background processes, thermal throttling, GC) — say so honestly rather than ship unstable numbers.
4. Compare like with like. Same inputs across all callables. Don't bench json_encode($small) against serialize($large).
5. Don't benchmark trivial work. If a call is sub-microsecond and being looped 10 000 times, you are measuring loop overhead, not the work.

When not to write a benchmark

• "Is this fast enough?" without a baseline — measure two implementations against each other, never in isolation.
• I/O-bound code (DB, HTTP, FS) — variance swamps any signal. Use timing tests with clear thresholds instead.
• Code that allocates large objects — GC will dominate. Either disable GC for the bench or reuse buffers.

Reading results

• Mean — sensitive to outliers; high mean with low median = a few slow iterations.
• Median — preferred summary for noisy workloads.
• RStDev — relative standard deviation as a %. Above 2% means the result isn't reproducible at one significant figure.

When reporting comparison results, always include RStDev for each variant. A "30% faster" claim with 8% RStDev is noise.

Pitfalls

• Don't put assertions inside a benched callable — they inflate the timing and obscure the signal.
• Don't share state between iterations unless the benchmark is explicitly testing amortized cost.
• Don't compare benchmarks across machines, CI runners, or PHP versions without re-running both sides on the same host.
• Don't bench in a method that also uses #[Test] or #[TestInline] — keep benchmark classes separate.