performance-testing

# Performance testing Run every benchmark workflow through `tools/performance/perf.sh` from the repository root. `tools/performance/benchmarking.md` is the human-facing guide to the same tooling. Load [benchmark patterns](references/benchmark-patterns.md) before adding or editing a `[Benchmark]` method or benchmark class. ## Workflows `evaluate` and `spot` require `jq`. ```bash # Spot check: one indicator against its baselines (fast; the dev-loop check) bash tools/performance/perf.sh spot Ema bash tools/performance/perf.sh spot Adx Stream # Series | Buffer | Stream | All (default) # Evaluate: full baseline suite, report regressions and improvements (~1 hour) bash tools/performance/perf.sh evaluate # Reset: full baseline suite, overwrite tools/performance/baselines/ (~1 hour) bash tools/performance/perf.sh reset ``` - `spot` matches the indicator name as a substring, so `spot Ema` also runs `Dema` and `Tema`. - VS Code tasks: "Perf: Spot check (indicator vs baseline)", "Perf: Evaluate against baselines", "Perf: Reset baselines". - `reset --prune` also deletes committed baseline files for suites no longer in the set. ## Baseline set The baseline set is the no-argument `dotnet run -c Release` list in `tools/performance/Program.cs`: `SeriesIndicators`, `BufferIndicators`, `StreamIndicators`, `StreamObserver`, `Utility`, `UtilityNullMath`, `UtilityStdDev`. `BASELINE_CLASSES` in `perf.sh` must list the same suites; change both together. `StyleComparison`, `StyleWallTime`, `StreamCrossover`, `StreamExternal`, and `ManualTestDirect` are diagnostics with no baseline. Run them with a raw filter. ## Raw runs For exploration only, never for baseline comparison. Run from `tools/performance`: ```bash dotnet run -c Release -- --filter "*StyleComparison*" # one suite dotnet run -c Release -- --filter "*.ToEmaBatch" # one method # Large-N harness: PERF_TEST_PERIODS bars (default 500000); PERF_TEST_CAP below it exercises pruning PERF_TEST_KEYWORD=ema PERF_TEST_PERIODS=500000 dotnet run -c Release -- --filter "Performance.ManualTestDirect*" PERF_TEST_KEYWORD=adl PERF_TEST_PERIODS=500000 PERF_TEST_CAP=100000 dotnet run -c Release -- --filter "Performance.ManualTestDirect*" ``` Results land in `tools/performance/BenchmarkDotNet.Artifacts/results/` as `Performance.{Suite}-report-full.json` and `-report-github.md`. ## Regression detection `detect-regressions.sh` pairs each `*-report-full.json` result with the same-named baseline and compares per method, so a spot run compares only what it ran. The default threshold is 10%. Run it alone on existing results with `bash tools/performance/detect-regressions.sh [--threshold 15]`. Exit codes: `0` no regression, `1` regression found, `2` usage or I/O error. ## Targets | Style | Target time vs Series | | ----- | --------------------- | | BufferList | ≤ 1.2x | | StreamHub | ≤ 1.5x | These are optimization targets, not merge gates; current implementations vary by indicator. Measure the ratio with the `StyleComparison` suite. Read current per-indicator timings from `tools/performance/baselines/Performance.SeriesIndicators-report-github.md` and its Buffer and Stream siblings. ## Do not do these - Do not pass tuning options (`--job`, `--warmupCount`, thresholds) to `perf.sh` runs; each suite pins its job in code, and plain runs stay comparable to the committed baselines. - Do not run `perf.sh reset` except to record an intended, verified performance change, and commit the refreshed baselines with that change. - Do not compare baselines across machines or gate merges on CI timings; the committed baselines come from a developer machine, and the `test-performance*.yml` workflows are manual and informational. - Do not add LINQ, boxing, or per-period allocations other than the result itself to an indicator's per-period loop.