| name | benchmarking |
| description | The reproducible benchmark protocol for this workstation — which entry points to run, warm-up policy, how results are recorded and compared, and what counts as a regression. Use before reporting any latency/throughput/memory number. |
Benchmarking protocol
Reference for producing comparable performance numbers on the RTX 5090 / Ryzen 9950X
stack. Follow .styles/TEST_OUTPUT_STYLE.md for all output.
Sandboxed execution: agents cannot reach the host, its GPU, or the running services.
Every command in this skill is executed by the human on the workstation. The agent
prepares the exact command line (per this protocol), waits for the human to report the
output, and interprets the reported numbers. Preconditions (make status,
make health) are likewise confirmed via human-reported output.
Entry points
There is no make benchmark target — run the scripts directly against a running service:
- CPU:
./scripts/benchmark/benchmark_cpu.sh -n <name> [-l basic|detailed|deep] [-p N]
(default container llama-cpu, 20 prompts, basic = perf stat core events).
- GPU:
./scripts/benchmark/benchmark_gpu.sh -n <name> [-l basic|deep] [-p N]
(default container llama-gpu, basic = nvidia-smi polling, deep = Nsight Systems).
- Perplexity/quality:
./scripts/benchmark/test_perplexity.sh.
- The load generator underneath both is
scripts/benchmark/server-bench.py.
Preconditions: the target service must be healthy first — check make status (and
make health) before starting. A benchmark against a cold or unhealthy service is invalid.
Warm-up and sampling
- Discard the first run (warm-up): caches, JIT, and clocks are not yet settled.
- Take multiple samples (>= 3 prompts-batches) at a fixed prompt count and dataset.
- Report median and spread, never a single best number.
- Hold the workload constant across comparisons: same model, same
-p prompt count,
same profiling level, same dataset under scripts/benchmark/datasets/.
Recording and comparison
- Name each run with
-n <name> so its output files are traceable to a configuration.
- Record the fixed conditions alongside the number: model, quantization, prompt count,
profiling level, driver/CUDA versions, and the exact command line.
- Compare only like-for-like runs (same workload, same service).
Regression criterion
A change is a regression when the median of the new run is more than 5% slower than the
recorded baseline for the same workload, outside the run-to-run spread. Flag it explicitly;
do not average a regression away against unrelated wins.
