benchmark-harness

name	benchmark-harness
description	Reproducibility rules, CSV schema, and measurement procedures for the benchmark defined in spec 0009. Use when implementing or reviewing files in benchmark/ or scripts/. Trigger phrases: "benchmark", "harness", "wrk", "rps_per_mb", "/benchmark-harness".
applyTo	["benchmark/**","scripts/**"]

Benchmark Harness (spec 0009)

Pre-run system preparation (Raspberry Pi 5, arm64)

These steps must be applied before starting any measurement run. Skip them and measurements will be noisy and non-reproducible.

CPU governor — performance mode

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# Verify
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor   # must print "performance"

Disable swap

sudo swapoff -a
# Verify
free -h   # Swap row should be "0B"

Thermal cooldown between runs

Measure CPU temperature between runs:

vcgencmd measure_temp         # Raspberry Pi utility
# or
cat /sys/class/thermal/thermal_zone0/temp   # divide by 1000 for °C

Record baseline temperature at the start of the session. Before each run, wait until temperature drops to ≤ baseline + 5°C. Do not proceed if the SBC is thermally throttling.

Warmup runs

Each server must receive 3 warmup runs before any measured run. Warmup runs are tagged is_warmup=True in the CSV and excluded from analysis.

# benchmark.py pattern
for run_number in range(1, N_WARMUP + N_MEASURED + 1):
    is_warmup = run_number <= N_WARMUP
    result = run_wrk(server)
    write_row(result, is_warmup=is_warmup)

N_WARMUP = 3, N_MEASURED = 30 (spec 0009).

Randomized server order

Server order must be randomized across runs to eliminate position bias (thermal state, OS caches, etc.):

import random

languages = ["go", "rust", "python", "nodejs", "dotnet"]
order = random.sample(languages, len(languages))  # shuffle each round

For each of the 30 measured runs, generate a new random order.

wrk configuration

wrk -t4 -c50 -d30s -L http://127.0.0.1:${PORT}/items

Flag	Value	Meaning
-t	4	Threads
-c	50	Concurrent connections
-d	30s	Test duration
-L	—	Latency histogram (enables p50/p75/p90/p99)

Use the Lua script for POST testing:

wrk -t4 -c50 -d30s -L -s scripts/wrk_post.lua http://127.0.0.1:${PORT}/items

RAM measurement

Poll /proc/<pid>/status every 100ms during the wrk run. Take the peak VmRSS value (resident set size):

import subprocess, time, re

def measure_ram_peak_mb(pid: int, duration_s: float) -> float:
    peak_kb = 0
    deadline = time.monotonic() + duration_s
    while time.monotonic() < deadline:
        try:
            with open(f"/proc/{pid}/status") as f:
                for line in f:
                    if line.startswith("VmRSS:"):
                        kb = int(re.search(r"\d+", line).group())
                        peak_kb = max(peak_kb, kb)
                        break
        except FileNotFoundError:
            break   # process died
        time.sleep(0.1)
    return peak_kb / 1024.0

Derived metric

rps_per_mb = rps / ram_peak_mb

This is the primary comparison metric. Do not store it as measured; compute it from rps and ram_peak_mb columns. The CSV column is included as a convenience.

Startup time measurement

import subprocess, time

t0 = time.monotonic()
proc = subprocess.Popen(["./server"])
# Poll until server is accepting connections
while True:
    try:
        urllib.request.urlopen("http://127.0.0.1:8080/items", timeout=0.1)
        break
    except Exception:
        time.sleep(0.01)
startup_s = time.monotonic() - t0

CSV schema — 20 columns, exact order

run_id, language, run_number, is_warmup, rps, p50_ms, p75_ms, p90_ms, p99_ms,
errors, ram_peak_mb, cpu_avg_pct, startup_s, binary_size_bytes, rps_per_mb,
cpu_governor, swap_mb, pi_model, kernel_version, timestamp_utc

Column	Type	Notes
run_id	str	UUID4
language	str	go, rust, python, nodejs, dotnet
run_number	int	1-indexed within this session (warmup included)
is_warmup	bool	True for warmup runs — exclude from analysis
rps	float	requests/sec from wrk
p50_ms	float	50th percentile latency in ms
p75_ms	float	75th percentile
p90_ms	float	90th percentile
p99_ms	float	99th percentile
errors	int	Non-2xx/3xx responses from wrk
ram_peak_mb	float	Peak VmRSS in MB
cpu_avg_pct	float	Average CPU usage during run (%)
startup_s	float	Seconds from process start to first accepted request
binary_size_bytes	int	os.path.getsize(binary_path)
rps_per_mb	float	rps / ram_peak_mb
cpu_governor	str	Should always be performance
swap_mb	float	Should always be 0.0
pi_model	str	From /proc/cpuinfo, Hardware field
kernel_version	str	From uname -r
timestamp_utc	str	ISO-8601 UTC, e.g. 2026-05-07T14:30:00Z

The CSV file is data/benchmarks.csv. Append rows; do not overwrite.

Anti-patterns

Anti-pattern	Problem	Fix
Running without scaling_governor=performance	Variable clock speed → noisy data	Set governor before any run
Swap enabled	OS can page memory, inflating RAM readings	sudo swapoff -a
Including warmup rows in analysis	First runs have cold JIT/cache effects	Filter is_warmup=True rows
Fixed server order across runs	Position bias (thermal, cache)	Randomize per round
Measuring VmPeak instead of VmRSS	VmPeak is historical max, not actual RSS	Use VmRSS
Using -t1 or -c1 with wrk	Doesn't stress I/O multiplexing	Use -t4 -c50