| name | benchmark |
| description | Plan, run, and interpret Locust load benchmarks (mock-backend and real-GPU) including A/B routing experiments. Use when the user wants to measure cache-hit ratio, TTFT, or throughput, compare routing modes, chase a benchmark regression, or design a benchmark experiment. |
| argument-hint | ["what-to-measure"] |
BENCHMARK TASK
Goal: $ARGUMENTS
Ref: .dev-context/claude-context.md for architecture. Benchmarks run on the developer's machine or GPU rig, never in this sandbox — your deliverables are the exact commands, the experiment design, and the interpretation of results the developer pastes back.
RUNNERS
| Runner | Command | What it measures |
|---|
| Mock-backend load test | make benchmark | Routing/proxy behavior without GPUs. Docker Compose benchmark profile; builds images inline (set SKIP_BUILD=1 to reuse); reports to benchmark-reports/ |
| CI regression | .github/workflows/benchmark.yml | Same, on PRs and main; baseline in tests/integration/benchmark-baseline.json |
| Real GPU A/B | scripts/bench.sh | TTFT / cache-hit against real vLLM backends |
| Regression bisect | tests/integration/bisect-benchmark.sh | Which commit moved the number |
| Micro/EPP | make bench-hot-prefix, make bench-epp, make bench-inline-vs-sidecar | Component-level costs |
HARD-WON GOTCHAS (read before any long run)
These come from real lost benchmark days — see .dev-context/next-benchmark-checklist.md:
- Env-var prefixes do NOT work with bench.sh.
bench.sh unconditionally exports RANVIER_LOAD_AWARE_ROUTING=true, overwriting any VAR=x ./bench.sh prefix. Use the CLI flags: --no-load-aware, --load-imbalance-factor <N>, --load-imbalance-floor <N>, --cache-residency-threshold <N>, --client-tokenize, --multi-depth, --compare.
- Confirm the "Effective Routing Config" banner bench.sh prints at startup matches the intended experiment before letting a 30-minute run proceed. A mislabeled run is worse than no run.
- Routing has multiple diversion layers (prefix affinity, load-aware override, cache-residency override — unified scoring in
src/route_scorer.hpp). An A/B that toggles one layer while another stays active measures nothing. State explicitly which layers are on in each arm.
- Python/C++ coupling is silent. If the change touched
router_service.cpp, http_controller.cpp, request_rewriter.hpp, or metrics names, verify .dev-context/claude-locust-sync-map.md first — drift produces wrong metrics, not errors.
- SQLite state persists across runs. For clean comparisons, remove the container (not just stop it) to clear
/tmp/ranvier.db, re-register backends, and warm up identically.
EXPERIMENT DESIGN (A/B discipline)
- One variable per experiment. Same model, duration, user count, spawn rate, prompt distribution, and prefix ratio across arms.
- Decide the discriminating metric and the decision rule before running ("if p99 TTFT gap < X%, the override is not the cause").
- Do not write a code fix until the data points to one — run the smoking-gun experiment first.
- Record every run: date, commit (
git rev-parse --short HEAD), full command line, banner config, and the report path. Multi-run campaigns get a checklist file in .dev-context/ (follow next-benchmark-checklist.md as the format).
INTERPRETING RESULTS
Key metrics and where they come from:
- Cache-hit ratio —
ranvier_router metrics (:9180); the point of the product. Prefix mode target vs consistent-hash baseline is documented in docs/internals/prefix-affinity-routing.md (~81% vs 49%).
- TTFT p50/p99 — Locust report; the headline claim is ~48% TTFT improvement, so regressions here are release blockers.
- Affinity thrashing — cache-hit oscillation + backend flip-flopping under load; see
.dev-context/investigation-may22-affinity-thrashing-reproduction.md for the signature.
- Compare against
tests/integration/benchmark-baseline.json; results parser: tests/integration/results_parser.py.
OUTPUT FORMAT
- Experiment plan: arms, fixed variables, discriminating metric, decision rule
- Exact commands for each arm (flags, not env prefixes), plus reset steps between arms
- What to paste back (report files, banner output, metric snapshots)
- Interpretation once results arrive: verdict against the decision rule, and next step (fix / more data /
/incident if it's a regression)
