메뉴
Plan, run, and interpret Locust load benchmarks (mock-backend and real-GPU) including A/B routing experiments. Use when the user wants to measure cache-hit ratio, TTFT, or throughput, compare routing modes, chase a benchmark regression, or design a benchmark experiment.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Hostile security and robustness audit of src/ assuming malicious traffic and 100x load. Use before releases, after large feature merges, or when the user asks for a security audit, adversarial review, or "what breaks under attack".
Holistic system audit for architecture drift, Hard Rules compliance, and async integrity across recently changed code. Use after completing a batch of tasks, before a release, or when the user asks for a health check or audit.
Semantic-correctness audit of hot-path code - enumerate the invariants a component must maintain, then hunt code paths that violate them (wrong answers, not crashes). Use after changing routing, eviction, accounting, or counter logic, before a release, or when metrics drift without any crash.
Orientation for a session starting work on this codebase - reading order, where things live, which skill to use for which task, and the non-negotiable constraints. Use at the start of a session, when unsure where to look, or when the user asks how to approach a task here.
Triage build or runtime failures. Use when the user pastes a compile error, link error, runtime crash, test failure, or error log, or mentions a build failure.
Post-implementation maintenance - write unit tests, sync documentation, and update BACKLOG.md. Use after code is implemented and reviewed, or when the user asks for tests or doc updates for a change.
| name | benchmark |
| description | Plan, run, and interpret Locust load benchmarks (mock-backend and real-GPU) including A/B routing experiments. Use when the user wants to measure cache-hit ratio, TTFT, or throughput, compare routing modes, chase a benchmark regression, or design a benchmark experiment. |
| argument-hint | ["what-to-measure"] |
Goal: $ARGUMENTS
Ref:
.dev-context/claude-context.mdfor architecture. Benchmarks run on the developer's machine or GPU rig, never in this sandbox — your deliverables are the exact commands, the experiment design, and the interpretation of results the developer pastes back.
| Runner | Command | What it measures |
|---|---|---|
| Mock-backend load test | make benchmark | Routing/proxy behavior without GPUs. Docker Compose benchmark profile; builds images inline (set SKIP_BUILD=1 to reuse); reports to benchmark-reports/ |
| CI regression | .github/workflows/benchmark.yml | Same, on PRs and main; baseline in tests/integration/benchmark-baseline.json |
| Real GPU A/B | scripts/bench.sh | TTFT / cache-hit against real vLLM backends |
| Regression bisect | tests/integration/bisect-benchmark.sh | Which commit moved the number |
| Micro/EPP | make bench-hot-prefix, make bench-epp, make bench-inline-vs-sidecar | Component-level costs |
These come from real lost benchmark days — see .dev-context/next-benchmark-checklist.md:
bench.sh unconditionally exports RANVIER_LOAD_AWARE_ROUTING=true, overwriting any VAR=x ./bench.sh prefix. Use the CLI flags: --no-load-aware, --load-imbalance-factor <N>, --load-imbalance-floor <N>, --cache-residency-threshold <N>, --client-tokenize, --multi-depth, --compare.src/route_scorer.hpp). An A/B that toggles one layer while another stays active measures nothing. State explicitly which layers are on in each arm.router_service.cpp, http_controller.cpp, request_rewriter.hpp, or metrics names, verify .dev-context/claude-locust-sync-map.md first — drift produces wrong metrics, not errors./tmp/ranvier.db, re-register backends, and warm up identically.git rev-parse --short HEAD), full command line, banner config, and the report path. Multi-run campaigns get a checklist file in .dev-context/ (follow next-benchmark-checklist.md as the format).Key metrics and where they come from:
ranvier_router metrics (:9180); the point of the product. Prefix mode target vs consistent-hash baseline is documented in docs/internals/prefix-affinity-routing.md (~81% vs 49%)..dev-context/investigation-may22-affinity-thrashing-reproduction.md for the signature.tests/integration/benchmark-baseline.json; results parser: tests/integration/results_parser.py./incident if it's a regression)