SLA-first GPU capacity sizing for a serving deployment: given a tokens-per-minute (TPM) target AND the interactivity SLA (output tokens/s/user), compute the pods and GPUs needed from a model's measured tok/s/user-vs-concurrency curve. Sizing MUST start from…
Measure the low-concurrency (c=1..c=8) decode hot-path of a live vLLM pod FAST and CORRECTLY: where each token's time actually goes (GPU-busy vs host-idle gap vs comm), whether the workload is kernel-bound / host-bound / comm-bound, and what the addressable…
Capture per-kernel CUDA hardware-counter data (occupancy, achieved warps active, regs/thread, smem/block, DRAM throughput, arithmetic intensity, branch divergence %, warp-stall reason %) from a live vLLM inference pod via an `ncu` debug sidecar (no production…
Capture per-kernel CUDA profile data from a live vLLM inference pod via an nsys debug sidecar (no production image rebuild). Outputs `.nsys-rep` + summary CSV + top-kernels table that joins with the zymtrace per-kernel breakdown and the inference-perf-bench…
End-to-end orchestrator that takes a NEW model from a bare HuggingFace id to a perf-lake-published, validated CROSS-ENGINE (vLLM + SGLang) champion on B200/GB300. Scaffolds a per-model harness + evidence bundle (run-id == experiment-id), then drives…
Canonical inference perf-bench skill (formal name. The colloquial alias is `ai-bench` - identical behaviour). Drives NVIDIA AIPerf + the replay-playback dataset against an in-cluster vLLM endpoint to measure TTFT, ITL, throughput, tok/s/user, request latency,…
Applies Jeff Dean and Sanjay Ghemawat's Performance Hints to GPU inference. Builds a back-of-the-envelope cost ledger, identifies the hot-path class, bounds each optimization with profile share, and routes the highest-value hypothesis to the repo's benchmark,…
Search a vLLM serving model's config space for its best inference performance: a `perftunereport campaign_run` matrix sweep over concurrency x max_num_batched_tokens x kv-cache-dtype x cudagraph_mode, plus a same-node controlled A/B (`run-controlled-ab.sh`)…