| name | llm-inference-serving |
| description | Deep LLM-inference-serving operational intuition — vLLM, SGLang RadixAttention, TensorRT-LLM tradeoffs; PagedAttention and KV-cache math, chunked prefill, prefix caching, disaggregated serving (DistServe, Mooncake), speculative decoding, quantization, MoE parallelism, TTFT/TPOT. Load when tuning serving-engine knobs, KV-cache sizing, batching design, quantization-format selection, speculative-decoding choice, or MoE layout. Skip for fine-tuning, embeddings serving, or hosted-API integration. Triggers on: "vLLM", "SGLang", "TensorRT-LLM", "PagedAttention", "RadixAttention", "chunked prefill", "prefix caching", "kv-cache-dtype", "EAGLE-3", "Medusa", "MLA", "AWQ", "NVFP4", "DistServe", "Mooncake", "EPLB", "TTFT", "TPOT", "goodput".
|
LLM Inference Serving Operational Guide
Concise operational pointers for deep LLM-serving troubleshooting and tuning.
Assumes you already know what an LLM, KV cache, and a token are. This skill covers the operational layer — the parts models tend to gloss over: V1-engine internals, KV-cache math (MHA/GQA/MLA), continuous-batching/chunked-prefill economics, prefill/decode disaggregation, speculative decoding tradeoffs, quantization-format choice, and MoE+parallelism layout — current as of late 2025/early 2026.
When to use
Load when the question is about:
- Tuning vLLM/SGLang/TGI/TRT-LLM serving knobs in production
- Sizing GPUs / KV-cache budget for a model + concurrency target
- Picking a quantization format for the deployment target (server vs edge)
- Diagnosing throughput-vs-latency regressions (TTFT vs ITL trade)
- Choosing between TP, PP, EP, DP for an MoE or dense model
- Deciding whether disaggregated prefill/decode is worth it
- Selecting a speculative-decoding strategy and predicting acceptance rate
- Long-context tuning (RoPE scaling, sliding window, chunked prefill)
- KV-cache memory math when adding GQA/MLA models
- Prefix-caching impact on RAG / multi-turn / system-prompt-heavy workloads
- Benchmark interpretation: goodput vs raw throughput, P99 latency
- Scaling MoE serving (Mixtral, DeepSeek-V3/R1) across nodes
- Deciding when to keep llama.cpp / GGUF vs server-grade engines
- Migrating between vLLM V0 and V1 engine
Do NOT load for: training / fine-tuning, embedding or reranker serving, agent / tool-calling protocol design, hosted-API integration, plain "what's an LLM" questions. For training fundamentals see dl-training; for fine-tuning see fine-tuning-llms.
vLLM and the V1 engine
V1 is the default engine; V0 is deprecated. V1 unifies prefill and decode under one scheduler with a {request_id: num_tokens} budget. EngineCore runs in its own process; scheduler and worker 0 are decoupled (symmetric TP); Persistent Batch caches input tensors and applies diffs.
Key knobs and how they interact:
--gpu-memory-utilization (default 0.9) — fraction of GPU memory for the executor. KV cache pool ≈ this fraction × VRAM − weights − activations − buffers.--max-num-seqs (default 256 in V1) — max concurrent sequences (decode parallelism cap).--max-num-batched-tokens (default ≈8192 in V1; was 2048 in V0) — per-iteration token budget shared by prefill + decode chunks. Lower → better ITL; higher → better TTFT/throughput. Sweep 8k–64k.--enable-chunked-prefill (default True in V1) — splits long prompts into chunks so a giant prefill doesn't stall queued decodes.--enable-prefix-caching (default True in V1) — hash-keyed shared KV blocks across requests. Massive win for RAG / system-prompt / multi-turn; near-zero loss otherwise.--kv-cache-dtype fp8 (E4M3 by default on Hopper+) — halves KV bytes vs FP16 with negligible quality loss for most models.--quantization {fp8,awq,gptq,awq_marlin,gptq_marlin,nvfp4,...} — weight scheme. Marlin/Machete kernels accelerate GPTQ/AWQ vs naïve dequant.--tensor-parallel-size, --pipeline-parallel-size, --data-parallel-size, --enable-expert-parallel.--speculative-config '{...}' (V1 unified) replaces the older --speculative-model.
KV-token capacity ≈ max-num-seqs × max-model-len. Setting max-model-len=128k with max-num-seqs=256 blows the KV budget on a single H100 — drop one. gpu-memory-utilization ≥ 0.95 leaves no headroom for activations under bursty prefill; OOM appears under load, not at boot.
KV-cache math
Per-token MHA bytes: 2 × n_layers × n_kv_heads × head_dim × bytes_per_elem.
Worked: Llama 3 70B (80 layers, 8 KV heads via GQA, head_dim 128, BF16) = 2 × 80 × 8 × 128 × 2 = 327,680 B ≈ 0.31 MB/token. At 100K tokens that's ~31 GB just for one sequence.
GQA collapses n_kv_heads from n_query_heads → n_groups (Llama 3: 64→8 = 8× KV reduction).
MLA (DeepSeek V2/V3/R1) replaces per-head K/V with a per-token shared low-rank latent (d_c ≈ 512) plus a small RoPE key (d_R ≈ 64). Bytes ≈ batch × seq × n_layers × (d_c + d_R) × bytes. DeepSeek-V3 at FP16 ≈ ~70 KB/token vs ~860 KB on a hypothetical MHA equivalent (~60× reduction, ~12× vs comparable GQA). Why DeepSeek can serve 128K cheaply and why vLLM treats MLA via a separate hybrid_kv_cache_manager.
KV quant: FP8 E4M3 cuts bytes 2×. NVFP4 KV (Blackwell) halves again with micro-block scaling (block 16 + FP8 scale + per-tensor FP32 scale).
Batching evolution
- Static: all in-batch requests run to completion together. Padding waste, head-of-line blocking.
- Dynamic (Triton-style): wait window then batch. Predictable latency, lower throughput.
- Continuous (iteration-level) (Orca 2022; vLLM/SGLang/TGI/TRT-LLM): scheduler reconsiders the batch every forward step. Finished sequences leave; new ones enter freed KV slots immediately. Large reported speedups vs static / HF Transformers baselines.
- Chunked prefill: split a 32K-prompt prefill into per-step chunks sharing
max-num-batched-tokens with decodes. Eliminates head-of-line ITL spikes; smooths P99. Default-on in V1.
Prefill vs decode and disaggregated serving
- Prefill: compute-bound; full S×S attention per head per layer; saturates Tensor Cores.
- Decode: memory-bandwidth-bound; one token per step requires a full pass over weights and the entire growing KV. Tensor cores idle, HBM saturated.
Co-locating both wastes the resource the other phase needs. PD disaggregation runs prefill and decode on separate GPU pools and transfers KV across; goal is to satisfy TTFT and TPOT independently and optimize goodput (requests/s meeting SLO), not raw throughput. Specific systems (Splitwise, DistServe, Mooncake, etc.) churn fast — pick what your engine integrates with at the time.
KV transfer is hundreds of MB/request — the new bottleneck. RDMA preferred over NCCL for cross-pod transfers. When worth it: medium-to-long prompts, strict TTFT SLO, concurrency high enough that PD interference is real. Skip for short-prompt high-frequency chat — KV-transfer overhead dominates.
Speculative decoding
Verifier runs N draft tokens per step, accepts the longest matching prefix. Speedup ≈ accepted_length × target_step_cost / (target_step_cost + draft_cost).
- Draft model: small same-family LM (e.g., 1B drafting 70B). Quality drives acceptance.
- N-gram / prompt lookup: matches recent generated/prompt tokens; zero training. Acceptance 0.75–0.85 on code/structured/repetitive output, 0.5–0.65 on creative.
- Medusa: extra prediction heads on the target; trains heads, no separate draft.
- EAGLE / EAGLE-2 / EAGLE-3: predicts at the feature level (residual stream); reuses target features. EAGLE-3 was the current SOTA at last update — re-check before committing.
- MTP (multi-token prediction, native in DeepSeek-V3) treated as in-model spec decode.
- PARD / MLP / suffix decoding: lighter alternatives.
vLLM CLI: --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_min":2,"prompt_lookup_max":5}' or '{"method":"eagle3","model":"…","num_speculative_tokens":2}'.
Footgun: under high concurrency the batch fills naturally — speculation eats compute that would have served other requests, net throughput drops. Use spec decode for low-concurrency latency-sensitive workloads, not at saturation.
Quantization formats
Server-grade:
- FP8 W8A8 (E4M3 weights+activations): vLLM
--quantization fp8 on Hopper (cc ≥ 8.9) and Ada. E4M3 (range ±448) for weights/activations; E5M2 (range ±57344) historically for grads. Per-tensor scale standard; per-channel for sensitive layers.
- NVFP4 (Blackwell-class GPUs): block-16 microscaling with FP8 (E4M3) block scales + FP32 per-tensor. Better accuracy than MXFP4 (block 32) at comparable throughput.
- AWQ (4-bit) + Marlin: weight-only INT4, activation-aware. Often highest throughput on GPU; quality close to FP16 for ≥7B.
- GPTQ + Marlin: similar 4-bit with Hessian calibration. GPTQModel emits Marlin/Machete-ready checkpoints.
- INT8 W8A8 (SmoothQuant): less common since FP8 landed on Hopper; useful pre-Hopper.
- BNB load-in-4bit / 8bit: Transformers-side, slow; prototyping only.
Edge / CPU / Mac:
- GGUF (llama.cpp). Q4_K_M is the sweet spot (mixed 4-bit with selective higher precision); Q5_K_M for headroom; Q3_K_S/Q2_K when you must squeeze. CPU, Apple Metal, Vulkan, CUDA, ROCm.
Per-channel scales > per-tensor on sensitivity-prone layers (e.g., down_proj); per-tensor cheaper, often fine elsewhere.
Frameworks compared
- vLLM (Apache 2.0): broadest model coverage, V1 engine, strong NVIDIA + growing AMD/Inferentia/TPU. Default for general-purpose serving; wins on heterogeneous fleets and unique-prompt workloads.
- SGLang (Apache 2.0): RadixAttention reuses prefix-tree KV across siblings — meaningful throughput edge over vLLM at modest sizes, large gains on prefix-heavy RAG/multi-turn, modestly lower TTFT P95. Caveat: Python-router GIL ceiling at very high concurrency. Default for DeepSeek, structured output, multi-turn agentic.
- TensorRT-LLM (NVIDIA): peak Hopper/Blackwell perf; ahead-of-time engine compile; first-class NVFP4. Best raw throughput on NVIDIA; least flexible; engine rebuild on every config change.
- TGI (HuggingFace): maintenance mode since late 2025 — bug fixes only. Has TRT-LLM/vLLM backends. Pick only for legacy compatibility.
- llama.cpp: GGUF native; CPU + Metal + Vulkan + CUDA + ROCm; tiny binaries; no continuous batching at server-grade quality. Best on Mac/edge/single-user.
- MLC-LLM (TVM-based): broad hardware (Vulkan, WebGPU, mobile); compile-once-run-anywhere; weaker server batching.
Parallelism and MoE
- TP shards weight matrices intra-layer; AllReduce per layer. Needs NVLink/NVSwitch for ≥8-way; degrades fast on PCIe.
- PP cuts the model by layers; AllReduce-free but bubble unless micro-batched.
- EP shards experts across ranks for MoE; AllToAll (not AllReduce). Pair with DP-attention for the modern wide-EP MoE pattern.
- DP replicates.
MoE serving: Mixtral-style 8-expert models are straightforward TP. Wide-MoE designs (DeepSeek-V3/R1 with hundreds of experts, MLA) are the hard case — the bottleneck is per-step expert load imbalance, addressed by hot-expert duplication / load-balancing schemes (EPLB-family). Capacity factor caps per-expert tokens-per-step; over-cap tokens drop or reroute. Specific layout/balancer choices change quarterly — re-check what your engine ships.
Long context, latency goals
- Long-context techniques: PagedAttention page tables for non-contiguous KV; sliding-window attention gets a dedicated KV-cache group in vLLM's hybrid manager so windowed and full-attention layers share blocks without waste. RoPE scaling:
rope_scaling={"type":"dynamic","factor":2.0}, linear, yarn, llama3 for trained extensions. ChunkAttention: prefix-tree-aware FlashAttention.
- Latency targets: TTFT (prefill-dominated), TPOT/ITL (decode-dominated). Production: TTFT P99 < 500–1000 ms for chat; ITL P99 < 50–100 ms (≥ 10 tok/s feels live). Goodput = requests/s meeting both SLOs simultaneously — what DistServe/Mooncake actually optimize. Naive throughput hides that batching too aggressively wins tokens/s but blows TTFT P99.
Authoritative references
Engine docs:
Speculation, kernels:
Guardrails
Before recommending a non-trivial serving change (KV dtype, quantization scheme, parallelism layout, spec-decoding method, disaggregation):
- Quote the engine flag/parameter and its current default
- Cite the engine's docs / release blog for the version in use
- Make the recommendation conditional on observed metrics (TTFT/ITL P99, KV-cache hit rate, expert-utilization variance) — never blanket-tune
- Verify the engine version — defaults shift fast in this space.
Tuning without measurement is worse than defaults.
