Three techniques every production serving system on NVIDIA ships with — confirm all three are in place before pursuing workload-specific optimizations:
-
references/models/image-generation.md — Image generation serving — diffusion (U-Net, DiT) and flow-matching.
-
references/models/omni-multimodal.md — Omni-modal serving — multi-modality in AND out.
-
references/models/speech-generation.md — Speech generation serving — TTS and speech-to-speech.
-
references/models/speech-language.md — Speech-language serving — audio understanding (ASR, speech translation, audio-text chat).
-
references/models/ssm-hybrid.md — State-space and hybrid SSM+attention serving — Mamba/Mamba2, Jamba, Zamba/Zamba2, Nemotron-H (dense and hybrid), Jet-Nemotron, Falcon-Mamba.
-
references/models/text-dense.md — The foundational architecture most modern LLMs build on.
-
references/models/text-moe.md — Mixture-of-Experts text decoders for serving — coarse-grained MoE (Mixtral 8x7B, 8x22B) to fine-grained MoE with shared experts (DeepSeek V2/V3/R1, Qwen3-MoE 30B-A3B / 235B-A22B, Llama-4 Scout/Maverick) and DeepSeek's MLA+MoE+MTP stack.
-
references/models/video-generation.md — Video generation serving — diffusion-based with 3D attention, temporal+spatial denoising, large activations.
-
references/models/vision-language.md — Vision-language serving — LLaVA / LLaVA-NeXT / LLaVA-OneVision (fixed and dynamic tiling), Qwen-VL / 2-VL / 2.5-VL / 3-VL (native-resolution ViT + M-RoPE), InternVL, mllama (Llama-3.2 Vision, cross-attention), Molmo, DeepSeek-VL.
-
references/algorithms/async-scheduling.md — Hide CPU scheduler / model-runner / Python overhead behind the GPU forward.
-
references/algorithms/attention-variants.md — Attention variants in modern serving — three orthogonal axes: head sharing (MHA / MQA / GQA / MLA), masking pattern (causal / bidirectional / sliding-window / cross-attention / 3D / tree), complexity class (quadratic vs SSM / linear / RetNet / RWKV / hybrid).
-
references/algorithms/batched-sampling.md — Efficient batched sampling on GPU for LLM serving — temperature, top-p, top-k, min-p, repetition / presence / frequency penalties, typical, rejection sampling for combined filters — without per-request CPU-GPU sync.
-
references/algorithms/chunked-prefill.md — Chunked prefill scheduling — split long prompts into smaller token chunks interleaved with decode requests in a single forward pass, preventing a long prefill from stalling decode latency.
-
references/algorithms/continuous-batching.md — Implement continuous batching for an LLM inference server.
-
references/algorithms/disaggregated-serving.md — Disaggregated prefill / decode (P/D) serving — separate worker pools for prefill and decode stages, with KV cache transferred between them.
-
references/algorithms/heterogeneous-kv-cache.md — Memory management and prefix caching for hybrid models (full-attn + sliding-window, attention + SSM/Mamba, attention + linear).
-
references/algorithms/moe-routing-dispatch.md — MoE routing and dispatch for serving — top-k gating (softmax, group-limited, auxiliary-loss-free), token-to-expert dispatch/combine (padding, permutation, DeepEP all-to-all), grouped-GEMM expert FFN, Marlin-MoE / CUTLASS-MoE kernels, expert parallelism (EP), expert load balancing (EPLB), shared experts.
-
references/algorithms/paged-attention.md — Paged KV cache design for LLM serving — block-based non-contiguous KV storage with a page table per request, allowing dynamic growth without fragmentation.
-
references/algorithms/parallelism.md — Parallelism strategies for LLM serving — TP, PP, EP, DP, SP, and combos (TP+EP, DP-attention + EP-MoE, TP+SP, TP+PP, context parallel).
-
references/algorithms/quantization-schemes.md — Quantization schemes for LLM serving — FP8 (E4M3/E5M2, per-tensor vs per-channel vs block), INT4 weight-only (AWQ, GPTQ, Marlin, GGUF, petit), INT8, FP4 (MXFP4, NVFP4), FP quant mixed-precision (qutlass / modelopt), weight-only vs.
-
references/algorithms/radix-prefix-caching.md — Prefix / radix KV cache reuse — share KV cache pages across requests with common prefixes via a radix tree, LRU eviction.
-
references/algorithms/speculative-decoding.md — Speculative decoding for LLM serving — draft proposals verified in one target pass.
-
references/algorithms/structured-output.md — Structured output at serving time — grammar-guided decoding (XGrammar, Outlines, llguidance, lm-format-enforcer), JSON mode, regex-constrained output, CFG / pushdown grammars, tool / function calling, logits biasing.
-
references/frameworks/mlx.md — MLX framework for LLM serving on Apple Silicon — unified-memory array model, lazy evaluation and mx.eval, mx.compile, mx.fast kernels, mlx-lm reference serving path, native INT4/INT8 quantization via mx.quantize, custom Metal kernels via mx.fast.metal_kernel.
-
references/frameworks/pytorch.md — PyTorch idioms for LLM serving — nn.Module + weight loading, torch.compile (Dynamo / inductor / dynamic shapes / reduce-overhead), state_dict remapping, custom op registration with torch.library, NCCL setup, HF transformers integration, inference_mode.
-
references/frameworks/neuron-pytorch.md — PyTorch on AWS Trainium (Neuron) — torch-neuronx vs TorchNeuron Native, neuronx-cc compilation, static-shape bucketing, persistent compile cache, BF16, static KV cache, host-sync pitfalls, when to drop to NKI.
-
references/frameworks/nxd-inference.md — NxD Inference (NeuronX Distributed Inference) — AWS's Trainium inference library (NeuronConfig / ModelBuilder / generate); full-turnkey or as building blocks inside a bespoke server.
-
references/frameworks/nxd-kv-cache.md — NxD KVCacheManager — the device-resident, in-place KV cache for Trainium (resident nn.Parameter buffers + ModelBuilder input/output aliasing). The key decode optimization; what raw torch_neuronx.trace can't do.
-
references/frameworks/neuron-flash-attention.md — flash attention on Trainium via the NKI kernel nki_flash_attn_func. Flash attention is NOT NVIDIA-only — the algorithm ships on Neuron; use it instead of naive softmax(QKᵀ)V to cut the activation peak (and the HBM OOM that caps batch size).
-
references/frameworks/triton.md — Triton as a framework-level decision — when a custom Triton kernel pays off in serving vs reusing FlashInfer / FlashAttention / CUTLASS / liger / sgl-kernel.
-
references/tooling/accuracy-checker.md — Create an accuracy verification script that compares a custom generation implementation against HuggingFace model.generate() as the reference.
-
references/tooling/fastapi-serving.md — Create a production-ready FastAPI inference server for HuggingFace transformer models.
-
references/tooling/io-handling.md — Request-time I/O handling for LLM/multimodal serving — tokenization and chat templates, tool-call prompt formatting, image preprocessing (resize / tile / normalize / patchify), video frame sampling, audio feature extraction (log-mel), detokenization and UTF-8-safe streaming, tool-call parsing per model family, structured-output extraction.
-
references/tooling/lora-serving.md — Multi-adapter LoRA serving — single base model dispatching different LoRA adapters per request at serving throughput.
-
references/tooling/openai-api.md — OpenAI-compatible HTTP per modality — text (/v1/completions, /v1/chat/completions), image (/v1/images/generations), TTS (/v1/audio/speech), STT (/v1/audio/transcriptions), video (/v1/videos + async polling), realtime audio (WS /v1/realtime).
-
references/tooling/profiler.md — GPU performance profilers for LLM/multimodal serving: PyTorch Profiler (Python-level op aggregation, Chrome trace), Nsight Systems (nsys, system-timeline / launch-gap / NCCL diagnosis), Nsight Compute (ncu, kernel metrics / occupancy / roofline).
-
references/tooling/serving-benchmark.md — Benchmark an LLM serving endpoint — TTFT, TPOT, ITL, end-to-end latency, throughput, p50/p95/p99 across concurrency and ISL/OSL sweeps.
Kernel implementation (writing CUDA / Triton / CUTLASS). For that, use the separate agent-gpu-skills collection. Exception — NKI: writing NeuronCore kernels for AWS Trainium is in scope here, via the bundled neuron-nki-* skills (neuron-nki-writing, -docs, -debugging, -profiling, -profile-querying); there is no separate Trainium kernel collection.
