Menu
Estimate LLM inference latency, throughput, and memory requirements. Roofline model, batch sizing, KV cache analysis.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Validate GRPO training configs against 36 cross-framework bug avoidance rules (19 MUST DO + 17 MUST NOT). Covers 7 frameworks (DeepSpeed, Megatron, vLLM, verl, SGLang, rLLM, MindIE), 7 pattern classes, and MoE NaN avoidance.
Debug and diagnose GRPO training issues across verl, Megatron, DeepSpeed, and RTX 4090. Symptom-to-bug mapping, NaN debugging, Muon clipping avoidance, and CUDA stream safety analysis.
Navigate SGLang source code. Quick reference for subsystem locations, key classes, and data flow.
Navigate mechanistic interpretability concepts, tools, and SAE architectures. Quick reference for SAE variants, feature steering methods, circuit discovery, and trust calibration.
Navigate model merging strategies and theory. Quick reference for Task Arithmetic, DARE, TIES merging methods, alpha selection, and RTX 4090 validation results.
Navigate RDMA, InfiniBand, RoCE v2, AI cluster networking topology, and NCCL transport internals. Quick reference for Verbs API, QP/CQ/SRQ/MR, fat-tree/rail-aligned topology, DCQCN congestion control, GPUDirect RDMA, and cluster configuration.
| name | inference-perf |
| description | Estimate LLM inference latency, throughput, and memory requirements. Roofline model, batch sizing, KV cache analysis. |
| triggers | ["inference latency","throughput estimate","roofline model","inference memory","KV cache size","推理性能","latency estimate"] |
Latency per token = (2 × params × bytes_per_param) / HBM_bandwidth
Throughput = batch_size / latency_per_token
TTFT = (2 × params × seq_len) / (TFLOPS × utilization)
KV per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element
Total KV = KV_per_token × seq_len × batch_size
Examples (FP16):
LLaMA-7B: 2×32×32×128×2 = 512KB/token
LLaMA-70B: 2×80×8×128×2 = 320KB/token (GQA saves 8x!)
7B@128K: 512KB × 128K = 64GB (> model weight!)
Ridge Point = Peak TFLOPS / HBM BW (ops/byte)
A100: 312/2.0 = 156 ops/byte
H100: 990/3.35 = 148 ops/byte
A16: 14.7/0.170 = 86.5 ops/byte
Arithmetic Intensity:
Decode: AI ≈ 1.0 (always memory-bound, only 0.7% peak FLOPS on A100)
Prefill: AI = seq_len (compute-bound when seq > ridge_point)
| GPU | FP16 TFLOPS | HBM GB/s | Memory | NVLink |
|---|---|---|---|---|
| A16 | 14.7 | 170 | 15GB | No (PCIe) |
| A100 | 312 | 2000 | 80GB | 600 GB/s |
| H100 | 990 | 3350 | 80GB | 900 GB/s |
| H200 | 990 | 4800 | 141GB | 900 GB/s |
Max batch = available_memory / KV_per_token_per_seq
A100 80GB, 7B FP16: (80-14)GB / 512KB ≈ 128K tokens
A16 15GB, 125M FP16: (15-0.25)GB / 128KB ≈ 116K tokens
Throughput vs Batch:
batch=1: 2% peak throughput
batch=32: ~50% peak
batch=128: ~85% peak
batch=512: ~95% peak (diminishing returns after 128)
Decode TP overhead ≈ 11.5% (constant, independent of batch size)
Because: AllReduce size = hidden_dim × batch × 2B, ~constant ratio
TP efficiency:
TP=2: ~97%
TP=4: ~95%
TP=8: ~94% (405B)
Speedup ≈ K / (1 + draft_cost_ratio × K)
K=3, draft=0.1x: 2.3x speedup
K=5, draft=0.1x: 3.3x speedup
K=5, draft=0.5x: 2.0x speedup (diminishing returns)
Acceptance rate depends on distribution sharpness, NOT temperature.
Savings = prefix_hit_rate × (prefix_len / total_len) × 100%
Scenarios:
System prompt: 100% hit rate (fixed prefix)
Multi-turn (16 rounds): 80% cumulative hit
GRPO (n=8, prompt=512): 67% KV saved
Minimum ROI: prefix > 128 tokens
$/M tokens = (GPU_cost/hr × 1000) / (throughput_tok/s × 3600)
Examples:
7B FP8 on A100: ~$0.24/M tok
Mixtral FP8 on H200: ~$0.09/M tok (MoE sparse)
A100 most cost-efficient for dense models
tools/gpu_inference_sim.py — Full inference simulationtools/gpu_perf_model.py — Performance model with rooflinetools/gpu_memory_calculator.py — Memory calculatortools/inference_cost_calculator.py — Cost calculatortools/llm_latency_estimator.py — Latency estimator