메뉴
Estimate LLM inference latency, throughput, and memory requirements. Roofline model, batch sizing, KV cache analysis.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Validate GRPO training configs against 36 cross-framework bug avoidance rules (19 MUST DO + 17 MUST NOT). Covers 7 frameworks (DeepSpeed, Megatron, vLLM, verl, SGLang, rLLM, MindIE), 7 pattern classes, and MoE NaN avoidance.
Debug and diagnose GRPO training issues across verl, Megatron, DeepSpeed, and RTX 4090. Symptom-to-bug mapping, NaN debugging, Muon clipping avoidance, and CUDA stream safety analysis.
Navigate SGLang source code. Quick reference for subsystem locations, key classes, and data flow.
Navigate mechanistic interpretability concepts, tools, and SAE architectures. Quick reference for SAE variants, feature steering methods, circuit discovery, and trust calibration.
Navigate model merging strategies and theory. Quick reference for Task Arithmetic, DARE, TIES merging methods, alpha selection, and RTX 4090 validation results.
Navigate RDMA, InfiniBand, RoCE v2, AI cluster networking topology, and NCCL transport internals. Quick reference for Verbs API, QP/CQ/SRQ/MR, fat-tree/rail-aligned topology, DCQCN congestion control, GPUDirect RDMA, and cluster configuration.
| name | inference-perf |
| description | Estimate LLM inference latency, throughput, and memory requirements. Roofline model, batch sizing, KV cache analysis. |
| triggers | ["inference latency","throughput estimate","roofline model","inference memory","KV cache size","推理性能","latency estimate"] |
Latency per token = (2 × params × bytes_per_param) / HBM_bandwidth
Throughput = batch_size / latency_per_token
TTFT = (2 × params × seq_len) / (TFLOPS × utilization)
KV per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element
Total KV = KV_per_token × seq_len × batch_size
Examples (FP16):
LLaMA-7B: 2×32×32×128×2 = 512KB/token
LLaMA-70B: 2×80×8×128×2 = 320KB/token (GQA saves 8x!)
7B@128K: 512KB × 128K = 64GB (> model weight!)
Ridge Point = Peak TFLOPS / HBM BW (ops/byte)
A100: 312/2.0 = 156 ops/byte
H100: 990/3.35 = 148 ops/byte
A16: 14.7/0.170 = 86.5 ops/byte
Arithmetic Intensity:
Decode: AI ≈ 1.0 (always memory-bound, only 0.7% peak FLOPS on A100)
Prefill: AI = seq_len (compute-bound when seq > ridge_point)
| GPU | FP16 TFLOPS | HBM GB/s | Memory | NVLink |
|---|---|---|---|---|
| A16 | 14.7 | 170 | 15GB | No (PCIe) |
| A100 | 312 | 2000 | 80GB | 600 GB/s |
| H100 | 990 | 3350 | 80GB | 900 GB/s |
| H200 | 990 | 4800 | 141GB | 900 GB/s |
Max batch = available_memory / KV_per_token_per_seq
A100 80GB, 7B FP16: (80-14)GB / 512KB ≈ 128K tokens
A16 15GB, 125M FP16: (15-0.25)GB / 128KB ≈ 116K tokens
Throughput vs Batch:
batch=1: 2% peak throughput
batch=32: ~50% peak
batch=128: ~85% peak
batch=512: ~95% peak (diminishing returns after 128)
Decode TP overhead ≈ 11.5% (constant, independent of batch size)
Because: AllReduce size = hidden_dim × batch × 2B, ~constant ratio
TP efficiency:
TP=2: ~97%
TP=4: ~95%
TP=8: ~94% (405B)
Speedup ≈ K / (1 + draft_cost_ratio × K)
K=3, draft=0.1x: 2.3x speedup
K=5, draft=0.1x: 3.3x speedup
K=5, draft=0.5x: 2.0x speedup (diminishing returns)
Acceptance rate depends on distribution sharpness, NOT temperature.
Savings = prefix_hit_rate × (prefix_len / total_len) × 100%
Scenarios:
System prompt: 100% hit rate (fixed prefix)
Multi-turn (16 rounds): 80% cumulative hit
GRPO (n=8, prompt=512): 67% KV saved
Minimum ROI: prefix > 128 tokens
$/M tokens = (GPU_cost/hr × 1000) / (throughput_tok/s × 3600)
Examples:
7B FP8 on A100: ~$0.24/M tok
Mixtral FP8 on H200: ~$0.09/M tok (MoE sparse)
A100 most cost-efficient for dense models
tools/gpu_inference_sim.py — Full inference simulationtools/gpu_perf_model.py — Performance model with rooflinetools/gpu_memory_calculator.py — Memory calculatortools/inference_cost_calculator.py — Cost calculatortools/llm_latency_estimator.py — Latency estimator