sglang-diffusion-performance

# SGLang Diffusion Performance Tuning Use this skill when the user wants the fastest command line, lower VRAM, or the right performance flags for a specific model and GPU setup. Before running any `sglang generate` command below inside the diffusion container: - use `python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/diffusion_skill_env.py` to derive the repo root, verify write access, and choose idle GPU(s) - export `HF_TOKEN` first when the selected model lives in a gated Hugging Face repo such as `black-forest-labs/FLUX.*` - export `FLASHINFER_DISABLE_VERSION_CHECK=1` - when a run downloads weights, use a task-owned cache and delete that model's cache after its eager/BCG/quality/profile group finishes; the benchmark skill's `--quality-bcg-matrix --model-cache-root --cleanup-model-cache` keeps one cache for the group and writes a zero-residual cleanup ledger - hold one idle GPU set for the complete A/B matrix and verify no foreign process appears at run boundaries - `cd` to the repo root resolved from `sglang.__file__` ## Native Backend Gate Performance numbers are useful only when the intended backend actually ran. - Treat any log containing `Falling back to diffusers backend`, `Using diffusers backend`, or `Loaded diffusers pipeline` as invalid for native SGLang performance tuning. - Use `--backend diffusers` only for an explicit diffusers baseline. For native recipes, leave the default backend or pin `--backend sglang`. - If a fallback happened, fix pipeline registration/model-path/config issues first, then rerun. Do not compare perf dumps collected from a fallback run. - When the runtime auto-selects parallel settings because the user omitted them, keep the result as an auto-tuned baseline. For reproducible tuning, pin `--num-gpus`, `--ulysses-degree`, `--ring-degree`, and `--enable-cfg-parallel` explicitly. Reference: [SGLang-Diffusion Advanced Optimizations Blog](https://lmsys.org/blog/2026-02-16-sglang-diffusion-advanced-optimizations/) --- ## Section 1: Lossless Optimizations These options are intended to preserve output quality. In practice, some paths (most notably `torch.compile`) can still introduce small floating-point drift, so validate on your target model when numerical parity matters. | Option | CLI Flag / Env Var | What It Does | Speedup | Limitations / Notes | |---|---|---|---|---| | **Performance Mode** | `--performance-mode auto\|speed\|memory\|manual` (`--mode` alias) | Applies model-aware residency, FSDP/CFG, and compile defaults without overriding explicit flags. `auto` is the safe default; `speed` favors GPU residency; `memory` favors offload; `manual` leaves performance args explicit. | Fastest way to establish a sensible deployment baseline | `speed` may OOM and enables `torch.compile` only when the model deployment config allows it. Explicit offload/FSDP/parallelism/compile flags win. Use `manual` for controlled A/B benchmarks. | | **torch.compile** | `--enable-torch-compile` | Applies `torch.compile` to the DiT forward pass. Treat it as a measured comparator, not an assumed upgrade. | Model- and shape-dependent; recent B300 coverage found eager or valid BCG faster or within 1% for every valid compile control | First request is slow and some models time out or drift numerically. Keep eager as the ground truth, use a warmup watchdog, and validate the target model. See the [H200/B300 survey](https://github.com/BBuf/how-to-optim-algorithm-in-cuda/issues/21). | | **Breakable CUDA Graph** | `--enable-breakable-cuda-graph` plus optional `--warmup-resolutions <WxH...>` and `--bcg-text-buckets ...` | Captures fixed-resolution DiT segments while leaving attention/collectives eager, reducing launch overhead on supported pipelines. | Large on launch-bound paths; merged SANA and LTX-2 cases show material e2e gains | Mutually exclusive with `torch.compile` and Cache-DiT; BCG takes priority. The model's default resolution is captured automatically; declare every additional production resolution. Current support is model-specific (Ideogram4, LTX-2/2.3, LongCat-Image, MiniMax-H3, Qwen-Image, SANA1.5, SANA-Video, Z-Image, GLM-Image), but an allowlisted model is not automatically a validated recipe. A valid run must log capture and no disable/failure/signature miss. `--warmup-resolutions` covers only `WxH`; video frame/conditioning mismatches can still fall back to Eager. MiniMax-H3 remains eager in the validated deployment because prompt-dependent packed host boundaries can miss the captured signature. | | **Warmup** | `--warmup-mode request` | Runs dummy forward passes to warm up CUDA caches, JIT, and `torch.compile`. Eliminates cold-start penalty. | Removes first-request latency spike | Adds startup time. Without `--warmup-resolutions`, warmup happens on first request. | | **Warmup Resolutions** | `--warmup-resolutions 256x256 720x720` | Pre-compiles and warms up specific resolutions at server startup (instead of lazily on first request). | Faster first request per resolution | Each resolution adds to startup time. Serving mode only; useful when you know your target resolutions in advance. | | **Multi-GPU (SP)** | `--num-gpus N --ulysses-degree N` | Sequence parallelism across GPUs. Shards sequence tokens (not frames) to minimize padding. | Near-linear scaling with N GPUs | Requires NCCL; inter-GPU bandwidth matters. `ulysses_degree * ring_degree = sp_degree`. For Wan2.2 video, start by benchmarking pure Ulysses before assuming a mixed Ulysses/Ring layout is fastest. | | **Cross-node SP** | `--nnodes`, `--node-rank`, `--dist-init-addr` with total `--num-gpus`; combine node-local Ulysses with cross-node Ring | Extends sequence parallel groups across multiple nodes. | Capacity and long-sequence scaling beyond one host | Prefer Ulysses within a node and Ring across nodes; all-to-all is usually the least cross-node-friendly. Use `--encoder-parallel replicate` today and verify the model's Ring admission and determinism. MiniMax-H3 is the current end-to-end validated recipe. | | **CFG Parallel** | `--enable-cfg-parallel` | Runs conditional and unconditional CFG branches in parallel across GPUs. For CFG models on multi-GPU, benchmark this against pure Ulysses on your topology instead of assuming one always wins. | Often faster than pure SP for CFG models | Requires `num_gpus >= 2`. Halves the Ulysses group size (e.g. 8 GPU → two 4-GPU groups). Only for models that use CFG. Nightly coverage configs may intentionally use smaller Ulysses groups to keep ring behavior exercised; that does not automatically make them the lowest-latency choice. | | **Layerwise Offload** | `--dit-layerwise-offload` | Async layer-by-layer H2D prefetch with compute overlap. Only ~2 DiT layers reside on GPU at a time, dramatically reducing VRAM. For some video models the copy stream can be almost fully hidden behind compute. | Saves VRAM (40 GB → ~11 GB for Wan A14B); can be near-zero speed cost on the right workload | Enabled by default for Wan/MOVA video models. Compatible with Cache-DiT (skipped blocks are not streamed). For **image models** or highly parallelized setups (many GPUs, small per-GPU compute), the copy stream may not be fully hidden and can cause slowdown. | | **Offload Prefetch Size** | `--dit-offload-prefetch-size F` | Fine-grained control over layerwise offload: how many layers to prefetch ahead. `0.0` = 1 layer (min VRAM), `0.1` = 10% of layers, `≥1` = absolute layer count. | Tune for cases where default offload has copy stream interference (e.g. image models). 0.05–0.1 is a good starting point. | Values ≥ 0.5 approach no-offload VRAM with worse performance. Use lower values when copy overlap is weak; disable offload when memory allows and latency dominates. | | **FSDP Inference** | `--use-fsdp-inference` | Uses PyTorch FSDP to shard model weights across GPUs with prefetch. Low latency, low VRAM. | Reduces per-GPU VRAM | Mutually exclusive with `--dit-layerwise-offload`. More overhead than SP on high-bandwidth interconnects. | | **CPU Offload (components)** | `--text-encoder-cpu-offload`, `--image-encoder-cpu-offload`, `--vae-cpu-offload`, `--dit-cpu-offload` | Offloads specific pipeline components to CPU when not in use. | Reduces peak VRAM | Adds H2D transfer latency when the component is needed. Auto-enabled for low-VRAM GPUs (<30 GB). **Tip:** after the first request completes, the console prints a peak VRAM analysis with suggestions on which offload flags can be safely disabled — look for the `"Components that could stay resident"` log line. | | **Pin CPU Memory** | `--pin-cpu-memory` | Uses pinned (page-locked) memory for CPU offload transfers. | Faster H2D transfers | Slightly higher host memory usage. Enabled by default; disable only as workaround for CUDA errors. | | **Attention Backend (lossless)** | `--attention-backend fa` | Selects a lossless attention kernel for SGLang-native pipelines: `fa` (FlashAttention 2/3/4 alias) or `torch_sdpa`. | FA is usually faster than SDPA on long sequences | FA requires compatible GPU (Ampere+). For `--backend diffusers`, valid backend names differ; use the names documented in `docs/docs/sglang-diffusion/attention_backends.mdx`. | | **Parallel Folding** | *(automatic when SP > 1)* | Reuses the SP process group as TP for the T5 text encoder, so text encoding is parallelized "for free". | Faster text encoding on multi-GPU | Automatic; no user action needed. Only applies to T5-based pipelines. | ### Choosing what goes in `--layerwise-offload-components` Layerwise offload pays one H2D of a component's weights **per pass over that component**, overlapped with that pass's compute. So the question is not how big the component is, it is **how many passes per request it makes** — counted from the code, not from the pipeline diagram: | Component | Passes per request | Placement | |---|---|---| | DiT | one per denoising step | Stream it. The transfer amortizes over every step and hides behind attention/FFN. | | Video VAE | one per **temporal chunk**, not one | Declare it streamed and keep its blocks resident. | | Text / image encoder | one | Resident if it fits; otherwise streamed with its blocks resident. | | Vocab table | a gather, one row per token | Neither. Declare it in `host_resident_table_names` and leave it in host memory. | "One-shot" is a property of the code, not of the diagram. `_decode_temporal_streaming` in `runtime/models/vaes/minimax_h3_video_vae/klvae.py` calls the whole video decoder once per temporal chunk, so streaming its 36 blocks pays 36 block transfers per chunk. On MiniMax-H3 at 864x480 / 124 frames that is **150 s** of decode against **13 s** with the blocks held. **There are three placements, not two.** A component can be declared streamed and still hold its blocks: | Placement | How | Transfers | |---|---|---| | Resident | leave it out of the list | once at load; the VRAM is held for the whole process | | Streamed, blocks resident | in the list **plus** `--layerwise-resident-layers video_vae=36` | once, not per pass, and the VRAM comes back when the component finishes | | Streamed | in the list, resident layers 0 | every pass — worth it only for a component that makes many passes, i.e. the DiT | So do not read "drop it from the list" as the fix for a one-shot component: that keeps it resident for the whole process, which is exactly the VRAM a 12-24 GB budget does not have. Measured on MiniMax-H3, 1x RTX 4090 24 GB, 672x384, 4 steps, prefetch 1, **no resident VAE blocks** in either row: | `--layerwise-offload-components` | denoise | decode | peak | |---|---|---|---| | `dit,text_encoder,vae` | 15.51 s | **39.85 s** | 16.4 GB | | `dit,text_encoder` | 17.33 s | **5.32 s** | 22.2 GB | Those two rows are the first and third placements. Taking the VAE out of the stream cut decode 7.5x and the request 76 s -> 29 s, at the cost of 5.8 GB of peak and ~12% on denoise because the DiT's staging buffers have less room. The middle placement is what gets the decode without paying the peak. ### Prefetch depth has a knee `--dit-offload-prefetch-size` is not monotonic. Deeper prefetch hides more of the copy but its staging buffers crowd out activations. Same H3 configuration, VAE resident: | prefetch | denoise | decode | peak | |---|---|---|---| | 1 (default) | 17.33 s | 5.32 s | 22.2 GB | | **2** | **15.29 s** | **4.89 s** | 22.1 GB | | 3 | 15.79 s | 5.25 s | 21.5 GB | | 4 | 16.73 s | 5.17 s | 23.5 GB (96% of the card) | Sweep it rather than assuming the default, and sweep it on the target configuration: the direction depends on model, resolution and card, so a value carried over from another model means nothing. `--dit-layerwise-residency-policy strided` is the other knob on the same bytes — same VRAM, same volume, spread over the step instead of crammed into its tail. ### Host memory is part of the placement decision Per-component placement is not independent, for two reasons: - **One host budget.** Pinning a one-shot component's weights takes host RAM that the page cache needs to serve a streamed component's mapped weights. Pinning something that runs once can slow down the thing that runs every step. - **Pinned is asynchronous, mapped is not.** A pinned source overlaps its transfer with compute. An unpinned or mapped source is synchronous whatever the code requests, because the driver stages it through its own buffer. Same bytes, different wall clock. MiniMax-H3 fl2va, 1x RTX 4090, 864x480 / 124 frames / 20 NFE, identical DiT bytes per step, only the host-side source differs: | DiT weight source | denoise | per step | configuration | |---|---|---|---| | pinned host memory | **122.84 s** | 6.10 s | host uncapped, 116.7 GB pinned | | checkpoint mapping | 330.74 s | 17.4 s | host capped at 32 GiB, allocator at 23 GiB | | checkpoint mapping | 318.94 s - 356.37 s | 16.8 - 18.7 s | host capped at 32 GiB, allocator at 12 GiB | The last row is the same configuration measured twice; the 12% spread tracked host load, so treat differences smaller than that on a shared machine as unresolved. When a streamed run is inexplicably slow, check the host side before touching prefetch or residency: whether the weights are pinned or served from a mapping, and whether host memory pressure pushed them onto one. ### When the transfer knobs do nothing Before tuning residency or prefetch, measure whether the transfer is exposed at all — and measure it, do not infer it from bytes. Bytes over bandwidth is an upper bound on what *could* be exposed, not what is; prefetch exists to hide exactly that. Wan2.1-1.3B on a 12 GB RTX 3060, `--dit-layerwise-resident-layers` 0/5/10/20: 1.10 / 1.04 / 1.05 / 1.06 s per step. Flat and non-monotonic, i.e. noise — even though at 65 MB a layer a step moves 2.64 GB, on the order of 100 ms of a 1.04 s step if none of it overlapped. It overlaps, so residency buys nothing. MiniMax-H3 is 1.36 GB a layer, 21x that, and a step moves about 66 GB; there the same flags decide whether the model runs at all. Same flags, opposite conclusion — so sweep two or three values and keep the measured winner instead of reasoning from the checkpoint size. Residency changes are lossless either way — across a residency sweep on Wan2.1-1.3B every output had the same SHA-256. ### Single-GPU large-VRAM notes (measured on 1x B300, 275 GB, SM103) A single large-VRAM card changes two common assumptions: - **Launch-bound small models gain the most from BCG.** SANA1.5-1.6B (image) denoise dropped 0.70s -> 0.23s (-67%) with `--enable-breakable-cuda-graph`; SANA-Video (832x480, 17 frames) dropped 1.24s -> 0.96s (-22%) once `--warmup-num-frames 17` matched the served frame count. Both bit-identical. Compute-bound models (LongCat-Image, Qwen-Image, Z-Image, Cosmos3-Edge, FLUX.1-dev, Wan2.1-1.3B) saw no BCG or `torch.compile` gain — profiles show GEMM + flash-attention + already-fused norm/GELU saturating the device. - **Component CPU offload is often a pessimization, not a free win.** Many presets enable `--text-encoder-cpu-offload` for memory-bound cards, but the whole model fits in 275 GB, so the H2D/D2H round trip is pure overhead. Dropping it on LingBot-Video-MoE was 8% faster end to end (bit-identical). On a large-VRAM single GPU, re-test each `*-cpu-offload` flag before keeping it. --- ## Section 2: Lossy Optimizations These options **trade output quality** for speed or VRAM savings. Results will differ from the baseline. | Option | CLI Flag / Env Var | What It Does | Speedup | Quality Impact / Limitations | |---|---|---|---|---| | **Request Quality Fast Paths** | `--quality {extra-high,high}` (`lossless` is default) | `extra-high` mounts only request-gated DiT/VAE fusions. `high` includes that complete set and may add model-owned approximate paths such as Cache-DiT or lower-precision decode. | Model- and shape-specific | Support is per model and may be a no-op. Keep `--quality lossless` as the A/B ground truth, then compare `extra-high` before `high` to isolate fusion wins. Report aggregate and worst-frame SSIM/PSNR for every non-bit-exact path; defaults are 0.95/28 dB for images and 0.92/24 dB for video unless checked-in model metadata overrides them. Do not confuse this with `--output-quality`, which controls file compression. | | **Approximate Attention** | Server-wide: `--attention-backend sage_attn` / `sage_attn_3` / `sliding_tile_attn` / `video_sparse_attn` / `sparse_video_gen_2_attn` / `vmoba_attn` / `sla_attn` / `sage_sla_attn`. Per-request (dense drop-ins only): `--attention-backend-override sage_attn` sampling param / API `extra_body` — valid values `fa`, `torch_sdpa`, `sage_attn`, `sage_attn_3`; rejected (with a log) under BCG, torch.compile, sparse server backends, or a non-ring-capable target with ring parallelism. | Replaces exact attention with approximate or sparse variants. `sage_attn`: INT8/FP8 quantized Q·K; `sliding_tile_attn`: spatial-temporal tile skipping; others: model-specific sparse patterns. | ~1.5–2x on attention (varies by backend) | Quality degradation varies by backend and model. `sage_attn` is the most general; sparse backends (`sliding_tile_attn`, `video_sparse_attn`, etc.) are video-model-specific, may require config files (e.g. `--mask-strategy-file-path` for STA), and are server-level only. Requires corresponding packages installed. |