sglang-diffusion-performance

name	sglang-diffusion-performance
description	Use when choosing the fastest SGLang Diffusion flags for a model, GPU, and VRAM budget.

SGLang Diffusion Performance Tuning

Use this skill when the user wants the fastest command line, lower VRAM, or the right performance flags for a specific model and GPU setup.

Before running any sglang generate command below inside the diffusion container:

• use python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/diffusion_skill_env.py to derive the repo root, verify write access, and choose idle GPU(s)
• export HF_TOKEN first when the selected model lives in a gated Hugging Face repo such as black-forest-labs/FLUX.*
• export FLASHINFER_DISABLE_VERSION_CHECK=1
• cd to the repo root resolved from sglang.__file__

Native Backend Gate

Performance numbers are useful only when the intended backend actually ran.

• Treat any log containing Falling back to diffusers backend, Using diffusers backend, or Loaded diffusers pipeline as invalid for native SGLang performance tuning.
• Use --backend diffusers only for an explicit diffusers baseline. For native recipes, leave the default backend or pin --backend sglang.
• If a fallback happened, fix pipeline registration/model-path/config issues first, then rerun. Do not compare perf dumps collected from a fallback run.
• When the runtime auto-selects parallel settings because the user omitted them, keep the result as an auto-tuned baseline. For reproducible tuning, pin --num-gpus, --ulysses-degree, --ring-degree, and --enable-cfg-parallel explicitly.

Reference: SGLang-Diffusion Advanced Optimizations Blog

---

Section 1: Lossless Optimizations

These options are intended to preserve output quality. In practice, some paths (most notably torch.compile) can still introduce small floating-point drift, so validate on your target model when numerical parity matters.

Option	CLI Flag / Env Var	What It Does	Speedup	Limitations / Notes
Performance Mode	--performance-mode auto|speed|memory|manual (--mode alias)	Applies model-aware residency, FSDP/CFG, and compile defaults without overriding explicit flags. auto is the safe default; speed favors GPU residency; memory favors offload; manual leaves performance args explicit.	Fastest way to establish a sensible deployment baseline	speed may OOM and enables torch.compile only when the model deployment config allows it. Explicit offload/FSDP/parallelism/compile flags win. Use manual for controlled A/B benchmarks.
torch.compile	--enable-torch-compile	Applies torch.compile to the DiT forward pass, fusing ops and reducing kernel launch overhead.	~1.2–1.5x on denoising	First request is slow (compilation). May cause minor precision drifts due to PyTorch issue #145213. Pair with --warmup-mode request for best results.
Breakable CUDA Graph	--enable-breakable-cuda-graph --warmup-resolutions <WxH...> plus optional --bcg-text-buckets ...	Captures fixed-resolution DiT segments while leaving attention/collectives eager, reducing launch overhead on supported pipelines.	Large on launch-bound paths; merged SANA and LTX-2 cases show material e2e gains	Mutually exclusive with torch.compile and Cache-DiT; BCG takes priority. Every served resolution must be declared for warmup capture. Current support is model-specific (Ideogram4, LTX-2, MiniMax-H3, Qwen-Image, SANA1.5, Z-Image, GLM-Image); benchmark before keeping it.
Warmup	--warmup-mode request	Runs dummy forward passes to warm up CUDA caches, JIT, and torch.compile. Eliminates cold-start penalty.	Removes first-request latency spike	Adds startup time. Without --warmup-resolutions, warmup happens on first request.
Warmup Resolutions

---

Section 2: Lossy Optimizations

These options trade output quality for speed or VRAM savings. Results will differ from the baseline.

Option	CLI Flag / Env Var	What It Does	Speedup	Quality Impact / Limitations
Request Quality Fast Paths	--quality high (lossless is default)	Mounts model-owned accelerated DiT/VAE paths that are validated for high quality but are not bit-exact to the reference path.	Model- and shape-specific	Support is per model and may be a no-op. Keep --quality lossless as the A/B ground truth. Do not confuse this with --output-quality, which controls file compression.
Approximate Attention	--attention-backend sage_attn / sage_attn_3 / sliding_tile_attn / video_sparse_attn / sparse_video_gen_2_attn / vmoba_attn / sla_attn / sage_sla_attn	Replaces exact attention with approximate or sparse variants. sage_attn: INT8/FP8 quantized Q·K; sliding_tile_attn: spatial-temporal tile skipping; others: model-specific sparse patterns.	~1.5–2x on attention (varies by backend)	Quality degradation varies by backend and model. sage_attn is the most general; sparse backends (sliding_tile_attn, video_sparse_attn, etc.) are video-model-specific and may require config files (e.g. --mask-strategy-file-path for STA). Requires corresponding packages installed.
Cache-DiT	Native: SGLANG_CACHE_DIT_ENABLED=true plus SGLANG_CACHE_DIT_* env vars. Diffusers backend: --backend diffusers --cache-dit-config <yaml-or-json>	Caches intermediate residuals across denoising steps and skips redundant computations via DBCache, TaylorSeer, and optional SCM.	~1.5-2x on supported models	Quality depends on cache policy. Incompatible with --dit-layerwise-offload. Do not pass --cache-dit-config for native SGLang tuning unless you are intentionally using the diffusers backend flow.
TeaCache	--enable-teacache (uses model sampling presets)	Reuses residuals when adjacent denoising steps are sufficiently similar.

---

Quick Recipes

MiniMax-H3 first: lossless joint video/audio

H3 has a stricter contract than the generic recipes below. Keep its DiT eager for consistency ground truth, use Ulysses rather than Ring, do not enable CFG parallel, and leave the released overlapping tiled video-VAE decode in place.

Four H200 GPUs can keep the complete BF16/FP32 pipeline resident:

sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --performance-mode speed \
  --enable-torch-compile false \
  --port 30010

On 4x H100 80 GB, start from the fastest measured lossless resident topology:

sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --tp-size 2 \
  --ulysses-degree 2 \
  --performance-mode speed \
  --enable-torch-compile false \
  --port 30010

On B200/B300, the verified resident sweep uses 8 GPUs with Ulysses8. H3 also has a verified 4x B200 FSDP-capacity path, but FSDP all-gathers are a memory policy rather than the default latency choice. Benchmark the target topology with the H3 driver from sglang-diffusion-benchmark-profile.

Use the FL2VA partition for both t2va and fl2va; use --model-variant ref2va for image/video/audio reference conditioning. The root IDs are MiniMaxAI/MiniMax-H3 on Hugging Face and MiniMax/MiniMax-H3 on ModelScope. Do not point --model-path at a partition subdirectory.

Current H3 restrictions:

• torch.compile is opt-in experimentation only because it changes numerical output; it is not a lossless baseline
• Ring attention and CFG parallel are incompatible with the packed single denoising branch
• SageAttention is rejected for the current packed multi-segment attention
• --vae-config.parallel-decode-mode spatial, spatial_shard, and patch VAE decode are rejected after mismatches; use the default tiled recipe
• Breakable CUDA Graph is opt-in and signature-specific; the validated 1344x768 Ref2VA capture uses --bcg-text-buckets 5504, but it did not show a measured speedup
• the quality=high|medium|low Cache-DiT profiles and online FP8 are approximate; keep them outside lossless comparisons

Maximum speed, video model, multi-GPU, lossless (Wan A14B, 8 GPUs)

sglang generate --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --num-gpus 8 --enable-cfg-parallel --ulysses-degree 4 \
  --enable-torch-compile --warmup-mode request \
  --text-encoder-cpu-offload true \
  --prompt "..." --save-output

Note: --dit-layerwise-offload is enabled by default for Wan/MOVA video models and is often a good default, but still benchmark it on your exact workload if latency matters.

For Wan2.2 specifically:

• the nightly-aligned 4-GPU benchmark may use --enable-cfg-parallel --ulysses-degree=2 to keep CFG and ring behavior covered
• that is a coverage choice, not a guaranteed best-performance choice
• for pure latency tuning, benchmark pure Ulysses too, for example --ulysses-degree=4 --ring-degree=1 on 4 GPUs
• on 8 GPUs, compare pure --ulysses-degree=8 against --enable-cfg-parallel --ulysses-degree=4

Current-source model, 2 GPUs: LTX-2 two-stage

sglang generate --model-path Lightricks/LTX-2 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --prompt "A cat and a dog baking a cake together in a kitchen." \
  --width 768 --height 512 \
  --num-frames 121 \
  --seed 42 --num-gpus 2 --enable-cfg-parallel \
  --enable-torch-compile --warmup-mode request --save-output

Note: LTX-2 is a current-source benchmark preset rather than a nightly comparison case. The command uses runtime-default steps and guidance. LTX2TwoStagePipeline is a native path and auto-resolves the spatial upsampler plus distilled LoRA from the same model snapshot unless you override them.

Nightly-aligned model, 2 GPUs: LTX-2.3 TI2V two-stage

sglang generate --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --prompt "The cat starts walking slowly towards the camera." \
  --image-path "${ASSET_DIR}/cat.png" \
  --width 768 --height 512 \
  --num-frames 121 \
  --seed 42 --num-gpus 2 --cfg-parallel-size 2 \
  --enable-torch-compile --warmup-mode request --save-output

Note: this matches the nightly comparison case ltx2.3_twostage_ti2v_2gpus. The nightly config omits explicit steps and guidance, so this command omits them too and uses runtime defaults. Download ${ASSET_DIR}/cat.png with the benchmark/profile skill before running it.

Native baseline, 2 GPUs: LTX-2.3 one-stage

sglang generate --model-path Lightricks/LTX-2.3 \
  --prompt "A beautiful sunset over the ocean" \
  --negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." \
  --width 768 --height 512 \
  --num-frames 121 --fps 24 \
  --num-inference-steps 30 --guidance-scale 3.0 \
  --seed 1234 --num-gpus 2 \
  --enable-torch-compile --warmup-mode request --save-output

Note: use this as the native LTX2Pipeline baseline for LTX-2.3. It keeps the validated one-stage resolution and explicit LTX-2.3 sampling defaults, and matches the ltx23-one-stage benchmark preset in sglang-diffusion-benchmark-profile.

Skill-only stress target, 2 GPUs: LTX-2.3 two-stage high resolution

sglang generate --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --prompt "A beautiful sunset over the ocean" \
  --negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." \
  --width 1536 --height 1024 \
  --num-frames 121 --fps 24 \
  --num-inference-steps 30 --guidance-scale 3.0 \
  --seed 1234 --num-gpus 2 \
  --enable-torch-compile --warmup-mode request --save-output

Note: this is a high-resolution stress target for the native LTX-2.3 two-stage path. It matches the skill-only ltx23-two-stage benchmark preset, not a nightly comparison case.

Maximum speed, image model, single GPU, lossless

sglang generate --model-path <IMAGE_MODEL> \
  --enable-torch-compile --warmup-mode request \
  --dit-layerwise-offload false \
  --dit-cpu-offload false \
  --prompt "..." --save-output

Note: for image models, per-layer compute is smaller, so layerwise offload may not fully hide H2D transfer. Disable DiT layerwise and CPU offload if VRAM allows; otherwise a large image DiT can stay resident on CPU and make the denoise loop H2D-bound.

Launch-bound fixed-resolution path: Breakable CUDA Graph

sglang serve --model-path Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers \
  --performance-mode speed \
  --enable-torch-compile false \
  --enable-breakable-cuda-graph \
  --warmup-resolutions 1024x1024 \
  --port 30010

Keep torch.compile off, declare every production resolution, and benchmark the exact prompt-length distribution. Add --bcg-text-buckets only when the default buckets create excessive padding or miss a served prompt signature.

Compare request-scoped high-quality fast paths

sglang generate --model-path <MODEL> \
  --quality lossless --prompt "..." --seed 42 \
  --perf-dump-path baseline.json --save-output

sglang generate --model-path <MODEL> \
  --quality high --prompt "..." --seed 42 \
  --perf-dump-path quality-high.json --save-output

Keep every other flag fixed and compare the generated artifact as well as the perf dumps. If the model has no registered quality-gated sites, high may be a no-op.

Image-edit baselines: JoyAI and FireRed

sglang generate --backend=sglang \
  --model-path jdopensource/JoyAI-Image-Edit-Diffusers \
  --prompt "Make the cat wear a red hat" \
  --image-path "${ASSET_DIR}/cat.png" \
  --width 1024 --height 1024 \
  --num-inference-steps 40 --guidance-scale 4.0 \
  --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 \
  --dit-layerwise-offload false --dit-cpu-offload false \
  --enable-torch-compile --warmup-mode request --save-output

sglang generate --backend=sglang \
  --model-path FireRedTeam/FireRed-Image-Edit-1.1 \
  --prompt "Make the cat wear a red hat" \
  --image-path "${ASSET_DIR}/cat.png" \
  --width 1024 --height 1024 \
  --num-inference-steps 40 --guidance-scale 4.0 \
  --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 \
  --dit-layerwise-offload false --dit-cpu-offload false \
  --enable-torch-compile --warmup-mode request --save-output

Use FireRedTeam/FireRed-Image-Edit-1.0 in the same command when comparing FireRed 1.0. These are native image-edit paths; keep the reference image, prompt, seed, and output size fixed when comparing denoise numbers. On H100, 2-GPU CFG parallel was faster than the otherwise matching 2-GPU Ulysses command: FireRed 1.0 improved from 13419.15 ms to 10955.90 ms, and FireRed 1.1 improved from 13414.72 ms to 10934.21 ms.

Hunyuan3D shape baseline

OUTPUT_DIR=$(python3 "$ENV_PY" print-output-dir --kind benchmarks --mkdir)
CONFIG_DIR="${OUTPUT_DIR}/generated_configs"
mkdir -p "${CONFIG_DIR}"
printf '{"paint_enable": false}\n' > "${CONFIG_DIR}/hunyuan3d-shape.json"

sglang generate --backend=sglang \
  --model-path tencent/Hunyuan3D-2 \
  --prompt "generate 3d mesh" \
  --image-path "${ASSET_DIR}/cat.png" \
  --config "${CONFIG_DIR}/hunyuan3d-shape.json" \
  --num-inference-steps 50 --guidance-scale 5.0 \
  --dit-layerwise-offload false --dit-cpu-offload false \
  --enable-torch-compile --warmup-mode request --save-output

For Hunyuan3D, treat Hunyuan3DShapeDenoisingStage as the primary latency metric. Mesh export and paint stages are useful end-to-end checks but should not drive DiT optimization decisions.

Low VRAM, decent speed (single GPU)

sglang generate --model-path <MODEL> \
  --enable-torch-compile --warmup-mode request \
  --dit-layerwise-offload --dit-offload-prefetch-size 0.1 \
  --text-encoder-cpu-offload true --vae-cpu-offload true \
  --prompt "..." --save-output

Maximum speed, lossy native path (SageAttention + Cache-DiT)

SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path <MODEL> \
  --attention-backend sage_attn \
  --dit-layerwise-offload false \
  --enable-torch-compile --warmup-mode request \
  --prompt "..." --save-output

Add native Cache-DiT knobs such as SGLANG_CACHE_DIT_SCM_PRESET=medium, SGLANG_CACHE_DIT_RDT=0.24, or SGLANG_CACHE_DIT_TAYLORSEER=true only after you have a BF16 baseline output to compare against.

For a diffusers-backend Cache-DiT YAML/JSON config baseline, make the fallback explicit:

sglang generate --backend diffusers --model-path <MODEL> \
  --cache-dit-config <config.yaml> \
  --dit-layerwise-offload false \
  --prompt "..." --save-output

---

Model-Specific Starting Points

Use these as first commands to benchmark, not as universal winners.

Model family	First performance shape	Starting flags	Notes
MiniMax-H3	1344x768 resolved canvas, 5 seconds / 124 frames at 24 fps, 50 joint video/audio steps	H200: --num-gpus 4 --ulysses-degree 4 --performance-mode speed --enable-torch-compile false; H100: TP2 + Ulysses2	Root ID plus --model-variant fl2va for T2VA/FL2VA or ref2va for Ref2VA. Ulysses only; no Ring/CFG/SageAttention. Preserve tiled video-VAE decode. Profile joint denoise, video VAE, audio VAE/vocoder, encoder, and collectives separately.
FLUX.1 / FLUX.2 image	1024x1024, runtime-default steps/guidance, 1 GPU	--enable-torch-compile --warmup-mode request --dit-layerwise-offload false	black-forest-labs/FLUX.* repos are gated; for FP8/NVFP4 use validated --transformer-path or --transformer-weights-path flows from the quant skill.
FLUX.2 Klein / Klein Base	1024x1024, runtime-default steps/guidance, 1 GPU	--enable-torch-compile --warmup-mode request --dit-layerwise-offload false	Current registry has black-forest-labs/FLUX.2-klein-4B, FLUX.2-klein-9B, and base variants. Klein is step-distilled; Klein Base is not.
Qwen-Image / Qwen-Image-Edit	1024x1024, runtime-default steps/guidance, 1 GPU	--enable-torch-compile --warmup-mode request; optionally native SGLANG_CACHE_DIT_ENABLED=true	Cache-DiT is lossy. For edit tasks, keep reference image, seed, and output size fixed.
Krea-2	1024x1024, distilled oss_turbo defaults (8 steps, guidance 1.0)	--performance-mode speed --warmup-mode request	Native krea/Krea-2 text-to-image path with Qwen3-VL text conditioning. The repo may require HF access; keep the 8-step distilled baseline separate from non-turbo sampling experiments.
Z-Image / Z-Image-Turbo	1024x1024, runtime-default steps/guidance, 1 GPU	--enable-torch-compile --warmup-mode request	Keep base Z-Image separate from Turbo: base uses 50-step CFG defaults, Turbo uses 9-step zero-CFG defaults. Mainline has bf16-native Triton RMSNorm scale and tanh-residual fusions.
Wan2.2 A14B T2V/I2V

Historical PR Watchlist

Treat these performance PRs as direction and prior art only. Re-check the PR state and the active source tree before relying on any path, flag, or claim about whether the work has merged:

• Fusion/kernel: #24025 LTX2 QK norm, #24059 Helios norm modulation, #24117 Z-Image packed QKV, #19488 Wan elementwise cross-block fusion, #19249 Z-Image gate/norm fusion, #20429 Qwen-Image layernorm/modulation, #20530 MOVA RMSNorm+RoPE.
• VAE/decode: #22531 LTX2 parallel VAE, #20927 batched tiled VAE decode.
• Runtime/parallel/cache: #22805 FLUX.2 packed QKV for A2A, #21742 hybrid attention schedule, #24053 USP replicated-prefix fix, #21613 TeaCache refactor, #24227 WanVideo TeaCache fix, #18764 dynamic batching, #24200 disaggregated diffusion.

Tips

• Benchmarking: always use --warmup-mode request and look for the line ending with (with warmup excluded) for accurate timing.
• Preset vs experiment control: start with --performance-mode auto or speed for deployment, but use --performance-mode manual and pin the relevant residency/parallelism flags for controlled A/B claims.
• Perf dump: use --perf-dump-path result.json to save structured metrics, then compare with python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json result.json.
• Offload tuning: after the first request, the runtime logs peak GPU memory and which components could stay resident. Use this to decide which --*-cpu-offload flags to disable.
• Backend selection: --backend sglang (default, auto-detected) enables native optimizations (fused kernels, SP, native Cache-DiT env knobs, etc.). --backend diffusers falls back to Diffusers pipelines and is the path that accepts --cache-dit-config plus diffusers attention backend names.
• Wan2.2-I2V sizing: explicit --width/--height on Wan2.2-I2V-A14B control the target area while preserving the condition-image aspect ratio.
• Mainline diffusion fast paths: before proposing a new kernel or overlap scheme, check sglang-diffusion-benchmark-profile/existing-fast-paths.md. It covers H3 indexed modulation, fused QK norm + RoPE, packed Ulysses QKV/USP relayout and batched TP AdaLN; FLUX/GLM/SANA bit-exact LayerNorm+modulate; request-scoped quality gates; Wan causal-VAE data movement; GroupNorm+SiLU, Z-Image bf16-native norm modulation, LTX2 split RoPE/residual-gate add, varlen USP pack/scatter, packed QKV/NVFP4, breakable CUDA graph, and existing distributed overlap families.
• NVFP4 trace interpretation: on FLUX.2 NVFP4 and Nunchaku-style checkpoints, packed QKV is expected. SGLang intentionally uses fused projection modules such as to_qkv / to_added_qkv instead of separate to_q / to_k / to_v, so a split-QKV trace usually means the quantized path did not engage rather than a brand new fusion opportunity.
• Hotspot workflow split: use sglang-diffusion-benchmark-profile to prove and classify a slowdown with perf dumps plus ; hand concrete kernel work off with the perf/profile evidence attached instead of expanding the benchmark skill.