Sglang Diffusion Performance

SGLang Diffusion Performance Tuning

Use this skill when the user wants the fastest command line, lower VRAM, or the right performance flags for a specific model and GPU setup.

Before running any sglang generate command below inside the diffusion container:

• use python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/diffusion_skill_env.py to derive the repo root, verify write access, and choose idle GPU(s)
• export HF_TOKEN first when the selected model lives in a gated Hugging Face repo such as black-forest-labs/FLUX.*
• export FLASHINFER_DISABLE_VERSION_CHECK=1
• when a run downloads weights, use a task-owned cache and delete that model's cache after its eager/BCG/quality/profile group finishes; the benchmark skill's --quality-bcg-matrix --model-cache-root --cleanup-model-cache keeps one cache for the group and writes a zero-residual cleanup ledger
• hold one idle GPU set for the complete A/B matrix and verify no foreign process appears at run boundaries
• cd to the repo root resolved from sglang.__file__

Native Backend Gate

Performance numbers are useful only when the intended backend actually ran.

• Treat any log containing Falling back to diffusers backend, Using diffusers backend, or Loaded diffusers pipeline as invalid for native SGLang performance tuning.
• Use --backend diffusers only for an explicit diffusers baseline. For native recipes, leave the default backend or pin --backend sglang.
• If a fallback happened, fix pipeline registration/model-path/config issues first, then rerun. Do not compare perf dumps collected from a fallback run.
• When the runtime auto-selects parallel settings because the user omitted them, keep the result as an auto-tuned baseline. For reproducible tuning, pin --num-gpus, --ulysses-degree, --ring-degree, and --enable-cfg-parallel explicitly.

Reference: SGLang-Diffusion Advanced Optimizations Blog

Section 1: Lossless Optimizations

These options are intended to preserve output quality. In practice, some paths (most notably torch.compile) can still introduce small floating-point drift, so validate on your target model when numerical parity matters.

Choosing what goes in --layerwise-offload-components

Layerwise offload pays one H2D of a component's weights per pass over that component, overlapped with that pass's compute. So the question is not how big the component is, it is how many passes per request it makes — counted from the code, not from the pipeline diagram:

"One-shot" is a property of the code, not of the diagram. _decode_temporal_streaming in runtime/models/vaes/minimax_h3_video_vae/klvae.py calls the whole video decoder once per temporal chunk, so streaming its 36 blocks pays 36 block transfers per chunk. On MiniMax-H3 at 864x480 / 124 frames that is 150 s of decode against 13 s with the blocks held.

There are three placements, not two. A component can be declared streamed and still hold its blocks:

So do not read "drop it from the list" as the fix for a one-shot component: that keeps it resident for the whole process, which is exactly the VRAM a 12-24 GB budget does not have.

Measured on MiniMax-H3, 1x RTX 4090 24 GB, 672x384, 4 steps, prefetch 1, no resident VAE blocks in either row:

Those two rows are the first and third placements. Taking the VAE out of the stream cut decode 7.5x and the request 76 s -> 29 s, at the cost of 5.8 GB of peak and ~12% on denoise because the DiT's staging buffers have less room. The middle placement is what gets the decode without paying the peak.

Prefetch depth has a knee

--dit-offload-prefetch-size is not monotonic. Deeper prefetch hides more of the copy but its staging buffers crowd out activations. Same H3 configuration, VAE resident:

Sweep it rather than assuming the default, and sweep it on the target configuration: the direction depends on model, resolution and card, so a value carried over from another model means nothing. --dit-layerwise-residency-policy strided is the other knob on the same bytes — same VRAM, same volume, spread over the step instead of crammed into its tail.

Host memory is part of the placement decision

Per-component placement is not independent, for two reasons:

• One host budget. Pinning a one-shot component's weights takes host RAM that the page cache needs to serve a streamed component's mapped weights. Pinning something that runs once can slow down the thing that runs every step.
• Pinned is asynchronous, mapped is not. A pinned source overlaps its transfer with compute. An unpinned or mapped source is synchronous whatever the code requests, because the driver stages it through its own buffer. Same bytes, different wall clock.

MiniMax-H3 fl2va, 1x RTX 4090, 864x480 / 124 frames / 20 NFE, identical DiT bytes per step, only the host-side source differs:

The last row is the same configuration measured twice; the 12% spread tracked host load, so treat differences smaller than that on a shared machine as unresolved. When a streamed run is inexplicably slow, check the host side before touching prefetch or residency: whether the weights are pinned or served from a mapping, and whether host memory pressure pushed them onto one.

When the transfer knobs do nothing

Before tuning residency or prefetch, measure whether the transfer is exposed at all — and measure it, do not infer it from bytes. Bytes over bandwidth is an upper bound on what could be exposed, not what is; prefetch exists to hide exactly that.

Wan2.1-1.3B on a 12 GB RTX 3060, --dit-layerwise-resident-layers 0/5/10/20: 1.10 / 1.04 / 1.05 / 1.06 s per step. Flat and non-monotonic, i.e. noise — even though at 65 MB a layer a step moves 2.64 GB, on the order of 100 ms of a 1.04 s step if none of it overlapped. It overlaps, so residency buys nothing. MiniMax-H3 is 1.36 GB a layer, 21x that, and a step moves about 66 GB; there the same flags decide whether the model runs at all. Same flags, opposite conclusion — so sweep two or three values and keep the measured winner instead of reasoning from the checkpoint size.

Residency changes are lossless either way — across a residency sweep on Wan2.1-1.3B every output had the same SHA-256.

Single-GPU large-VRAM notes (measured on 1x B300, 275 GB, SM103)

A single large-VRAM card changes two common assumptions:

• Launch-bound small models gain the most from BCG. SANA1.5-1.6B (image) denoise dropped 0.70s -> 0.23s (-67%) with --enable-breakable-cuda-graph; SANA-Video (832x480, 17 frames) dropped 1.24s -> 0.96s (-22%) once --warmup-num-frames 17 matched the served frame count. Both bit-identical. Compute-bound models (LongCat-Image, Qwen-Image, Z-Image, Cosmos3-Edge, FLUX.1-dev, Wan2.1-1.3B) saw no BCG or torch.compile gain — profiles show GEMM + flash-attention + already-fused norm/GELU saturating the device.
• Component CPU offload is often a pessimization, not a free win. Many presets enable --text-encoder-cpu-offload for memory-bound cards, but the whole model fits in 275 GB, so the H2D/D2H round trip is pure overhead. Dropping it on LingBot-Video-MoE was 8% faster end to end (bit-identical). On a large-VRAM single GPU, re-test each *-cpu-offload flag before keeping it.

Section 2: Lossy Optimizations

These options trade output quality for speed or VRAM savings. Results will differ from the baseline.

Quick Recipes

MiniMax-H3 first: lossless joint video/audio

H3 has a stricter contract than the generic recipes below. Keep its DiT eager for consistency ground truth, use Ulysses rather than Ring, do not enable CFG parallel, and leave the released overlapping tiled video-VAE decode in place.

Four H200 GPUs can keep the complete BF16/FP32 pipeline resident:

sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --performance-mode speed \
  --enable-torch-compile false \
  --port 30010

On 4x H100 80 GB, start from the fastest measured lossless resident topology:

sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --tp-size 2 \
  --ulysses-degree 2 \
  --performance-mode speed \
  --enable-torch-compile false \
  --port 30010

On B200/B300, the verified resident sweep uses 8 GPUs with Ulysses8. H3 also has a verified 4x B200 FSDP-capacity path, but FSDP all-gathers are a memory policy rather than the default latency choice. Benchmark the target topology with the H3 driver from sglang-diffusion-benchmark-profile.

A single 24 GB consumer card also runs H3, below the 2x32 GB the deployment picker documents. Keep the video VAE out of the stream and prefetch two layers (see "Choosing what goes in --layerwise-offload-components"):

sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 1 \
  --layerwise-offload-components dit,text_encoder \
  --dit-offload-prefetch-size 2 \
  --port 30010

Measured on 1x RTX 4090 24 GB at 672x384, 4 steps: 29 s per request, 22.1 GB peak. Cross-GPU is a separate matter on consumer cards -- 4090s have no P2P, so NCCL falls back to its SHM transport, and TP2 there segfaulted in ncclShmAllocateShareableBuffer during VAE decode at both 384 and 768. Single card avoids that path entirely.

Use the FL2VA partition for both t2va and fl2va; use --model-variant ref2va for image/video/audio reference conditioning. The root IDs are MiniMaxAI/MiniMax-H3 on Hugging Face and MiniMax/MiniMax-H3 on ModelScope. Do not point --model-path at a partition subdirectory.

Current H3 restrictions:

• torch.compile is opt-in experimentation only because it changes numerical output; it is not a lossless baseline
• Ring attention and CFG parallel are incompatible with the packed single denoising branch
• SageAttention is rejected for the current packed multi-segment attention
• --vae-config.parallel-decode-mode spatial, spatial_shard, and patch VAE decode are rejected after mismatches; use the default tiled recipe
• Breakable CUDA Graph is opt-in and signature-specific; the validated 1344x768 Ref2VA capture uses --bcg-text-buckets 5504, but it did not show a measured speedup
• the quality=high|medium|low Cache-DiT profiles and online FP8 are approximate; keep them outside lossless comparisons

Maximum speed, video model, multi-GPU, lossless (Wan A14B, 8 GPUs)

sglang generate --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --num-gpus 8 --enable-cfg-parallel --ulysses-degree 4 \
  --enable-torch-compile --warmup-mode request \
  --text-encoder-cpu-offload true \
  --prompt "..." --save-output

Note: --dit-layerwise-offload is enabled by default for Wan/MOVA video models and is often a good default, but still benchmark it on your exact workload if latency matters.

For Wan2.2 specifically:

• the nightly-aligned 4-GPU benchmark may use --enable-cfg-parallel --ulysses-degree=2 to keep CFG and ring behavior covered
• that is a coverage choice, not a guaranteed best-performance choice
• for pure latency tuning, benchmark pure Ulysses too, for example --ulysses-degree=4 --ring-degree=1 on 4 GPUs
• on 8 GPUs, compare pure --ulysses-degree=8 against --enable-cfg-parallel --ulysses-degree=4

Current-source model, 2 GPUs: LTX-2 two-stage

sglang generate --model-path Lightricks/LTX-2 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --prompt "A cat and a dog baking a cake together in a kitchen." \
  --width 768 --height 512 \
  --num-frames 121 \
  --seed 42 --num-gpus 2 --enable-cfg-parallel \
  --enable-torch-compile --warmup-mode request --save-output

Note: LTX-2 is a current-source benchmark preset rather than a nightly comparison case. The command uses runtime-default steps and guidance. LTX2TwoStagePipeline is a native path and auto-resolves the spatial upsampler plus distilled LoRA from the same model snapshot unless you override them.

Nightly-aligned model, 2 GPUs: LTX-2.3 TI2V two-stage

sglang generate --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --prompt "The cat starts walking slowly towards the camera." \
  --image-path "${ASSET_DIR}/cat.png" \
  --width 768 --height 512 \
  --num-frames 121 \
  --seed 42 --num-gpus 2 --cfg-parallel-size 2 \
  --enable-torch-compile --warmup-mode request --save-output

Note: this matches the nightly comparison case ltx2.3_twostage_ti2v_2gpus. The nightly config omits explicit steps and guidance, so this command omits them too and uses runtime defaults. Download ${ASSET_DIR}/cat.png with the benchmark/profile skill before running it.

Native baseline, 2 GPUs: LTX-2.3 one-stage

sglang generate --model-path Lightricks/LTX-2.3 \
  --prompt "A beautiful sunset over the ocean" \
  --negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." \
  --width 768 --height 512 \
  --num-frames 121 --fps 24 \
  --num-inference-steps 30 --guidance-scale 3.0 \
  --seed 1234 --num-gpus 2 \
  --enable-torch-compile --warmup-mode request --save-output

Note: use this as the native LTX2Pipeline baseline for LTX-2.3. It keeps the validated one-stage resolution and explicit LTX-2.3 sampling defaults, and matches the ltx23-one-stage benchmark preset in sglang-diffusion-benchmark-profile.

Skill-only stress target, 2 GPUs: LTX-2.3 two-stage high resolution

sglang generate --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --prompt "A beautiful sunset over the ocean" \
  --negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." \
  --width 1536 --height 1024 \
  --num-frames 121 --fps 24 \
  --num-inference-steps 30 --guidance-scale 3.0 \
  --seed 1234 --num-gpus 2 \
  --enable-torch-compile --warmup-mode request --save-output

Note: this is a high-resolution stress target for the native LTX-2.3 two-stage path. It matches the skill-only ltx23-two-stage benchmark preset, not a nightly comparison case.

Maximum speed, image model, single GPU, lossless

sglang generate --model-path <IMAGE_MODEL> \
  --enable-torch-compile --warmup-mode request \
  --dit-layerwise-offload false \
  --dit-cpu-offload false \
  --prompt "..." --save-output

Note: for image models, per-layer compute is smaller, so layerwise offload may not fully hide H2D transfer. Disable DiT layerwise and CPU offload if VRAM allows; otherwise a large image DiT can stay resident on CPU and make the denoise loop H2D-bound.

Launch-bound fixed-resolution path: Breakable CUDA Graph

sglang serve --model-path Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers \
  --performance-mode speed \
  --enable-torch-compile false \
  --enable-breakable-cuda-graph \
  --warmup-resolutions 1024x1024 \
  --port 30010

Keep torch.compile off, declare every production resolution, and benchmark the exact prompt-length distribution. Add --bcg-text-buckets only when the default buckets create excessive padding or miss a served prompt signature. Do not keep the timing unless the log contains [Diffusion BCG] captured and contains no disable, capture-failure, or serving signature MISSED message. For video, also match the captured frame and conditioning shape; WxH alone does not prove replay.

For a repeated discovery sweep, use the benchmark/profile helper. This runs lossless, extra-high, and high Eager/BCG ABBA pairs on one GPU set, then deletes the model group cache once:

python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py \
  --model <PRESET> --quality-bcg-matrix \
  --model-cache-root /path/to/task-owned/model-caches \
  --cleanup-model-cache

Compare cumulative request-quality fast paths

sglang generate --model-path <MODEL> \
  --quality lossless --prompt "..." --seed 42 \
  --perf-dump-path baseline.json --save-output

sglang generate --model-path <MODEL> \
  --quality extra-high --prompt "..." --seed 42 \
  --perf-dump-path quality-extra-high.json --save-output

sglang generate --model-path <MODEL> \
  --quality high --prompt "..." --seed 42 \
  --perf-dump-path quality-high.json --save-output

Keep every other flag fixed and compare the generated artifact as well as the perf dumps. high must retain every fusion observed under extra-high. If the model has no registered request-gated or high-only sites, either tier may be a no-op.

Image-edit baselines: JoyAI and FireRed

sglang generate --backend=sglang \
  --model-path jdopensource/JoyAI-Image-Edit-Diffusers \
  --prompt "Make the cat wear a red hat" \
  --image-path "${ASSET_DIR}/cat.png" \
  --width 1024 --height 1024 \
  --num-inference-steps 40 --guidance-scale 4.0 \
  --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 \
  --dit-layerwise-offload false --dit-cpu-offload false \
  --enable-torch-compile --warmup-mode request --save-output

sglang generate --backend=sglang \
  --model-path FireRedTeam/FireRed-Image-Edit-1.1 \
  --prompt "Make the cat wear a red hat" \
  --image-path "${ASSET_DIR}/cat.png" \
  --width 1024 --height 1024 \
  --num-inference-steps 40 --guidance-scale 4.0 \
  --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 \
  --dit-layerwise-offload false --dit-cpu-offload false \
  --enable-torch-compile --warmup-mode request --save-output

Use FireRedTeam/FireRed-Image-Edit-1.0 in the same command when comparing FireRed 1.0. These are native image-edit paths; keep the reference image, prompt, seed, and output size fixed when comparing denoise numbers. On H100, 2-GPU CFG parallel was faster than the otherwise matching 2-GPU Ulysses command: FireRed 1.0 improved from 13419.15 ms to 10955.90 ms, and FireRed 1.1 improved from 13414.72 ms to 10934.21 ms.

Hunyuan3D shape baseline

OUTPUT_DIR=$(python3 "$ENV_PY" print-output-dir --kind benchmarks --mkdir)
CONFIG_DIR="${OUTPUT_DIR}/generated_configs"
mkdir -p "${CONFIG_DIR}"
printf '{"paint_enable": false}\n' > "${CONFIG_DIR}/hunyuan3d-shape.json"

sglang generate --backend=sglang \
  --model-path tencent/Hunyuan3D-2 \
  --prompt "generate 3d mesh" \
  --image-path "${ASSET_DIR}/cat.png" \
  --config "${CONFIG_DIR}/hunyuan3d-shape.json" \
  --num-inference-steps 50 --guidance-scale 5.0 \
  --dit-layerwise-offload false --dit-cpu-offload false \
  --enable-torch-compile --warmup-mode request --save-output

For Hunyuan3D, treat Hunyuan3DShapeDenoisingStage as the primary latency metric. Mesh export and paint stages are useful end-to-end checks but should not drive DiT optimization decisions.

Low VRAM, decent speed (single GPU)

sglang generate --model-path <MODEL> \
  --enable-torch-compile --warmup-mode request \
  --dit-layerwise-offload --dit-offload-prefetch-size 0.1 \
  --text-encoder-cpu-offload true --vae-cpu-offload true \
  --prompt "..." --save-output

Maximum speed, lossy native path (SageAttention + Cache-DiT)

SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path <MODEL> \
  --attention-backend sage_attn \
  --dit-layerwise-offload false \
  --enable-torch-compile --warmup-mode request \
  --prompt "..." --save-output

Add native Cache-DiT knobs such as SGLANG_CACHE_DIT_SCM_PRESET=medium, SGLANG_CACHE_DIT_RDT=0.24, or SGLANG_CACHE_DIT_TAYLORSEER=true only after you have a BF16 baseline output to compare against.

For a diffusers-backend Cache-DiT YAML/JSON config baseline, make the fallback explicit:

sglang generate --backend diffusers --model-path <MODEL> \
  --cache-dit-config <config.yaml> \
  --dit-layerwise-offload false \
  --prompt "..." --save-output

Model-Specific Starting Points

Use these as first commands to benchmark, not as universal winners.

…(truncated)