Llada2 Diffusion Scaling

Skill Summary

LLaDA2.0 scales diffusion language models to 100B parameters by systematically converting pre-trained autoregressive checkpoints through Warmup-Stable-Decay (WSD) training strategy. Rather than training from scratch, the approach progressively transforms AR checkpoints across three phases: Warmup increases block size gradually transforming AR model into full-sequence masked diffusion model, Stable performs full-sequence diffusion training on large-scale corpora, and Decay reverts to smaller blocks for practical inference. Additional innovations include document-level attention masking and confidence-aware parallel training, achieving 535 tokens-per-second inference speed.

When To Use

• Scaling diffusion language models to 100B+ parameters
• Projects converting existing large AR checkpoints to parallel generation
• Scenarios where parallel decoding benefits outweigh autoregressive efficiency
• Research on efficient diffusion model scaling

When NOT To Use

• Applications requiring pure autoregressive inference properties
• Scenarios with strict latency requirements (WSD training has overhead)
• Small models where diffusion complexity adds overhead
• Domains benefiting specifically from left-to-right generation order

Core Technique

Three coordinated training phases enable efficient transformation:

1. Warmup Phase "Progressively increase the block size in block diffusion language models (BDLM) to gradually transform the AR model into a full-sequence masked diffusion language model." This gradual transformation preserves learned representations while shifting from AR to diffusion paradigm.

2. Stable Phase Perform full-sequence diffusion training on large-scale corpora with two innovations:

• Document-level attention masking: prevents spurious cross-document dependencies in packed training
• Complementary masking: during post-training ensures near-100% data utilization

3. Decay Phase Revert to efficient smaller block sizes for practical inference, maintaining diffusion model benefits while achieving practical throughput (535 tokens-per-second).

4. Confidence-Aware Parallel Training Sharpen predictions for faster decoding, allowing better utilization of parallel generation.

Results

• 100B parameter model
• 535 tokens-per-second inference speed
• Competitive performance vs. comparable autoregressive baselines
• Diffusion models' parallel decoding advantages preserved

Implementation Notes

Start with pre-trained 100B AR checkpoint. Implement progressive block size increase during warmup. Implement document-level attention masking for packed training. Apply complementary masking for 100% data utilization. Decay to smaller blocks for efficient inference. Measure throughput and accuracy.

References

• Original paper: LLaDA2.0 (Dec 2025)
• Block diffusion language models
• Large-scale model training and conversion