Sparse Vdit Video Acceleration

Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

Core Concept

Sparse-vDiT identifies a critical inefficiency in video diffusion transformers: attention mechanisms consume 77-81% of inference latency, yet their attention maps exhibit predictable sparse patterns (diagonal, multi-diagonal, vertical-stripe structures) that are largely independent of input content. Rather than computing dense attention, the framework systematically discovers and exploits these patterns per layer and head, achieving 1.58-1.85× actual speedup with minimal quality degradation.

The key insight: attention patterns correlate strongly with layer depth, not input variations, enabling offline optimization that generalizes across different inputs.

Architecture Overview

• Pattern Discovery: Identifies three recurring sparse structures: diagonal, multi-diagonal, and vertical-stripe attention patterns
• Offline Diffusion Search: Per-layer, per-head algorithm evaluating five attention modes (dense, diagonal, multi-diagonal, vertical-stripe, mixed) to determine optimal sparsity
• Hardware-Aware Kernels: Specialized CUDA/Triton implementations of sparse patterns with minimal overhead
• Head Fusion: Groups heads with identical patterns within layers, further accelerating computation
• Quality-Efficiency Trade-off: Balances sparsity penalty against reconstruction fidelity via tunable parameters

Implementation

1. Offline Sparse Search: Evaluate five attention modes per head using MSE loss with sparsity penalty

# Sparse pattern evaluation
def offline_sparse_search(layer_module, validation_data):
    """
    For each layer and head, find optimal sparse attention pattern.
    Searches five modes: dense, diagonal, multi-diagonal, vertical, mixed.
    """
    modes = ['dense', 'diagonal', 'multi_diagonal', 'vertical_stripe', 'mixed']
    best_patterns = {}

    for head_idx in range(num_heads):
        mode_scores = {}

        for mode in modes:
            # Apply sparsity pattern to attention
            sparse_attention = apply_pattern(layer_module, head_idx, mode)

            # Evaluate reconstruction fidelity and sparsity
            output = compute_with_sparse_attention(sparse_attention, validation_data)
            mse_loss = mean_squared_error(output, dense_output)
            sparsity_penalty = lambda_param * compute_sparsity(sparse_attention)

            mode_scores[mode] = mse_loss + sparsity_penalty

        # Select mode with lowest combined loss
        best_mode = min(mode_scores, key=mode_scores.get)
        best_patterns[head_idx] = best_mode

    return best_patterns  # Per-head sparse patterns

2. Sparse Pattern Definition: Implement five attention modes with increasing efficiency

# Sparse attention patterns
def create_sparse_mask(seq_len, mode):
    """Define attention patterns with different sparsity structures."""
    mask = torch.ones(seq_len, seq_len)

    if mode == 'diagonal':
        # Only attend to positions within local window
        for i in range(seq_len):
            for j in range(seq_len):
                if abs(i - j) > window_size:
                    mask[i, j] = 0

    elif mode == 'multi_diagonal':
        # Multiple diagonal bands for increased expressivity
        for band_offset in band_offsets:
            for i in range(seq_len):
                j = i + band_offset
                if 0 <= j < seq_len:
                    mask[i, j] = 1

    elif mode == 'vertical_stripe':
        # Attend to specific column positions (useful for spatial patterns)
        for i in range(seq_len):
            for stripe_pos in stripe_positions:
                mask[i, stripe_pos] = 1

    elif mode == 'mixed':
        # Combination of patterns for flexibility
        mask = combine_patterns([diagonal, multi_diagonal, vertical])

    return mask

3. Hardware-Accelerated Kernels: Deploy specialized implementations for sparse patterns

# CUDA kernel invocation (pseudo-code)
def forward_sparse_attention(query, key, value, sparse_pattern, mode):
    """
    Execute sparse attention using hardware-optimized kernels.
    Pattern determines kernel selection.
    """
    if mode == 'diagonal':
        # Use diagonal band CUDA kernel
        output = cuda_diagonal_attention(query, key, value, window_size)

    elif mode == 'vertical_stripe':
        # Use vertical stripe kernel with pre-computed stripe positions
        output = cuda_stripe_attention(query, key, value, stripe_indices)

    else:
        # Fall back to generic sparse kernel
        output = cuda_sparse_attention(query, key, value, sparse_pattern)

    return output

4. Head Fusion: Group heads with identical patterns to reduce kernel launches

def fuse_attention_heads(layer_module, patterns):
    """
    Identify heads with identical sparse patterns and fuse computation.
    Reduces kernel invocation overhead.
    """
    pattern_groups = {}

    for head_idx, pattern in patterns.items():
        if pattern not in pattern_groups:
            pattern_groups[pattern] = []
        pattern_groups[pattern].append(head_idx)

    fused_heads = {}
    for pattern, head_indices in pattern_groups.items():
        if len(head_indices) > 1:
            # Fuse multiple heads into single computation
            fused_heads[pattern] = head_indices

    return fused_heads

5. Tuning Parameters: Adjust λ (sparsity penalty) and ε (overall sparsity ratio) for desired speed-quality tradeoff
   • Higher λ: More aggressive sparsification, faster but lower quality
   • Lower λ: Denser attention, higher quality but slower

Practical Guidance

When to Apply:

• Deploying video diffusion models in latency-sensitive applications
• Need to reduce inference cost while maintaining visual quality
• Have diverse video datasets with stable attention patterns

Implementation Prerequisites:

• Video diffusion transformer models (CogVideoX, HunyuanVideo, or similar)
• Representative validation dataset for offline search
• GPU with good sparse tensor support (A100 or newer)

Performance Expectations:

• Theoretical FLOP reduction: 2.09-2.38×
• Actual wall-clock speedup: 1.58-1.85× (accounting for kernel overhead)
• Quality metrics (PSNR): 22.59-27.09 with minimal visual degradation
• Speedup varies by sparsity pattern type (vertical-stripe often fastest)

Key Tuning Decisions:

• Search window size for diagonal patterns: balance locality vs. receptive field
• Number of stripe positions for vertical patterns: affects semantic coherence
• Head fusion threshold: fuse if >70% heads share identical patterns
• Validation set size: 50-100 videos typically sufficient for stable patterns

Common Issues:

• Patterns vary across video resolutions—search must match deployment resolution
• Very low λ values can cause patterns to collapse to invalid configurations
• Some models may have task-specific patterns not captured by general validation data
• Kernel availability limits patterns (pure custom kernels needed for complex patterns)

Reference

Demonstrated on CogVideoX1.5, HunyuanVideo, and Wan2.1. Achieves 2.09-2.38× theoretical speedup with 1.58-1.85× actual speedup through pattern-optimized sparse kernels and head fusion. Quality maintained across standard video quality metrics.