sparse-vdit-video-acceleration

name	sparse-vdit-video-acceleration
title	Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.03065
keywords	["video-diffusion","sparse-attention","acceleration","efficiency","inference-optimization"]
description	Accelerate video diffusion transformer inference by 1.58-1.85× through discovering and exploiting sparse attention patterns that exhibit diagonal, multi-diagonal, and vertical-stripe structures.

Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

Core Concept

Sparse-vDiT identifies a critical inefficiency in video diffusion transformers: attention mechanisms consume 77-81% of inference latency, yet their attention maps exhibit predictable sparse patterns (diagonal, multi-diagonal, vertical-stripe structures) that are largely independent of input content. Rather than computing dense attention, the framework systematically discovers and exploits these patterns per layer and head, achieving 1.58-1.85× actual speedup with minimal quality degradation.

The key insight: attention patterns correlate strongly with layer depth, not input variations, enabling offline optimization that generalizes across different inputs.

Architecture Overview

• Pattern Discovery: Identifies three recurring sparse structures: diagonal, multi-diagonal, and vertical-stripe attention patterns
• Offline Diffusion Search: Per-layer, per-head algorithm evaluating five attention modes (dense, diagonal, multi-diagonal, vertical-stripe, mixed) to determine optimal sparsity
• Hardware-Aware Kernels: Specialized CUDA/Triton implementations of sparse patterns with minimal overhead
• Head Fusion: Groups heads with identical patterns within layers, further accelerating computation
• Quality-Efficiency Trade-off: Balances sparsity penalty against reconstruction fidelity via tunable parameters

Implementation

1. Offline Sparse Search: Evaluate five attention modes per head using MSE loss with sparsity penalty

# Sparse pattern evaluation
def offline_sparse_search(layer_module, validation_data):
    """
    For each layer and head, find optimal sparse attention pattern.
    Searches five modes: dense, diagonal, multi-diagonal, vertical, mixed.
    """
    modes = ['dense', 'diagonal', 'multi_diagonal', 'vertical_stripe', 'mixed']
    best_patterns = {}

    for head_idx in range(num_heads):
        mode_scores = {}

        for mode in modes:
            
            sparse_attention = apply_pattern(layer_module, head_idx, mode)

            
            output = compute_with_sparse_attention(sparse_attention, validation_data)
            mse_loss = mean_squared_error(output, dense_output)
            sparsity_penalty = lambda_param * compute_sparsity(sparse_attention)

            mode_scores[mode] = mse_loss + sparsity_penalty

        
        best_mode = (mode_scores, key=mode_scores.get)
        best_patterns[head_idx] = best_mode

     best_patterns