lite-attention-temporal-sparse

name	lite-attention-temporal-sparse
title	LiteAttention: Temporal Sparse Attention for Diffusion Transformers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.11062
keywords	["Sparse Attention","Diffusion","Video Generation","Temporal Coherence","Efficiency"]
description	Accelerate video diffusion generation by exploiting temporal attention sparsity—skip redundant attention tiles across denoising steps using persistent skip masks, achieving 40% speedup with quality retention.

Accelerate Diffusion Video Generation with Temporal Sparse Attention

Video diffusion transformers spend significant computational budget computing self-attention across redundant spatial regions across denoising timesteps. LiteAttention exploits temporal coherence: tiles deemed non-essential at denoising step t remain non-essential at t+δ. By maintaining a persistent skip-mask and propagating skip decisions forward, LiteAttention reduces self-attention cost without dynamic recomputation at each step.

The key insight is that attention sparsity patterns are stable across adjacent denoising steps. Rather than recomputing which regions to skip (dynamic, expensive) or using fixed patterns throughout (inflexible), LiteAttention propagates early sparsity decisions, combining adaptivity with efficiency.

Core Concept

Video diffusion models iterate through T denoising steps, each computing full self-attention on spatial tiles (flattened image regions). Most tiles have negligible attention weights across most steps—they can be safely skipped. However, dynamically identifying skippable tiles at every step multiplies computational overhead.

LiteAttention maintains a Skip-Mask tracking which tiles are non-essential. The mask is computed once early (step 0 or 1) using QK-Skip: a simple condition on query-key dot products. Tiles with max local QK score significantly smaller than cumulative maximum are flagged as skippable. This mask propagates across timesteps, and only if attention patterns diverge significantly does the mask update.

Architecture Overview

• Skip-Mask Data Structure: Bitmask (or Skip-List for high sparsity) encoding which attention tiles can be skipped per timestep
• QK-Skip Algorithm: Fast condition checking max(QK) vs cumulative max to identify low-attention tiles
• Accumulated-Error Calibration: Assign variable error budgets to timesteps (early steps tolerate less error)
• Hardware Integration: Built atop FlashAttention3 for H100 GPUs; skip logic integrates into softmax pipeline
• Persistent Propagation: Skip decisions carry forward unless attention distribution shifts significantly

Implementation Steps

Step 1: Sparsity Detection. Identify tiles with negligible QK scores early in denoising.

def compute_qk_skip_mask(queries, keys, threshold_ratio=0.1):
    """
    Compute skip mask using QK-Skip condition.
    Tiles with max QK << cumulative_max are marked as skippable.
    queries, keys: (batch, num_tiles, head_dim)
    threshold_ratio: tile is skipped if max_qk < threshold_ratio * cumulative_max
    """
    
    qk_scores = queries @ keys.transpose(-, -)  

    skip_mask = []
     b  (qk_scores.shape[]):
        local_maxs = qk_scores[b].(dim=-).values  
        cumulative_max = local_maxs.()
        skip_threshold = threshold_ratio * cumulative_max

        
        skippable = local_maxs < skip_threshold
        skip_mask.append(skippable)

     torch.stack(skip_mask)