Accelerate video diffusion models using sparse radial attention that exploits energy decay patterns. Achieves 3.7× speedup on long videos while maintaining quality through O(n log n) complexity instead of O(n²).
Accelerate video diffusion models using sparse radial attention that exploits energy decay patterns. Achieves 3.7× speedup on long videos while maintaining quality through O(n log n) complexity instead of O(n²).
Radial Attention: Efficient Long-Sequence Attention with Energy Decay
Video diffusion models need to attend over thousands of spatial-temporal tokens to generate coherent videos. Standard full attention requires O(n²) memory and computation—infeasible for long videos (256 or more frames). The challenge is finding a sparse pattern that preserves video quality while reducing complexity.
Radial Attention exploits a discovered property of video diffusion: attention scores naturally decay as distance between tokens increases, following an exponential pattern similar to physical signal decay. By converting this energy decay into a static sparse attention mask, you achieve O(n log n) complexity while capturing the meaningful long-range dependencies videos need.
Core Concept
The key insight is that dense attention weights are concentrated locally. In a 256-frame video:
Tokens attend strongly to nearby frames (temporal locality)
Tokens attend strongly to nearby spatial positions (spatial locality)
Attention to distant frames decays exponentially with distance
You can approximate this decay pattern with a sparse mask that has logarithmic bands
Rather than computing all n² attention pairs, radial attention creates attention bands:
Inner band (frames 0-5): full attention density
Next band (frames 5-15): 50% density
Next band (frames 15-50): 25% density
Outer band (frames 50+): sparse attention
This pattern requires only O(n log n) total comparisons while preserving the exponential decay structure the model learned.
Architecture Overview
Radial attention modifies the attention mechanism:
Sparse Attention Mask: A static binary mask (computed once, reused in every forward pass) that defines which token pairs compute attention
Exponential Band Structure: Attention density decreases exponentially with distance: density(distance) = (1/2)^floor(log₂(distance))
Spatial and Temporal Bands: Both spatial proximity (within frame) and temporal proximity (across frames) use the same exponential decay pattern
LoRA Extension: Lightweight fine-tuning adapters enable efficient adaptation to longer sequences without full retraining
Implementation
Step 1: Compute the radial attention mask
Create a sparse attention pattern that encodes exponential decay.
"""
Generate a sparse attention mask with radial structure.
Attention density decays exponentially with distance.
seq_length: total tokens (num_frames * spatial_tokens)
num_spatial_tokens: tokens per frame (e.g., 16x16 = 256)
num_temporal_frames: number of video frames
"""
# Create 2D attention mask
bool
for
in
range
# Which frame and spatial position is this token?
for
in
range
# Compute distances
abs
int
# Radial attention rule: attend based on distance decay
if
True
return
def
spatial_distance_on_grid
pos1, pos2, grid_size
"""
Compute Manhattan distance on a 2D spatial grid.
"""
Enable efficient adaptation to sequences longer than training length using Low-Rank Adapters.
from peft import get_peft_model, LoraConfig
defadd_lora_for_sequence_extension(model_with_radial_attention,
target_seq_length=512):
"""
Add LoRA adapters to enable efficient fine-tuning for longer sequences.
LoRA has low rank (r=32-64) so training is cheap.
"""# Target attention layers for LoRA
lora_config = LoraConfig(
r=32, # Low rank
lora_alpha=64,
target_modules=['q_proj', 'v_proj'], # Key and value projections
lora_dropout=0.05,
bias='none',
task_type='CAUSAL_LM'
)
# Apply LoRA to attention layers
model_lora = get_peft_model(model_with_radial_attention, lora_config)
# Generate a few examples with longer sequences# Fine-tune on these examplesdefcreate_long_sequence_examples(num_examples=10):
examples = []
for _ inrange(num_examples):
# Create synthetic long-sequence data
video_prompt = "Generate a smooth video transition"
examples.append(video_prompt)
return examples
long_examples = create_long_sequence_examples()
# Fine-tune with LoRA
optimizer = torch.optim.AdamW(model_lora.parameters(), lr=1e-4)
for epoch inrange(3):
for prompt in long_examples:
# Generate with target_seq_length# (requires updated mask creation for new length)
loss = model_lora.forward_with_loss(
prompt,
num_frames=int(target_seq_length / 256)
)
optimizer.zero_grad()
loss.backward()
optimizer.step()
return model_lora
Practical Guidance
Hyperparameter
Recommended Value
Notes
Decay factor
0.5-0.7
Controls sparsity pattern; 0.5 is standard
Temporal weight
2.0-3.0
Temporal distance matters more than spatial
Minimum density
0.05
Prevent total attention starvation
Band width
1
Logarithmic bands; change rarely
LoRA rank
32-64
Higher rank = more capacity but slower
When to use Radial Attention:
You're generating long videos (128+ frames)
You're memory-bound (not compute-bound)
Standard full attention causes OOM errors
You want 2-3× speedup without retraining from scratch
When NOT to use Radial Attention:
Your videos are short (< 64 frames; speedup negligible)
You're compute-bound (GPU util already high; memory not the bottleneck)
You need maximum quality regardless of speed (sparse attention trades some quality)
You can't modify the attention mechanism (inference-only systems)
Common pitfalls:
Mask too sparse: If density is too low, the model can't attend to important long-range patterns. Increase decay_factor (e.g., 0.7 instead of 0.5).
Flicker across frames: If temporal bands are too tight, frames don't correlate well. Increase temporal_weight (e.g., 3.0).
LoRA rank too low: If fine-tuning for longer sequences fails, increase LoRA rank (64 or 128).
Wrong sequence length mask: If you change num_frames during inference, you must regenerate the mask. Cache-friendly designs should reuse masks when possible.
