Implement dense-sparse switchable attention enabling LLMs to scale from short to long sequences with 4× speedup and 98-99.7% performance retention, requiring no extra parameters by reusing pretrained attention weights through trainable sparse pattern selection.
Implement dense-sparse switchable attention enabling LLMs to scale from short to long sequences with 4× speedup and 98-99.7% performance retention, requiring no extra parameters by reusing pretrained attention weights through trainable sparse pattern selection.
InfLLM-V2: Dense-Sparse Switchable Attention
Outcome
Enable language models pretrained on short sequences (4k tokens) to seamlessly adapt to long-context processing (32k+ tokens) with 4× computational speedup, 98.1% long-context performance retention, and zero additional parameters through a trainable sparse attention framework that dynamically switches attention patterns based on sequence length.
Problem Context
Standard Transformer self-attention exhibits quadratic complexity O(n²) in sequence length, creating severe computational and memory bottlenecks when processing long documents. Existing sparse attention approaches introduce substantial new parameters (NSA), disrupt the conventional pretrain-short-finetune-long training workflow, and cause slow convergence. Models need to adapt from short-sequence pretraining (where attention is computationally feasible) to long-sequence inference (where dense attention becomes prohibitive) without architectural mismatch or retraining overhead.
Core Concept
InfLLM-V2 implements a parameter-free dense-sparse switchable attention mechanism that reuses all pretrained dense attention parameters while introducing trainable sparse pattern selection. The framework automatically selects dense attention for short sequences and sparse attention for long sequences, eliminating the parameter explosion of competing methods. Key insight: rather than learning new KV projections for sparse heads, the method extracts sparse tokens from the same pretrained dense attention output space, preserving learned representations while reducing computation.
The mechanism integrates three sparse pattern types—Selected Attention (learns which tokens matter most), Sliding Attention (maintains local context), and Compressed Attention (hierarchical token aggregation)—into a unified sparse pathway controlled by learned routing coefficients. This unification removes redundant output projections and ensures compatibility with standard dense attention training.
Architecture Overview
Switchable Attention Framework:
Dense Pathway: Full self-attention for sequences below a learnable threshold length, using original pretrained weights
Sparse Pathway: Multi-pattern selection combining selected, sliding, and compressed attention for longer sequences
Pattern Router: Trainable coefficients determining contribution weight of each sparse pattern without adding KV projection parameters
Hardware Kernel: Two-pass CUDA implementation with LSE approximation fusing head-group summation into FlashAttention loop, reducing GPU memory transfers
Three Core Innovations:
Parameter-Free Adaptation: No new KV projection weights; sparse patterns reuse dense attention's pretrained output space
Unified Sparse Patterns: Consolidated Selected + Sliding + Compressed attention into single module removing redundant pathways
Sequence-Length Switching: Automatic mode selection based on input length with no training instability or architectural mismatch
Implementation
Step 1: Dense Attention Layer Foundation
Define the base dense attention layer reusing pretrained parameters. This serves as the initialization for both dense and sparse pathways.
Configure training to enable seamless short-to-long adaptation without architectural mismatch.
defcreate_switchable_model(
num_layers: int,
hidden_size: int,
num_heads: int,
intermediate_size: int,
vocab_size: int,
dense_threshold: int = 2048,
) -> nn.Module:
"""
Construct full language model with switchable attention throughout.
Compatible with standard training pipelines.
"""
layers = nn.ModuleList([
TransformerBlockWithSwitchableAttention(
hidden_size,
num_heads,
intermediate_size,
dense_threshold=dense_threshold,
)
for _ inrange(num_layers)
])
embedding = nn.Embedding(vocab_size, hidden_size)
lm_head = nn.Linear(hidden_size, vocab_size)
classLanguageModel(nn.Module):
def__init__(self):
super().__init__()
self.embedding = embedding
self.layers = layers
self.norm = nn.LayerNorm(hidden_size)
self.lm_head = lm_head
defforward(self, input_ids: torch.Tensor) -> torch.Tensor:
hidden_states = self.embedding(input_ids)
for layer inself.layers:
hidden_states = layer(hidden_states)
hidden_states = self.norm(hidden_states)
logits = self.lm_head(hidden_states)
return logits
return LanguageModel()
deffinetune_for_long_context(
model: nn.Module,
train_dataloader,
optimizer,
num_epochs: int = 2,
device: str = "cuda",
gradient_accumulation_steps: int = 4,
):
"""
Fine-tune pretrained short-context model for long-context capability.
Key procedure:
1. Model starts with pretrained dense attention parameters
2. Pattern router initialized equally, then learns during fine-tuning
3. No architectural changes, no new KV projections
4. Gradual activation of sparse patterns as sequence length increases
"""
model = model.to(device)
model.train()
for epoch inrange(num_epochs):
total_loss = 0for step, batch inenumerate(train_dataloader):
input_ids = batch["input_ids"].to(device)
labels = batch["labels"].to(device)
logits = model(input_ids)
loss = nn.functional.cross_entropy(
logits.view(-1, logits.shape[-1]),
labels.view(-1),
)
loss = loss / gradient_accumulation_steps
loss.backward()
if (step + 1) % gradient_accumulation_steps == 0:
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
optimizer.zero_grad()
total_loss += loss.item()
print(f"Epoch {epoch + 1}: Loss = {total_loss / len(train_dataloader):.4f}")
Practical Guidance
Hyperparameters and Configuration
Parameter
Recommended
Notes
dense_threshold
2048-4096
Sequence length below which dense attention is used. Adjust based on GPU memory.
block_size
64-128
Window size for sliding attention and block size for compression.
topk_ratio
0.05-0.2
Fraction of tokens selected in selected-attention pattern. Lower = faster but potentially lower quality.
pattern_initialization
[0.33, 0.33, 0.33]
Initialize pattern weights equally; training adapts automatically.
num_fine_tune_epochs
1-3
Few epochs sufficient since base parameters frozen.
learning_rate_sparse_router
1e-4 to 5e-4
Only pattern_weights need tuning; keep LR conservative.
When to Use
Deploying pretrained LLMs that require long-context capability without retraining from scratch
Inference scenarios with variable sequence lengths requiring dynamic efficiency
Models with computational constraints (mobile, edge devices with GPU)
Fine-tuning budgets limited to a few epochs after pretraining
Applications requiring seamless transition from short to long contexts without latency spikes
Multi-modal systems where context length varies significantly per sample
When NOT to Use
Very long sequences (100k+ tokens) where even sparse patterns become expensive; consider pure sparse methods or retrieval-augmented approaches
Pretraining from scratch; use standard dense attention or alternative sparse methods without the switching overhead
Models where maintaining exact dense attention behavior is critical (e.g., exact replication of reference outputs); sparse approximation introduces mathematical differences
Scenarios requiring causal attention without forward peeking; ensure sliding window respects causality (only attend to past tokens)
Hardware without efficient sparse attention kernels (CPUs, older GPUs); performance gains diminish without optimized CUDA implementation
Tasks requiring all-to-all token interaction (e.g., some music or video tasks); sparsity may hurt performance
Common Pitfalls
Threshold too low: Setting dense_threshold too low forces sparse patterns for short sequences where dense is already efficient. Set threshold where GPU memory becomes the bottleneck, typically 2-4k tokens on A100.
Unbalanced pattern weights: If one sparse pattern dominates, others contribute little. Monitor pattern weight evolution; add entropy regularization if weights collapse to single pattern.
Ignoring position encodings: Sparse attention may not properly extend pretrained position embeddings beyond their training range. Use position interpolation or ALiBi if extending significantly beyond pretraining length.
Gradient flow to frozen parameters: Dense attention parameters are frozen during fine-tuning to preserve learned representations. Ensure gradients only flow through pattern_weights, not through q_proj, k_proj, v_proj if using frozen parameter setup.
Memory bottleneck in pattern computation: Selected attention requires computing full similarity matrix before top-k selection. For very long sequences, compute top-k in blocks or use approximate methods like LSH to reduce memory.
Training instability during transition: Some models show training loss spikes when fine-tuning switches between dense and sparse pathways. Use gradient warmup or gradual threshold decay: start with high threshold, gradually lower during training to activate sparsity.
