| name | sparsemm-visual-attention |
| title | SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.05344 |
| keywords | ["multimodal-models","attention-sparsity","kv-cache-optimization","visual-language","efficiency"] |
| description | Discovers that <5% of attention heads process visual information in MLLMs; introduces SparseMM for asymmetric KV-cache allocation achieving 1.38x acceleration and 52% memory reduction. |
SparseMM: Head Sparsity in Multimodal LLMs
Core Concept
Multimodal Large Language Models (MLLMs) allocate uniform computational resources across all attention heads, despite empirical evidence that the vast majority are linguistically-focused. Only approximately 5% of heads actively engage with visual content. SparseMM discovers and exploits this sparsity by assigning asymmetric KV-cache budgets: visual heads receive full cache while text-only heads share a smaller baseline, achieving significant acceleration without accuracy loss. The identification is training-free, using OCR-based spatial grounding to quantify head-level visual relevance.
Architecture Overview
- Visual Head Discovery: OCR-based method identifying <5% of attention heads as "visual heads" across diverse MLLM architectures
- Training-Free Identification: No model retraining required; scores based on task-specific spatial grounding patterns
- Asymmetric Cache Allocation: Three-part strategy combining local windows, baseline allocation, and score-based priority distribution
- KV-Cache Optimization Framework: Prioritizes visually-important heads while maintaining performance guarantees
- Comprehensive Evaluation: Tested across DocVQA, OCRBench, TextVQA, MMBench benchmarks
- Real-Time Acceleration: Achieves 1.38× speedup with 52% memory reduction
Implementation
The following code demonstrates visual head discovery and SparseMM cache allocation:
import torch
import torch.nn as nn
from typing import Dict, List, Tuple, Optional
import numpy as np
class VisualHeadDiscovery:
"""
Identifies visual heads in MLLMs using OCR-based spatial grounding.
"""
def __init__(self, model: nn.Module, num_heads: int = 32):
self.model = model
self.num_heads = num_heads
self.visual_scores = None
def compute_visual_relevance(self, image: torch.Tensor,
text_input: str,
ocr_boxes: Dict[str, List[Tuple[int, int, int, int]]]) -> Dict[int, float]:
"""
Compute visual relevance score for each attention head.
image: (H, W, 3) input image
text_input: input text/question
ocr_boxes: dict mapping words to bounding boxes in image
Returns: dict mapping head_idx to visual_score in [0, 1]
"""
visual_scores = {}
with torch.no_grad():
attention_patterns = self._extract_attention_patterns(image, text_input)
head_idx (.num_heads):
head_attention = attention_patterns[head_idx]
visual_score = ._compute_grounding_score(head_attention, ocr_boxes, image.shape)
visual_scores[head_idx] = visual_score
visual_scores
() -> torch.Tensor:
batch_size =
attention_patterns = torch.randn(.num_heads, , )
attention_patterns
() -> :
ocr_boxes:
h, w = image_shape[:]
ocr_mask = torch.zeros(h, w, dtype=torch.float32)
word, boxes ocr_boxes.items():
x1, y1, x2, y2 boxes:
ocr_mask[y1:y2, x1:x2] =
seq_len = attention_weights.shape[]
patch_size = h // (np.sqrt(seq_len))
spatial_attention = torch.zeros(h, w)
i (seq_len):
y = (i // (np.sqrt(seq_len))) * patch_size
x = (i % (np.sqrt(seq_len))) * patch_size
y_end = (y + patch_size, h)
x_end = (x + patch_size, w)
spatial_attention[y:y_end, x:x_end] = attention_weights[i].mean()
overlap = (spatial_attention * ocr_mask).() / (ocr_mask.() + )
(overlap.clamp(, ))
:
():
.visual_scores = visual_scores
.num_heads = num_heads
.cache_budget_gb = cache_budget_gb
.model_dim = model_dim
.head_dim = model_dim // num_heads
.ranked_heads = (visual_scores.items(),
key= x: x[], reverse=)
() -> [, ]:
total_bytes = .cache_budget_gb * ( ** )
bytes_per_token_per_head = * * .head_dim
window_size =
local_window_budget = window_size * .num_heads * bytes_per_token_per_head
remaining_budget = total_bytes - local_window_budget
baseline_per_head = remaining_budget / (.num_heads * bytes_per_token_per_head)
baseline_tokens = (baseline_per_head)
visual_budget = remaining_budget *
visual_tokens_per_head = (visual_budget / (bytes_per_token_per_head * (
score _, score .ranked_heads[:]
)))
cache_allocation = {}
head_idx, visual_score .visual_scores.items():
cache_allocation[head_idx] = window_size + baseline_tokens
visual_score > :
cache_allocation[head_idx] += (visual_score * visual_tokens_per_head)
cache_allocation
() -> torch.Tensor:
allocated_tokens = cache_allocation.get(head_idx, key_cache.shape[])
k_sparse = key_cache[-allocated_tokens:] key_cache.shape[] > allocated_tokens key_cache
v_sparse = value_cache[-allocated_tokens:] value_cache.shape[] > allocated_tokens value_cache
scores = torch.matmul(query, k_sparse.transpose(-, -)) / (.head_dim ** )
attn = torch.softmax(scores, dim=-)
output = torch.matmul(attn, v_sparse)
output
(nn.Module):
():
().__init__()
.model = model
.num_heads = num_heads
.discovery = VisualHeadDiscovery(model, num_heads)
.sparse_mm =
.cache_allocation =
():
visual_scores = .discovery.compute_visual_relevance(
sample_image, sample_text, ocr_boxes
)
.sparse_mm = SparseMM(visual_scores, .num_heads)
.cache_allocation = .sparse_mm.allocate_cache(sequence_length=)
() -> torch.Tensor:
.sparse_mm :
.model(image, text)
output = .model(image, text,
attention_fn= q, k, v, h: .sparse_mm.apply_sparse_attention(
q, k, v, .cache_allocation, h
))
output
Practical Guidance
Visual Head Threshold: Heads with visual score > 0.1 are reliably visual across different MLLMs. Use this threshold for identifying visual heads without tuning.
OCR-Based Scoring: Ensure OCR quality before computing visual scores. Poor OCR results in noisy head rankings. Consider using multiple OCR engines and averaging their confidence.
Cache Budget Allocation: The three-part strategy (window + baseline + bonus) should maintain at least baseline tokens for all heads to prevent degenerate attention patterns.
Window Size Selection: Local window of 64 tokens captures recent context effectively. Increase to 128 for tasks requiring long-range dependencies within recent history.
Model Variants: The method works across different MLLM architectures (LLaVA, Qwen-VL, etc.) because visual head sparsity is universal. Test on your target model.
Deployment Integration: Apply SparseMM at inference time only; no retraining needed. Hook into attention computation to apply sparse cache lookup by head.
Reference
SparseMM achieves strong efficiency-accuracy tradeoffs:
- 1.38× real-time acceleration with sparse cache
- 52% memory reduction compared to full cache
- Consistent performance across DocVQA, OCRBench, TextVQA, MMBench
The discovery that attention heads in MLLMs exhibit extreme sparsity for visual information enables principled optimization without retraining. This makes it immediately applicable to existing deployed models.
