| name | elsaa-efficient-low-rank-sparse-attention |
| version | 1.0.0 |
| description | Efficient Low-Rank and Sparse Attention Approximation (ELSAA) methodology for training Transformers with longer contexts while preserving both sharp token-level interactions and broad contextual mixing. |
| author | Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon |
| license | MIT |
| arxiv_id | 2607.20214v1 |
| tags | ["transformer","attention","efficiency","long-context","low-rank","sparse"] |
ELSAA: Efficient Low-Rank and Sparse Attention Approximation
Overview
ELSAA (Efficient Low-Rank and Sparse Attention Approximation) addresses the quadratic $N\times N$ attention score matrix bottleneck in Transformers by approximating the attention score operator itself rather than decomposing learned projection matrices.
Key Components
Dual-Branch Architecture
- Sparse Branch: Captures selected high-similarity interactions between query-key pairs
- Low-Rank Branch: Summarizes diffuse global interactions through compressed representation
Denominator-Aware Fusion
- Introduces a fusion term that scales the sparse branch according to its estimated attention mass relative to the low-rank branch
- Enables proper normalization over supports with very different denominator mass
Implementation Guidelines
Training Workflow
- Use standard dense projections to produce Q, K, V matrices
- Apply ELSAA approximation to the attention score operator
- Compute sparse and low-rank branches in parallel
- Apply denominator-aware fusion to combine branches
- Continue with standard Transformer forward pass
Benefits
- Avoids materializing the full quadratic score matrix
- Enables longer-context training
- Preserves both sharp token-level interactions and broad contextual mixing
- Compatible with existing Transformer architectures
Use Cases
- Long-context language modeling
- Document-level understanding tasks
- Memory-efficient Transformer training
- Applications requiring both local precision and global context
Activation Keywords
ELSAA, efficient attention, low-rank attention, sparse attention, long-context transformers, quadratic bottleneck
References
- arXiv: 2607.20214v1
- Authors: Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon
- Published: July 22, 2026
