elsaa-efficient-low-rank-sparse-attention

name	elsaa-efficient-low-rank-sparse-attention
version	1.0.0
description	Efficient Low-Rank and Sparse Attention Approximation (ELSAA) methodology for training Transformers with longer contexts while preserving both sharp token-level interactions and broad contextual mixing.
author	Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon
license	MIT
arxiv_id	2607.20214v1
tags	["transformer","attention","efficiency","long-context","low-rank","sparse"]

ELSAA: Efficient Low-Rank and Sparse Attention Approximation

Overview

ELSAA (Efficient Low-Rank and Sparse Attention Approximation) addresses the quadratic $N\times N$ attention score matrix bottleneck in Transformers by approximating the attention score operator itself rather than decomposing learned projection matrices.

Key Components

Dual-Branch Architecture

• Sparse Branch: Captures selected high-similarity interactions between query-key pairs
• Low-Rank Branch: Summarizes diffuse global interactions through compressed representation

Denominator-Aware Fusion

• Introduces a fusion term that scales the sparse branch according to its estimated attention mass relative to the low-rank branch
• Enables proper normalization over supports with very different denominator mass

Implementation Guidelines

Training Workflow

1. Use standard dense projections to produce Q, K, V matrices
2. Apply ELSAA approximation to the attention score operator
3. Compute sparse and low-rank branches in parallel
4. Apply denominator-aware fusion to combine branches
5. Continue with standard Transformer forward pass

Benefits

• Avoids materializing the full quadratic score matrix
• Enables longer-context training
• Preserves both sharp token-level interactions and broad contextual mixing
• Compatible with existing Transformer architectures

Use Cases

• Long-context language modeling
• Document-level understanding tasks
• Memory-efficient Transformer training
• Applications requiring both local precision and global context

Activation Keywords

ELSAA, efficient attention, low-rank attention, sparse attention, long-context transformers, quadratic bottleneck

References

• arXiv: 2607.20214v1
• Authors: Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon
• Published: July 22, 2026