clsa-cross-layer-sparse-attention

name	clsa-cross-layer-sparse-attention
description	Cross-layer sparse attention sharing routing index across decoder layers for 7.6x decoding speedup and 17.1x throughput improvement at 128K context
version	1.0.0
category	ai_collection
tags	["deep-learning","transformer","attention","efficiency","long-context"]
arxiv	2606.06467v1
paper_title	You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
authors	["Yutao Sun","Yanqi Zhang","Li Dong","Jianyong Wang","Furu Wei"]
published	2026-06-04T00:00:00.000Z
activation_keywords	["sparse attention","cross-layer","KV-sharing","routing index","long-context","decoding efficiency","YOCO"]

CLSA: Cross-Layer Sparse Attention

Core Innovation

Share routing index across cross-decoder layers (not just KV cache), computing token-level top-k selection once and reusing across layers.

Problem Addressed

Existing sparse attention trade-offs:

• Block sparse: Strong acceleration, noticeable quality loss
• Token sparse: More accurate, limited speedup (expensive routing)

Methodology

Architecture (built on YOCO)

Cross-decoder layers:
├── Shared KV cache (YOCO base)
├── Shared routing index (CLSA innovation)
│   └── Compute once → reuse across layers
└── Token-level top-k selection

Key Benefits

1. Index once, reuse everywhere: Amortize routing overhead
2. Fine-grained selectivity: Preserve token-sparse accuracy
3. Joint optimization: Pre-filling + KV-cache + decoding

Performance

• Decoding speedup: 7.6x at 128K context
• Throughput: 17.1x overall improvement
• Quality: Maintains accuracy across benchmarks

Implementation Pattern

class CrossLayerSparseAttention:
    def __init__(self, base_model, top_k_ratio=0.1):
        self.kv_cache = SharedKVCache()  # YOCO-style
        self.router_index = None  # Shared across layers
        self.top_k = top_k_ratio
        
    def compute_routing_index(self, query, kv_cache):
        # Compute once for all cross-decoder layers
        self.router_index = self.select_top_k_tokens(
            query, kv_cache, self.top_k
        )
        return .router_index
    
     ():
        
         layer_idx == :
            .router_index = .compute_routing_index(query, .kv_cache)
        
        
        sparse_kv = .kv_cache[.router_index]
         sparse_attention(query, sparse_kv)