rectified-sparse-attention

name	rectified-sparse-attention
title	Rectified Sparse Attention: Efficient Long-Sequence Generation with Error Correction
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.04108
keywords	["sparse-attention","efficient-inference","long-sequences","error-rectification"]
description	Enable efficient long-sequence generation by combining block-sparse attention with periodic dense rectification to bound error accumulation.

Rectified Sparse Attention

Core Concept

ReSA solves a fundamental problem in sparse decoding for long-sequence generation: approximation errors accumulate and degrade generation quality as sequence length increases. By periodically "rectifying" the KV cache using dense forward passes, ReSA maintains near-lossless generation quality while achieving up to 2.42× speedup under long-context decoding (256K tokens).

Architecture Overview

• Group Block Sparse Attention: Query-dependent sparsity restricting computation to dynamically selected context blocks
• Dense Rectification Phase: Periodically re-encode recently generated tokens densely to refresh KV cache and bound error accumulation
• Block Descriptors: Min/max vectors enable efficient retrieval without exhaustive token scanning
• Continuous Batching Integration: Naturally compatible with existing LLM serving optimizations
• Quality Preservation: Near-lossless performance on math reasoning, language modeling, and retrieval tasks

Implementation

Step 1: Implement Group Block Sparse Attention

import torch
import torch.nn as nn
from typing import Tuple, Optional

class BlockDescriptor:
    """Compact representation of attention blocks"""

    def __init__(self, block_size: int):
        self.block_size = block_size

    def compute_descriptor(self, tokens: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Compute min/max vectors for a block of tokens.
        These descriptors enable efficient block selection without scanning all tokens.

        Args:
            tokens: [seq_len, hidden_dim] - Token representations

        Returns:
            min_vec: [num_blocks, hidden_dim] - Minimum values per block
            max_vec: [num_blocks, hidden_dim] - Maximum values per block
        """

        seq_len = tokens.shape[0]
        num_blocks = (seq_len + self.block_size - 1) // self.block_size

        min_vec = torch.full((num_blocks, tokens.shape[1]), float('inf'))
        max_vec = torch.full((num_blocks, tokens.shape[1]), float('-inf'))

        for block_idx in range(num_blocks):
            start = block_idx * self.block_size
            end = min(start + self.block_size, seq_len)

            block_tokens = tokens[start:end, :]
            min_vec[block_idx] = torch.min(block_tokens, dim=0)[0]
            max_vec[block_idx] = torch.(block_tokens, dim=)[]

         min_vec, max_vec

 (nn.Module):
    

     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .num_heads = num_heads
        .block_size = block_size
        .top_k_blocks = top_k_blocks
        .head_dim = hidden_dim // num_heads

        .query_proj = nn.Linear(hidden_dim, hidden_dim)
        .key_proj = nn.Linear(hidden_dim, hidden_dim)
        .value_proj = nn.Linear(hidden_dim, hidden_dim)
        .output_proj = nn.Linear(hidden_dim, hidden_dim)

        .block_descriptor = BlockDescriptor(block_size)

     () -> torch.Tensor:
        

        min_vec, max_vec = block_descriptors
        num_blocks = min_vec.shape[]

        
        
        query_norm = query / (torch.norm(query, p=, dim=-, keepdim=) + )

        block_scores = []
         block_idx  (num_blocks):
            
            min_overlap = torch.(query_norm, min_vec[block_idx])
            max_overlap = torch.(query_norm, max_vec[block_idx])

            overlap = torch.clamp(torch.(max_overlap - min_overlap), =)
            block_scores.append(overlap)

        block_scores = torch.stack(block_scores)

        
        num_active = (, (num_blocks * active_ratio))
        top_k = (.top_k_blocks, num_blocks)

        _, selected_indices = torch.topk(block_scores, k=top_k, largest=)

         selected_indices

     () -> [torch.Tensor, ]:
        

        batch_size, seq_len, _ = x.shape

        
        query = .query_proj(x).view(batch_size, seq_len, .num_heads, .head_dim)
        key = .key_proj(x).view(batch_size, seq_len, .num_heads, .head_dim)
        value = .value_proj(x).view(batch_size, seq_len, .num_heads, .head_dim)

        
         kv_block_descriptors   :
            selected_blocks = .select_relevant_blocks(
                query[:, -, :, :].mean(dim=),  
                kv_block_descriptors,
                active_ratio=active_ratio
            )
        :
            
            selected_blocks = torch.arange(seq_len // .block_size + )

        
        mask = .construct_sparse_mask(seq_len, selected_blocks)

        
        query = query.transpose(, )  
        key = key.transpose(, )
        value = value.transpose(, )

        scores = torch.matmul(query, key.transpose(-, -)) / (.head_dim ** )
        scores = scores.masked_fill(~mask, ())

        attn_weights = torch.softmax(scores, dim=-)
        attn_weights = attn_weights.masked_fill(~mask, )

        output = torch.matmul(attn_weights, value)

        
        output = output.transpose(, ).contiguous()
        output = output.view(batch_size, seq_len, .hidden_dim)
        output = .output_proj(output)

        
        new_kv = (key, value)  kv_cache    (kv_cache[], kv_cache[])
        new_descriptors = kv_block_descriptors  

         output, (new_kv, new_descriptors)

     () -> torch.Tensor:
        

        mask = torch.zeros((seq_len, seq_len), dtype=torch.)

         block_idx  selected_blocks:
            start = block_idx * .block_size
            end = (start + .block_size, seq_len)

            
            mask[:, start:end] = 

        
        mask[:, -] = 

         mask

Step 2: Implement Dense Rectification

class DenseRectificationModule(nn.Module):
    """Periodically refresh KV cache with dense forward passes"""

    def __init__(self, model, rectify_interval: int = 16):
        super().__init__()
        self.model = model
        self.rectify_interval = rectify_interval

    def rectify_kv_cache(self, recent_tokens: torch.Tensor,
                        full_kv_cache: Tuple) -> Tuple:
        """
        Use dense attention to re-encode recent tokens and refresh KV cache.
        This bounds error accumulation to constant windows.

        Args:
            recent_tokens: [batch, recent_length, hidden_dim]
            full_kv_cache: Current KV cache (may contain errors)

        Returns:
            fresh_kv_cache: Re-computed KV cache for recent tokens
        """

        batch_size, recent_len, hidden_dim = recent_tokens.shape

        # Run dense forward pass on recent tokens only
        # This is more efficient than re-encoding entire history
        fresh_key = self.model.key_proj(recent_tokens)
        fresh_value = self.model.value_proj(recent_tokens)

        # Replace latest entries in KV cache with fresh computations
        old_kv = full_kv_cache

        # Keep old KV for context window, replace recent with fresh
        context_window = self.rectify_interval

        fresh_kv = (
            torch.cat([
                old_kv[0][:, :-context_window, :, :],  # Keep old
                fresh_key.unsqueeze(1)  # Replace with fresh
            ], dim=1),
            torch.cat([
                old_kv[1][:, :-context_window, :, :],  # Keep old
                fresh_value.unsqueeze()  
            ], dim=)
        )

        
        descriptor_generator = BlockDescriptor(block_size=)
        min_vec, max_vec = descriptor_generator.compute_descriptor(fresh_key.squeeze())

         fresh_kv, (min_vec, max_vec)

 :
    

     ():
        .model = model
        .sparse_attn = GroupBlockSparseAttention(
            hidden_dim=model.hidden_dim,
            num_heads=model.num_heads,
            block_size=
        )
        .rectifier = DenseRectificationModule(model, rectify_interval)
        .rectify_interval = rectify_interval

     () -> torch.Tensor:
        

        generated = prompt.clone()
        kv_cache = 
        kv_descriptors = 
        recent_tokens = []

         step  (max_length):
            
            next_logits = .model.forward(
                generated[:, -:, :],  
                kv_cache=kv_cache,
                sparse_attn=.sparse_attn,
                kv_descriptors=kv_descriptors,
                active_ratio=
            )

            
            next_token = torch.argmax(next_logits, dim=-)
            generated = torch.cat([generated, next_token.unsqueeze(-)], dim=)
            recent_tokens.append(next_token)

            
             (step + ) % .rectify_interval == :
                ()

                recent_tensor = torch.stack(recent_tokens)
                kv_cache, kv_descriptors = .rectifier.rectify_kv_cache(
                    recent_tensor, kv_cache
                )

                recent_tokens = []  

         generated

Step 3: Integration with Continuous Batching

class LongSequenceInferenceEngine:
    """Production-ready inference with sparse attention and batching"""

    def __init__(self, model, batch_size: int = 32):
        self.model = model
        self.batch_size = batch_size
        self.pending_requests = []

    def add_request(self, prompt_ids: torch.Tensor, max_length: int):
        """Add generation request to queue"""

        self.pending_requests.append({
            'prompt_ids': prompt_ids,
            'max_length': max_length,
            'generated_ids': prompt_ids.clone(),
            'step': 0,
            'sparse_decoder': SparseDecodingWithRectification(self.model),
        })

    def serve_batch(self):
        """Process batch of requests with continuous batching"""

        active_requests = [r for r in self.pending_requests
                          if r['step'] < r['max_length']]

        # Group into batches respecting max_length constraints
        for batch_idx in range(0, len(active_requests), self.batch_size):
            batch = active_requests[batch_idx:batch_idx + self.batch_size]

            
            batch_input = torch.cat([r[][:, -:]  r  batch], dim=)

            
            batch_logits = .model.forward(
                batch_input,
                use_sparse_attention=,
                sparse_active_ratio=
            )

            
            batch_next_tokens = torch.argmax(batch_logits, dim=-)

             req_idx, request  (batch):
                next_token = batch_next_tokens[req_idx:req_idx+]
                request[] = torch.cat(
                    [request[], next_token], dim=
                )
                request[] += 

     () -> :
        

        completed = [r  r  .pending_requests
                     r[] >= r[]]

        .pending_requests = [r  r  .pending_requests
                                 r[] < r[]]

         completed

Practical Guidance

1. Block Size Tuning: Start with block_size=64. Smaller blocks (32) increase sparsity but may miss context; larger blocks (128) reduce speedup but improve quality.

2. Active Ratio Selection: active_ratio=0.25 (attending to 25% of blocks) provides good quality-speedup tradeoff. Increase to 0.5 for critical tasks, decrease to 0.1 for pure language modeling.

3. Rectification Interval: Rectify every 16-32 tokens. More frequent rectification preserves quality but reduces speedup. Less frequent rectification (64+) risks error accumulation.

4. Error Accumulation Bounds: Dense rectification bounds error to the window size (rectify_interval × block_size tokens). This is vastly better than error growth with sequence length.

5. Memory Savings: Block descriptors reduce memory access factor to (1/b + p + 1/f) where b=block_size, p=rectification frequency, f=forward pass factor. With defaults: (1/64 + 1/16 + 1/4) ≈ 0.33× dense attention memory.

6. Integration Points: Works seamlessly with continuous batching, multi-GPU inference, and existing KV cache optimizations.

Reference

• Paper: Rectified Sparse Attention (2506.04108)
• Key Innovation: Periodic dense rectification bounding error accumulation
• Speedup: Up to 2.42× on 256K tokens with near-lossless quality
• Architecture: Block sparse + dense rectification phases alternating