fasa-frequency-aware-sparse-attention

name	fasa-frequency-aware-sparse-attention
title	FASA: Frequency-Aware Sparse Attention
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.03152
keywords	["Sparse Attention","KV Cache Compression","Rotary Embeddings","Training-Free","Frequency Analysis"]
description	Identify dominant frequency components in RoPE embeddings to determine which attention dimensions are essential, reducing KV cache by 81% while maintaining performance. Training-free approach integrates with existing compression methods for 2.56x speedup on long-context reasoning.

FASA: Functional Sparsity in Attention via Frequency Analysis

Rotary Position Embeddings (RoPE) use multiple frequency components to encode position, but not all frequencies are equally important for each task. FASA discovers which frequency dimensions actually contribute to model performance, enabling aggressive KV cache reduction by computing attention only on critical frequencies. This training-free approach complements other compression methods and requires only one-time offline calibration.

The key insight is that high-frequency components (capturing fine-grained positional distinctions) often matter less than low-frequency components (capturing coarse relationships) for understanding context. By identifying task-dependent "dominant frequency chunks," FASA achieves near-oracle compression without task-specific training.

Core Concept

FASA operates in two stages:

1. Token Importance Prediction (TIP): Use pre-identified dominant frequency chunks to efficiently score token importance without full attention computation.

2. Focused Attention Computation (FAC): Perform full-precision attention only on selected critical tokens and dimensions, dramatically reducing KV cache memory requirements.

Unlike uniform sparsity, this approach exploits the structure of rotary embeddings to identify task-relevant dimensions.

Architecture Overview

• Frequency Analyzer: Identifies dominant frequency components via one-time calibration
• Dimension Selector: Maps tasks to critical frequency chunks
• Token Scorer: Efficient importance prediction using only dominant frequencies
• Attention Engine: Computes attention on selected dimensions/tokens
• Cache Manager: Stores only critical KV pairs in reduced format
• Integration Layer: Works with other compression methods (quantization, eviction)

Implementation

Step 1: Analyze RoPE Frequency Components

Create tools to identify which frequency dimensions matter for a given model/task.

# Frequency analysis for RoPE
import numpy as np
from typing import List, Tuple

class RoPEFrequencyAnalyzer:
     ():
        
        .hidden_dim = hidden_dim
        .rope_theta = rope_theta

        
        .frequencies = ._compute_frequencies()

     () -> np.ndarray:
        
        
        inv_freq =  / (.rope_theta ** (
            np.arange(, .hidden_dim, ) / .hidden_dim
        ))
         inv_freq

     () -> [[], []]:
        
        
        
        freq_activations = []

         freq_idx  (.hidden_dim // ):
            
            dim1 =  * freq_idx
            dim2 =  * freq_idx + 

            pair_act = activations[:, :, dim1:dim2+]
            
            importance = np.var(pair_act)
            freq_activations.append((freq_idx, importance))

        
        sorted_freqs = (freq_activations, key= x: x[], reverse=)

        
        indices = [f[]  f  sorted_freqs[:top_k]]
        scores = [f[]  f  sorted_freqs[:top_k]]

         indices, scores

 () -> [[]]:
    
    analyzer = RoPEFrequencyAnalyzer(model.config.hidden_size)

    
    all_activations = []
     batch  calibration_dataset:
         torch.no_grad():
            hidden = model.encoder(batch[])
        all_activations.append(hidden.cpu().numpy())

    activations = np.concatenate(all_activations, axis=)

    
    ranked_indices, scores = analyzer.rank_frequencies_by_importance(activations)

    
    target_freq_count = (
        (model.config.hidden_size // ) * compression_ratio
    )

    
    chunk_size = (, target_freq_count // num_chunks)
    chunks = []

     chunk_id  (num_chunks):
        start_idx = chunk_id * chunk_size
        end_idx = (start_idx + chunk_size, (ranked_indices))
        chunk_freqs = ranked_indices[start_idx:end_idx]
        chunks.append(chunk_freqs)

     chunks