kv-cache-compression

name	kv-cache-compression
title	Inference-Time Hyper-Scaling with KV Cache Compression
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.05345
keywords	["inference-optimization","memory-efficiency","kv-cache","scaling","reasoning"]
description	Enables 8x KV cache compression with minimal training overhead to improve reasoning accuracy by allowing more token generation within computational budgets.

Inference-Time Hyper-Scaling with KV Cache Compression

Core Concept

In transformer inference, the KV cache—storing key and value vectors for all previous tokens—becomes the primary memory bottleneck, often limiting reasoning performance more than computational capacity. Rather than generating tokens until memory exhaustion, KV cache compression allows strategic memory reduction to generate additional tokens within the same budget. Dynamic Memory Sparsification (DMS) achieves 8× compression with only 1,000 training steps, enabling inference-time hyper-scaling where extra tokens directly translate to improved reasoning accuracy.

Architecture Overview

• Dynamic Memory Sparsification (DMS): Compression technique that delays eviction of cached tokens, implicitly merging representations
• Training Efficiency: Requires minimal training overhead (1,000 steps) making adaptation practical
• Inference-Time Strategy: Uses freed memory budget to generate additional reasoning tokens
• Memory-Bandwidth Focus: Optimizes for actual inference bottleneck (memory, not computation)
• Multi-Model Compatibility: Applies across various LLM families without architectural changes
• Reasoning Task Optimization: Particularly effective for complex reasoning requiring extended token generation

Implementation

The following code demonstrates the DMS algorithm and hyper-scaling strategy:

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple, List, Optional

class DynamicMemorySparsification:
    """
    Dynamic Memory Sparsification for KV cache compression.
    """
    def __init__(self, compression_ratio: float = 0.125, merge_threshold: float = 0.5):
        self.compression_ratio = compression_ratio
        self.merge_threshold = merge_threshold

    def compute_token_importance(self, key_vectors: torch.Tensor,
                                query_vectors: torch.Tensor) -> torch.Tensor:
        """
        Compute importance scores for cached tokens based on attention patterns.

        key_vectors: (seq_len, hidden_dim) cached KV vectors
        query_vectors: (num_heads, hidden_dim) current query
        Returns: (seq_len,) importance scores in [0, 1]
        """
        # Normalize vectors for cosine similarity
        keys_norm = F.normalize(key_vectors, p=2, dim=1)
        query_norm = F.normalize(query_vectors, p=2, dim=1)

        # Compute attention similarity as importance
        importance = torch.matmul(keys_norm, query_norm.t())  # (seq_len, num_heads)
        importance = importance.max(dim=1).values  # (seq_len,)

        # Normalize to [0, 1]
        importance = (importance - importance.min()) / (importance.max() - importance.() + )

         importance

     () -> [torch.Tensor, torch.Tensor]:
        
        seq_len = key_cache.shape[]
        target_len = (, (seq_len * .compression_ratio))

        
        _, keep_indices = torch.topk(importance_scores, target_len)
        keep_indices = torch.sort(keep_indices)[]  

        
        compress_to = torch.zeros(seq_len, dtype=torch.long, device=key_cache.device)

        
         i  (seq_len):
             i  keep_indices:
                compress_to[i] = i
            :
                
                nearest = torch.argmin(torch.(keep_indices - i))
                compress_to[i] = keep_indices[nearest].item()

        
        key_merged = torch.zeros(target_len, key_cache.shape[],
                                device=key_cache.device, dtype=key_cache.dtype)
        value_merged = torch.zeros(target_len, value_cache.shape[],
                                  device=value_cache.device, dtype=value_cache.dtype)
        counts = torch.zeros(target_len, device=key_cache.device)

         orig_idx, merged_idx  (compress_to):
            key_merged[merged_idx] += key_cache[orig_idx]
            value_merged[merged_idx] += value_cache[orig_idx]
            counts[merged_idx] += 

        
        key_merged /= counts.unsqueeze().clamp(=)
        value_merged /= counts.unsqueeze().clamp(=)

         key_merged, value_merged


 (nn.Module):
    
     ():
        ().__init__()
        .model_dim = model_dim
        .num_heads = num_heads
        .head_dim = model_dim // num_heads

        
        .wq = nn.Linear(model_dim, model_dim)
        .wk = nn.Linear(model_dim, model_dim)
        .wv = nn.Linear(model_dim, model_dim)
        .wo = nn.Linear(model_dim, model_dim)

        .dms = DynamicMemorySparsification(compression_ratio=compression_ratio)

     () -> [torch.Tensor, torch.Tensor, torch.Tensor]:
        
        
        Q = .wq(query)  
        Q = Q.view(Q.shape[], Q.shape[], .num_heads, .head_dim)
        Q = Q.transpose(, )  

        
        K = .wk(key_cache)  
        V = .wv(value_cache)

        K = K.view(-, .num_heads, .head_dim).transpose(, )  
        V = V.view(-, .num_heads, .head_dim).transpose(, )

        
        scores = torch.matmul(Q, K.transpose(, )) / (.head_dim ** )
        attn = F.softmax(scores, dim=-)

        output = torch.matmul(attn, V)  
        output = output.transpose(, ).contiguous()
        output = output.view(output.shape[], output.shape[], .model_dim)
        output = .wo(output)

        
         compress  key_cache.shape[] > :
            importance = .dms.compute_token_importance(key_cache,
                                                           .wq(query).squeeze())
            key_cache, value_cache = .dms.merge_kv_tokens(key_cache, value_cache, importance)

         output, key_cache, value_cache


 :
    
     ():
        .model = model
        .total_budget_gb = total_budget_gb
        .bytes_per_token =  *  *   

     () -> :
        
        available_bytes = .total_budget_gb * ( ** )
        tokens_uncompressed = available_bytes / .bytes_per_token

        
        tokens_compressed = tokens_uncompressed / compression_ratio

         (tokens_compressed)

     () -> []:
        
        tokens = prompt_ids.copy()

        
        max_new_tokens = initial_max_tokens
         compression_enabled:
            available = .estimate_tokens_available()
            max_new_tokens = (available, initial_max_tokens * )  

        
         _  (max_new_tokens):
            
            
            next_token =   
            tokens.append(next_token)

         tokens

Practical Guidance

Training Schedule: DMS requires only 1,000 training steps, making it practical to apply to frozen pre-trained models. Use a learning rate of 1e-4 to 1e-5 for stability.

Compression Ratio Selection: Start with 0.125 (8× compression). For accuracy-critical tasks, use 0.25 (4× compression); for speed-critical tasks, try 0.06 (16× compression).

Token Importance Scoring: Use the provided attention-based scoring, but consider task-specific variants: for code, prioritize recent tokens; for reasoning, prioritize intermediate results.

Merged Token Representation: When averaging K and V vectors during merging, use weighted averaging based on importance scores rather than uniform averaging for better fidelity.

Memory Budget Planning: Calculate total GPU VRAM available, subtract model weights and activations, then allocate remaining budget to KV cache. This determines max-achievable sequence length.

Benchmark on Target Tasks: DMS shows 12+ point improvements on AIME and GPQA. Benchmark on your specific reasoning tasks to validate improvements.

Reference

DMS achieves strong empirical results on reasoning benchmarks:

• Qwen-R1 32B on AIME 24: +12.0 points improvement
• GPQA: +8.6 points
• LiveCodeBench: +9.7 points
• Training Efficiency: 1,000 steps on single GPU

The method is particularly valuable for reasoning-heavy applications where extended generation directly correlates with accuracy. By converting memory freed through compression into additional reasoning tokens, hyper-scaling enables better problem-solving within fixed computational budgets.