rlkv-cache-compression

name	rlkv-cache-compression
title	Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.08525
keywords	["kv-cache","compression","attention-heads","reinforcement-learning","reasoning-efficiency"]
description	Use reinforcement learning to identify reasoning-critical attention heads and apply selective KV cache compression, reducing cache by 20-50% while preserving reasoning quality. Maintains speedups up to 1.21x with minimal performance loss.

RL-Guided KV Cache Compression: Selective Head Importance

Transformer inference scales poorly with sequence length due to KV cache growth. Naive compression removes important information; reasoning-critical attention heads degrade when compressed uniformly. This technique uses reinforcement learning to discover which heads matter for reasoning and compresses selectively.

The key insight: attention heads have different roles. Some heads are critical for coherent reasoning chains, while others handle retrieval or aggregation and tolerate aggressive compression. By identifying these patterns through RL, you preserve reasoning quality while gaining 20-50% cache reduction.

Core Concept

Head-Importance Discovery: Use RL to directly optimize cache allocation against actual reasoning outcomes. Rather than heuristic importance measures, train an agent that observes head activations and decides cache allocation while generation quality is the reward signal.

Asymmetric Allocation: Allocate full KV cache to reasoning-critical heads while aggressively compressing others. This prevents the quality degradation that comes from uniform compression.

Architecture Overview

• Head Analyzer: Computes importance scores for each attention head using RL-derived signals
• Selective Compression Policy: Maintains full cache for critical heads, applies compression to others
• Reward Mechanism: Direct optimization against generation quality, not proxy metrics
• Generation Monitor: Tracks completion quality to guide policy updates

Implementation Steps

Stage 1: Establish Head Importance Baseline

Profile which heads contribute to reasoning by analyzing attention patterns during generation:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

def analyze_head_activations(model, batch):
    """
    Profile attention head activation patterns during generation.
    Identifies which heads show sustained activity during reasoning.
    """
    head_importance = {}

    # Hook into attention layers
    activations = []
    def capture_attention(module, input, output):
        activations.append(output[])  

    
     layer_idx, layer  (model.transformer.h):
        layer.self_attention.register_forward_hook(capture_attention)

    
     torch.no_grad():
        outputs = model.generate(
            batch,
            max_new_tokens=,
            output_attentions=
        )

     activations


activations = analyze_head_activations(model, reasoning_prompts)