more-than-quick-glance

name

more-than-quick-glance

description

Implement LASER-KV-style KV-cache compression for LLM inference pipelines using block-wise accumulative budgeting and hybrid exact-attention/LSH token selection. Use when: 'optimize KV cache for long context', 'compress KV cache without losing accuracy', 'implement LASER-KV', 'reduce LLM memory for 128k context', 'block-wise cache eviction strategy', 'fix attention-score greedy bias in cache pruning'.

LASER-KV: Block-Wise Accumulative KV-Cache Compression with Hybrid Token Selection

This skill enables Claude to implement KV-cache compression systems based on the LASER-KV framework (Layer Accumulated Selection with Exact-LSH Recall). Instead of relying solely on attention scores to decide which tokens to keep — a greedy approach that degrades 15-30% on long-context tasks — this technique partitions the cache budget into syntactic anchors, exact-attention winners, and LSH-recalled structural tokens, processed block-by-block with accumulative budgets. The result is stable performance up to 128k context with 75% cache reduction.

When to Use

• When the user needs to implement or optimize a KV-cache eviction/compression strategy for long-context LLM serving
• When deploying models at 16k-128k+ context lengths and hitting GPU memory limits from KV-cache growth
• When existing cache pruning (H2O, SnapKV, PyramidKV, FINCH) shows degradation on recall-heavy tasks like multi-hop QA
• When the user asks to reduce inference memory without retraining or changing model architecture
• When building a token selection policy that goes beyond raw attention scores
• When implementing block-wise processing pipelines for streaming or chunked LLM inference

Key Technique

The Greedy Bias Problem. Standard KV-cache compression ranks tokens by attention scores and evicts the lowest-scoring ones. This is greedy: it preserves tokens the model is currently attending to but discards "supporting tokens" that are structurally relevant but not actively queried. On benchmarks requiring recall of earlier context (Babilong QA tasks), this causes 15-30% accuracy drops at high compression ratios.

Block-Wise Accumulative Budgeting. LASER-KV divides the input context into fixed-size blocks (e.g., 4096 tokens) and processes each independently. The total budget per block is computed as B = floor(2 * r * S_block * L / (S_block + L)) where r is the compression ratio (0.25 = keep 25%), S_block is block size, and L is total context length. A protection divisor n (default: 4) splits this budget into three pools: B/n for global anchors (initial tokens), B/n for a local sliding window (recent tokens), and the remainder B - 2B/n for the long-term recall pool. This accumulative approach avoids the recursive degradation seen in fixed-summary-size methods.

Hybrid Exact-Attention + LSH Selection. Within the long-term recall pool, token selection uses a two-phase hybrid. First, attention scores are summed across all layers and heads (global consensus), and the top alpha * B_long tokens are kept (default alpha=0.75). Then, for the remaining candidates, Locality-Sensitive Hashing computes hash collision probabilities against the query, and the top (1-alpha) * B_long tokens are selected. This LSH phase recovers structurally important tokens that no single attention head strongly attends to but that share geometric similarity with the query in embedding space.

Step-by-Step Workflow

1. Compute the per-block budget. Given total context length L, block size S_block (4096), and compression ratio r (0.25), calculate: B = floor(2 * r * S_block * L / (S_block + L)). This is the total number of tokens retained per block.

2. Partition the budget with protection divisor n=4. Allocate B/n slots for global anchors (first tokens of the sequence), B/n slots for the local sliding window (most recent tokens), and B_long = B - 2*B/n slots for the long-term memory pool.

3. Divide the input into non-overlapping blocks of S_block tokens. Process blocks sequentially. For each block, expand the scoring window to include the tail of the previous block (look-back) to avoid boundary artifacts where tokens at block edges lack sufficient context.

4. Compute global attention consensus scores. For each candidate token in the current block, sum its attention scores across all layers and all attention heads: S_exact = sum over (layers, heads) of Attention(Q, C). This captures tokens important to any head, even specialized ones like induction heads.

5. Select the exact-attention winners. Take the top alpha * B_long tokens by S_exact score (default alpha=0.75). These are the highest-value tokens by direct attention measurement.

6. Apply LSH recall on the residual candidates. For tokens not selected in step 5, compute LSH collision scores: S_lsh = (1/R) * sum over hash functions of indicator[h_r(Q) == h_r(k)]. Select the top (1-alpha) * B_long tokens. This recovers structurally relevant tokens missed by greedy attention selection.

7. Merge all selected token sets. The final retained set for the block is: global anchors + local window tokens + exact-attention winners + LSH-recalled tokens. Append these to the compressed KV cache.

8. Repeat for all blocks, accumulating into the output cache. Each block's selections are additive (append-only), avoiding recursive recompression that causes compounding errors.

9. Forward the compressed KV cache to the decoding phase. The compressed cache is a drop-in replacement for the full cache during autoregressive generation. No architectural changes are needed.

Concrete Examples

Example 1: Implementing a KV-Cache Compressor for a Llama-3 Serving Pipeline

User: "I'm serving Llama-3.1-8B at 64k context and running out of GPU memory. Implement a KV-cache compression module that keeps 25% of tokens without destroying recall accuracy."

Approach:

1. Create a LaserKVCompressor class that takes the full KV cache tensors after prefill
2. Implement the budget formula: for L=65536, S_block=4096, r=0.25 => B = floor(20.254096*65536 / (4096+65536)) = floor(1524.5) = 1524 tokens per block
3. Partition: n=4 gives 381 anchor slots, 381 window slots, 762 long-term slots
4. For each of the 16 blocks: sum attention across all 32 heads and 32 layers, select top-571 by score (alpha=0.75 of 762), then LSH-select top-191 from residuals
5. Concatenate all block outputs into the compressed cache

Output:

import torch
import math

class LaserKVCompressor:
    def __init__(self, compression_ratio=0.25, block_size=4096,
                 protection_divisor=4, alpha=0.75, num_hash_rounds=32):
        self.r = compression_ratio
        self.s_block = block_size
        self.n = protection_divisor
        self.alpha = alpha
        self.R = num_hash_rounds

    def compute_budget(self, total_length: int) -> int:
        return math.floor(
            2 * self.r * self.s_block * total_length
            / (self.s_block + total_length)
        )

    def partition_budget(self, B: int):
        anchor_slots = B // self.n
        window_slots = B // self.n
        long_term_slots = B - anchor_slots - window_slots
        return anchor_slots, window_slots, long_term_slots

    def lsh_scores(self, queries, candidates, dim):
        """Compute LSH collision probability scores."""
        # Random hyperplane LSH
        planes = torch.randn(self.R, dim, device=queries.device)
        q_hashes = (queries @ planes.T) > 0        # [num_q, R]
        c_hashes = (candidates @ planes.T) >       
        
        collisions = (q_hashes.unsqueeze() == c_hashes.unsqueeze()).()
         collisions.mean(dim=(, ))  

     ():
        
        seq_len = kv_cache[].shape[]
        B = .compute_budget(seq_len)
        anchor_n, window_n, long_n = .partition_budget(B)
        num_blocks = math.ceil(seq_len / .s_block)

        selected_indices = ((anchor_n))  

         block_idx  (num_blocks):
            start = block_idx * .s_block
            end = (start + .s_block, seq_len)
            block_indices = ((start, end))

            
            attn_scores = attention_weights[:, :, :, -, start:end]
            global_scores = attn_scores.(dim=(, , ))  

            
            exact_k = (.alpha * long_n)
            _, top_exact = torch.topk(global_scores, (exact_k, (global_scores)))
            exact_selected = {block_indices[i]  i  top_exact.tolist()}

            
            residual_mask = torch.ones((block_indices), dtype=torch.)
            residual_mask[top_exact] = 
            residual_indices = [block_indices[i]  i, m  (residual_mask)  m]

             residual_indices  (long_n - exact_k) > :
                K = kv_cache[]
                residual_keys = K[, , residual_indices, :]
                query = K[, , -:, :]
                lsh_sc = .lsh_scores(query, residual_keys, K.shape[-])
                lsh_k = (long_n - exact_k, (residual_indices))
                _, top_lsh = torch.topk(lsh_sc, lsh_k)
                lsh_selected = {residual_indices[i]  i  top_lsh.tolist()}
            :
                lsh_selected = ()

            selected_indices.update(exact_selected)
            selected_indices.update(lsh_selected)

        
        window_indices = (((, seq_len - window_n), seq_len))
        selected_indices.update(window_indices)

        idx = (selected_indices)
        idx_tensor = torch.tensor(idx, device=kv_cache[].device)
        compressed_k = kv_cache[][:, :, idx_tensor, :]
        compressed_v = kv_cache[][:, :, idx_tensor, :]
         (compressed_k, compressed_v)

Example 2: Adding LSH Recall to an Existing Attention-Based Eviction Policy

User: "I already have a Top-K attention-based cache eviction. Accuracy tanks on long documents. How do I add the LSH recall component from LASER-KV?"

Approach:

1. After the existing Top-K selection, collect the residual (unchosen) candidates
2. Implement random-hyperplane LSH hashing for the residual keys against the current query
3. Score residuals by average hash collision rate across R rounds
4. Select additional (1-alpha) * budget tokens from residuals by LSH score
5. Merge with the existing Top-K set

Output:

def add_lsh_recall(query, candidate_keys, existing_topk_indices,
                   total_budget, alpha=0.75, num_rounds=32):
    """
    Augment a greedy Top-K selection with LSH recall.

    query:                [1, dim] - current query vector
    candidate_keys:       [seq_len, dim] - all candidate key vectors
    existing_topk_indices: list of int - indices already selected by attention
    total_budget:         int - total number of tokens to retain
    """
    all_indices = set(range(candidate_keys.shape[0]))
    residual_indices = sorted(all_indices - set(existing_topk_indices))

    if not residual_indices:
        return existing_topk_indices

    lsh_budget = int((1 - alpha) * total_budget)
    residual_keys = candidate_keys[residual_indices]

    # Random hyperplane LSH
    dim = query.shape[-1]
    planes = torch.randn(num_rounds, dim, device=query.device)
    q_hash = (query @ planes.T) > 0           # [1, R]
    r_hash = (residual_keys @ planes.T) > 0   # [num_residual, R]

    collision_rate = (q_hash == r_hash).float().mean(dim=1)  # [num_residual]
    k = min(lsh_budget, len(residual_indices))
    _, top_lsh = torch.topk(collision_rate, k)

    lsh_selected = [residual_indices[i] for i in top_lsh.tolist()]
    return sorted(set(existing_topk_indices) | set(lsh_selected))

Example 3: Configuring Block Size and Protection Divisor for a Specific Deployment

User: "I'm deploying at 128k context with tight memory. What parameters should I use for LASER-KV compression?"

Approach:

1. Calculate budget at different compression ratios
2. Show the effect of protection divisor on syntactic vs recall allocation
3. Recommend block size based on context length

Output:

For L=131072, S_block=4096, the budget formula gives:

  r=0.25: B = floor(2*0.25*4096*131072 / (4096+131072)) = 1993 tokens/block
  r=0.10: B = floor(2*0.10*4096*131072 / (4096+131072)) =  797 tokens/block

With protection divisor n=4 at r=0.25:
  - Global anchors:    498 tokens (initial context preserved)
  - Local window:      498 tokens (recent generation coherence)
  - Long-term recall:  997 tokens (split 75/25 exact/LSH = 747 + 250)

Recommendations for 128k:
  - Use S_block=4096 (matches paper's tested configuration)
  - Start with n=4; increase to n=6 if generation fluency is fine but recall drops
  - Keep alpha=0.75 unless your task is heavily recall-dependent (then try 0.6)
  - Total blocks: 32 at 128k — each processed independently
  - Expected memory: ~25% of full KV cache + O(seq_len * R) for LSH tables

Best Practices

• Do: Sum attention scores across all layers and heads before ranking tokens. Single-head or single-layer scores miss tokens important to specialized heads (induction heads, positional heads).
• Do: Use the look-back mechanism at block boundaries — include the tail of the previous block in the scoring window to prevent edge tokens from being unfairly evicted due to insufficient context.
• Do: Keep the append-only accumulation pattern. Never recompress already-compressed blocks; compounding errors degrade recall rapidly.
• Do: Profile the LSH overhead separately. The hash computation is O(|C| * R * d_h) — with large candidate sets and many hash rounds, this can become the bottleneck.
• Avoid: Setting alpha=1.0 (pure attention-based selection). This reverts to the greedy bias the paper identifies as the core failure mode.
• Avoid: Using very small block sizes (< 1024 tokens). Small blocks give insufficient local context for accurate attention scoring, increasing the chance of evicting important tokens.

Error Handling

• Budget exceeds block size: If B >= S_block, no compression is needed for that block — retain all tokens. This happens at low total context lengths where compression is unnecessary.
• Empty residual set: If the exact-attention phase selects all candidates (small block, large alpha), the LSH phase has nothing to select. This is fine — skip LSH gracefully.
• Degenerate attention scores: If attention is uniform across all tokens (e.g., early layers in some models), the exact phase provides little signal. Consider increasing (1-alpha) to rely more on LSH in these layers, or skip compression for uniform-attention layers.
• Hash collision degeneracy: With very low-dimensional head spaces, LSH collisions become noisy. Increase R (hash rounds) or use multi-probe LSH to compensate.

Limitations

• Prefill-phase only. LASER-KV operates during the prefill stage. It does not handle dynamic eviction during autoregressive decoding — a separate strategy is needed if the cache grows during generation.
• Compute overhead. The global attention consensus (summing across all layers/heads) and LSH scoring add latency to the prefill step. For latency-sensitive applications with short contexts (< 8k), the overhead may outweigh the memory savings.
• Task-dependent tuning. The default alpha=0.75 works well on Babilong QA tasks, but summarization or creative generation tasks may need different balances. There is no universal configuration.
• Single-model validation. The paper validates on Llama-3 variants (8B). Behavior on architectures with different attention patterns (e.g., MoE models, GQA with fewer KV heads) is unverified.
• No learned components. All selection is heuristic (attention scores + LSH). A learned selection policy could potentially outperform, but at the cost of requiring training data and fine-tuning.

Reference

Paper: More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression — Sood, Sharma, Agrawal (2026). Look for: Algorithm 1 (Exact-LSH Selection Policy), the accumulative budget formula in Section 3, and the Babilong ablation tables showing the 15-30% degradation of attention-only methods.