gradmem-context-memory-compression

name	gradmem-context-memory-compression
title	GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.13875
keywords	["Memory Compression","Context Storage","Gradient Descent","Long-Context","Test-Time Optimization"]
description	Compress long context into compact memory tokens via iterative gradient descent. Learn to write information into prefix memory without storing full KV-caches, enabling efficient long-context reasoning and retrieval.

GradMem: Test-Time Gradient Descent for Context Memory Compression

Large language models struggle with long contexts due to massive KV-cache overhead. GradMem enables models to compress context into compact memory tokens via test-time gradient descent. Rather than storing full cache for each query, the model learns to write information into a small set of learnable prefix tokens by optimizing a self-supervised reconstruction loss. This approach is particularly effective for context removal scenarios where the model must answer questions without accessing the original long context at inference time.

The key innovation is using gradient updates on memory tokens (with frozen model weights) rather than forward-only writes, enabling iterative error correction and much better context compression.

Core Concept

GradMem operates through three phases:

1. Context Ingestion — Process long context once to initialize memory tokens
2. Gradient-Based Writes — Iteratively update memory tokens using gradient descent on reconstruction loss
3. Query Answering — Use compressed memory (without original context) to answer questions

This contrasts with forward-only approaches which write information in a single forward pass. By optimizing iteratively, GradMem can pack more information into the same number of tokens.

Architecture Overview

• Memory Token Initializer — Create learnable prefix tokens initialized from context
• Reconstruction Loss — Self-supervised objective: can we reconstruct context from memory?
• Gradient-Based Writer — Optimize memory tokens via SGD on reconstruction loss
• Query Encoder — Generate question embeddings in the same space as memory
• Memory-Only Reader — Answer questions using only memory tokens (no original context)
• Scaling Mechanism — Multiple gradient steps improve capacity linearly

Implementation Steps

Start by designing the memory tokens and initializing them from context.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.optim import SGD

class ContextMemoryCompressor:
    """Compress long context into learnable memory tokens."""

     ():
        .hidden_dim = hidden_dim
        .num_memory_tokens = num_memory_tokens
        .context_seq_len = context_seq_len

        
        .memory_tokens = nn.Parameter(
            torch.randn(num_memory_tokens, hidden_dim) * 
        )

        
        .context_encoder = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=,
                                      dim_feedforward=,
                                      batch_first=),
            num_layers=
        )

        
        .context_decoder = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d_model=hidden_dim, nhead=,
                                      dim_feedforward=,
                                      batch_first=),
            num_layers=
        )

     ():
        
        
        batch_size = context_embeddings.size()

        
        memory = .memory_tokens.unsqueeze().expand(batch_size, -, -)

        
        cross_attn = nn.MultiheadAttention(.hidden_dim, num_heads=,
                                          batch_first=)
        memory_initialized, _ = cross_attn(memory, context_embeddings,
                                          context_embeddings)

         memory_initialized

     () -> torch.Tensor:
        
        
        reconstructed = .context_decoder(memory, context)

        
        loss = F.mse_loss(reconstructed, context)
         loss

     () -> torch.Tensor:
        
        batch_size = context_embeddings.size()

        
        memory = .initialize_memory(context_embeddings)

        
        memory_param = torch.nn.Parameter(memory.clone())
        optimizer = SGD([memory_param], lr=learning_rate)

        
         step  (num_gradient_steps):
            optimizer.zero_grad()

            
            loss = .reconstruction_loss(memory_param, context_embeddings)

            
            loss.backward()

            
            optimizer.step()

            ()

         memory_param.detach()

     () -> torch.Tensor:
        
        
        memory = context_embeddings.mean(dim=, keepdim=).expand(
            -, .num_memory_tokens, -
        )
         memory