memory-llm-ffn-decoupling

name	memory-llm-ffn-decoupling
title	MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.00398
keywords	["FFN Architecture","Memory Mechanism","Interpretability","Static Lookup","Transformer Components"]
description	Decouple feed-forward networks from self-attention by training FFNs on context-free token embeddings instead of residual streams. Enables pre-computation of FFN outputs as static lookup tables for inference efficiency and improved interpretability.

MemoryLLM: Decoupled Feed-Forward Memory

Standard Transformers intertwine attention outputs with FFN inputs through the residual stream, making each token's FFN output context-dependent. MemoryLLM decouples this by training FFNs independently on token-indexed embeddings rather than context-aware representations. This transforms FFNs into interpretable lookup tables that can be pre-computed and stored on-device, enabling faster inference and clearer understanding of what FFNs learn.

The key insight is that FFN parameters can be viewed as a retrieval table where tokens are keys and hidden states are values. By separating FFN training from attention, we get both efficiency and interpretability.

Core Concept

MemoryLLM operates on architectural decoupling:

1. Standard Transformer: Attention → FFN, where FFN sees attention-modified residuals
2. MemoryLLM: Attention and FFN work independently; FFN receives only token embeddings (context-free)

This enables:

• Static Lookup: FFN outputs pre-computed and cached
• Interpretability: FFN behavior per-token is visible
• Flexibility: FFN outputs can be selectively disabled or modified

Architecture Overview

• Independent FFN Training: FFNs trained only on token embeddings, not residual streams
• Token-Key-Value Framework: Think of FFN as (up_proj: key, down_proj: value)
• Parallel Processing: Attention and FFN execute independently
• Static Lookup Layer: FFN outputs cached as token-indexed lookup tables
• Flexible Integration: Optionally use context-aware variant for better performance

Implementation

Step 1: Design Context-Free FFN

Create FFN that operates only on token identity, not context.

# Context-free FFN implementation
class MemoryFFN(nn.Module):
    def __init__(self, hidden_dim: int, intermediate_dim: int,
                 vocab_size: int):
        """
        FFN operating on token embeddings (context-free).

        Args:
            hidden_dim: Hidden dimension
            intermediate_dim: Intermediate expansion dimension
            vocab_size: Vocabulary size for static lookup
        """
        ().__init__()

        
        .up_proj = nn.Linear(hidden_dim, intermediate_dim, bias=)

        
        .activation = nn.GELU()

        
        .down_proj = nn.Linear(intermediate_dim, hidden_dim, bias=)

        
        .static_lookup = 
        .vocab_size = vocab_size

     () -> torch.Tensor:
        
        x = .up_proj(token_embeddings)
        x = .activation(x)
        x = .down_proj(x)
         x

     ():
        
        token_ids = torch.arange(.vocab_size)
        token_embeddings = embedding_layer(token_ids)

         torch.no_grad():
            ffn_outputs = .forward(token_embeddings)

        
        .static_lookup = ffn_outputs.detach()
        .register_buffer(, .static_lookup)

     () -> torch.Tensor:
        
         .static_lookup  :
             

         .static_lookup[token_ids]