ultramemv2-memory-networks

name	ultramemv2-memory-networks
title	UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.18756
keywords	["memory-networks","sparse-models","long-context","parameter-scaling","activation-density"]
description	Scale memory networks to 120B parameters with improved long-context learning through integrated memory layers, simplified value projection, and optimized parameter ratios for superior memory-intensive tasks.

UltraMemV2: Memory Networks Scaling to 120B Parameters

Core Concept

UltraMemV2 advances memory-augmented transformer architectures to extreme scale while maintaining efficiency. The key insight is that activation density (which parameters actually compute) matters more than total parameter count. Through integrated memory layers in every transformer block, simplified value expansion, FFN-based processing, and principled initialization, UltraMemV2 achieves performance parity with 8-expert MoE while using significantly less memory. The approach enables 120B total parameters with only 2.5B activated, delivering +1.6 points on memorization and +7.9 points on in-context learning.

Architecture Overview

• Integrated Memory Layers: Memory in every transformer block
• Simplified Value Expansion: Single linear projections
• FFN-Based Value Processing: Efficient computation
• Principled Initialization: Stability at scale
• Sparse Activation: 2.5B active from 120B total

Implementation Steps

1. Design Memory Layer Architecture

Create efficient memory modules for each transformer block:

import torch
import torch.nn as nn
from typing import Tuple, Optional

class MemoryLayer(nn.Module):
    """Memory module integrated into transformer block."""

    def __init__(
        self,
        hidden_size: int = 2048,
        memory_dim: int = 512,
        num_memory_slots: int = 64,
        ffn_expansion: int = 4
    ):
        super().__init__()
        self.hidden_size = hidden_size
        self.memory_dim = memory_dim
        self.num_memory_slots = num_memory_slots

        
        .memory_slots = nn.Parameter(
            torch.randn(, num_memory_slots, memory_dim) * 
        )

        
        .query_proj = nn.Linear(hidden_size, memory_dim)

        
        .key_proj = nn.Linear(hidden_size, memory_dim)
        .value_proj = nn.Linear(hidden_size, memory_dim)

        
        .value_expand = nn.Linear(memory_dim, hidden_size)

        
        ffn_hidden = hidden_size * ffn_expansion
        .ffn = nn.Sequential(
            nn.Linear(hidden_size, ffn_hidden),
            nn.GELU(),
            nn.Linear(ffn_hidden, hidden_size)
        )

        
        .output_gate = nn.Linear(hidden_size * , hidden_size)

     () -> [torch.Tensor, torch.Tensor]:
        
        batch_size, seq_len, hidden_size = x.shape

        
        x_pooled = x.mean(dim=)  

        
        query = .query_proj(x_pooled)  
        key = .key_proj(x_pooled)      

        
        
        mem_keys = .memory_slots.expand(batch_size, -, -)  
        scores = torch.matmul(query.unsqueeze(), mem_keys.transpose(, ))  
        scores = scores / (.memory_dim ** )
        weights = torch.softmax(scores, dim=-)  

        
        memory_read = torch.matmul(weights, mem_keys).squeeze()  

        
        memory_value = .value_expand(memory_read)  

        
        memory_processed = .ffn(memory_value)

        
        
        write_gate = torch.sigmoid(.key_proj(x_pooled))
        .memory_slots.data = ( - ) * .memory_slots.data +  * write_gate.unsqueeze() * key.unsqueeze()

        
        combined = torch.cat([x_pooled, memory_processed], dim=-)
        output = .output_gate(combined)

        
        output = output + x_pooled

        
        output = output.unsqueeze().expand(-, seq_len, -)

         output, memory_state