diagonal-batching-recurrent-transformers

name	diagonal-batching-recurrent-transformers
title	Diagonal Batching Unlocks Parallelism in Recurrent Memory Transformers for Long Contexts
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.05229
keywords	["transformers","parallelism","long-context","inference-optimization","recurrent-models"]
description	Unlocks parallelism in recurrent memory transformers through diagonal batching of the layers-segments grid, achieving 3.3x speedup on 131K-token sequences without model retraining.

Diagonal Batching for Recurrent Memory Transformers

Core Concept

Recurrent Memory Transformers (RMTs) achieve linear time and constant memory complexity theoretically, but their sequential execution on GPUs undermines practical performance. The core bottleneck isn't algorithmic complexity—it's scheduling constraints that prevent GPU parallelism. Diagonal batching reorganizes the computation grid (layers × segments) into independent diagonals that can execute concurrently, unlocking substantial speedups without requiring model retraining or architectural changes.

Architecture Overview

• Problem Identification: RMT sequential execution creates GPU idle time despite constant memory requirements
• Diagonal Reorganization: Transforms (layer, segment) dependency graph into parallel-executable diagonals
• Dependency Preservation: Maintains exact layer-level recurrence through grouped weight organization
• GroupedGEMM Operations: Batches matrix multiplications across layers within diagonals
• GPU Kernel Optimization: Single grouped kernel launch per diagonal layer replaces many individual launches
• Zero Model Modification: Pure runtime optimization compatible with frozen pre-trained weights

Implementation

The following code demonstrates the diagonal batching algorithm:

import torch
import torch.nn as nn
from typing import List, Tuple

class DiagonalBatchingScheduler:
    """
    Reorganizes RMT computation from sequential to parallel-executable diagonals.
    """
    def __init__(self, num_layers: int, num_segments: int):
        self.num_layers = num_layers
        self.num_segments = num_segments

    def compute_diagonals(self) -> List[List[[, ]]]:
        
        diagonals = []

        
         diagonal_idx  (.num_layers + .num_segments - ):
            diagonal = []

            
             layer_idx  (.num_layers):
                segment_idx = diagonal_idx - layer_idx

                
                  <= segment_idx < .num_segments:
                    diagonal.append((layer_idx, segment_idx))

             diagonal:
                diagonals.append(diagonal)

         diagonals

     () -> torch.Tensor:
        
        stacked_weights = []

         layer  model_layers:
            
            w = layer.weight.data
            stacked_weights.append(w)

        
        grouped = torch.stack(stacked_weights, dim=)
         grouped

     () -> torch.Tensor:
        
        selected_weights = grouped_weights[batch_indices]

        
        B = inputs.shape[] // (batch_indices)
        D = inputs.shape[-]

        
        
        inputs_reshaped = inputs.reshape((batch_indices), B, D)
        outputs = torch.einsum(, inputs_reshaped, selected_weights)

         outputs.reshape(-, D)


 (nn.Module):
    
     ():
        ().__init__()
        .num_layers = num_layers
        .segment_length = segment_length
        .hidden_dim = hidden_dim

        
        .layers = nn.ModuleList([
            nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=, dim_feedforward=*hidden_dim,
                                      batch_first=, norm_first=)
             _  (num_layers)
        ])

        .scheduler = DiagonalBatchingScheduler(num_layers, -)  
        .grouped_weights = 

     ():
        
        .grouped_weights = .scheduler.create_grouped_weights(.layers)

     () -> [torch.Tensor, [torch.Tensor]]:
        
        outputs = []
         layer_idx, layer  (.layers):
            x = layer(x)
            memory[layer_idx] = x.clone()
            outputs.append(x)
         x, memory

     () -> [torch.Tensor, [torch.Tensor]]:
        
        
        B, T, D = x.shape
        num_actual_segments = (T + .segment_length - ) // .segment_length

        
        diagonals = .scheduler.compute_diagonals()
        .scheduler.num_segments = num_actual_segments
        diagonals = .scheduler.compute_diagonals()

        
        segment_hidden_states = []

         seg_idx  (num_actual_segments):
            seg_start = seg_idx * .segment_length
            seg_end = (seg_start + .segment_length, T)
            segment = x[:, seg_start:seg_end, :]

            
             diag_idx, diagonal  (diagonals):
                
                relevant_pairs = [(l, s)  l, s  diagonal  s == seg_idx]

                 relevant_pairs:
                    layer_indices = [l  l, _  relevant_pairs]

                    
                     .grouped_weights   :
                        
                         layer_idx  layer_indices:
                            segment = .layers[layer_idx](segment)

            segment_hidden_states.append(segment)

        
        output = torch.cat(segment_hidden_states, dim=)

         output, memory

     () -> torch.Tensor:
        
         .grouped_weights  :
            .prepare_grouped_weights()

        memory = [torch.zeros(x.shape[], x.shape[], .hidden_dim, device=x.device)
                   _  (.num_layers)]

         use_diagonal:
            output, _ = .forward_diagonal(x, memory, num_segments=-)
        :
            output, _ = .forward_sequential(x, memory)

         output