attention-residuals-for-depth-scaling

name	attention-residuals-for-depth-scaling
title	Attention Residuals
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.15031
keywords	["Attention Mechanism","Residual Connections","Depth Scaling","Hidden State Growth","Layer Aggregation"]
description	Replace uniform residual accumulation with depth-wise attention that selectively aggregates earlier layer representations. Improve gradient flow and model performance in deep architectures by learning content-dependent depth-wise selection.

Attention Residuals: Selective Layer Aggregation for Improved Depth Scaling

Standard residual connections in deep models use fixed unit weights to accumulate outputs from all previous layers, creating an uncontrolled growth of hidden state magnitudes and progressively diluting each layer's individual contribution. Attention Residuals solve this by allowing each layer to selectively attend over preceding representations with learned, input-dependent weights—analogous to multi-head attention but operating across depth dimension.

This technique improves gradient distribution in deep models and offers performance gains on downstream tasks, making it especially valuable for scaling to larger depths. The practical "Block" variant achieves this with minimal computational overhead by organizing layers into blocks and attending over block-level representations.

Core Concept

Attention Residuals replace the fixed accumulation operation with a learned attention mechanism:

Standard Residuals:

h_i = h_{i-1} + f_i(h_{i-1})  # Fixed unit weight accumulation

Attention Residuals (Full):

h_i = Attention(Q=h_{i-1}, K=[h_0,...,h_{i-1}], V=[h_0,...,h_{i-1}])
      # Each layer attends over all preceding layers

Attention Residuals (Block):

block_outputs = [output of block_0, output of block_1, ...]
h_i = Attention(Q=h_{i-1}, K=block_outputs, V=block_outputs)
      # Attend over block-level summaries, not individual layers

Architecture Overview

• Depth Dimension as Attention Axis — Treat layer depth as a sequence to attend over, similar to sequence position in standard attention
• Block Organization — Partition transformer into K blocks; each block attends over previous block representations rather than individual layers
• Cache-Based Pipeline Communication — Maintain incrementally updated cache of block outputs for efficiency
• Two-Phase Computation — Separate KV computation (per block) from Q computation (per layer within block)
• Drop-in Replacement — Compatible with standard attention infrastructure (FlashAttention-2)

Implementation Steps

Start by defining the block structure and implementing depth-wise attention. The key is treating layer indices as positions in a "depth sequence."

# Depth-wise attention replacing standard residual connections
import torch
import torch.nn.functional  F

 (torch.nn.Module):
    

     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .num_heads = num_heads

        .q_proj = torch.nn.Linear(hidden_dim, hidden_dim)
        .k_proj = torch.nn.Linear(hidden_dim, hidden_dim)
        .v_proj = torch.nn.Linear(hidden_dim, hidden_dim)
        .out_proj = torch.nn.Linear(hidden_dim, hidden_dim)

     ():
        
        
        all_layers = layer_cache + [current_hidden]
        stacked = torch.stack(all_layers, dim=)  
        batch_size, num_layers, seq_len, hidden_dim = stacked.shape

        
        Q = .q_proj(current_hidden)  
        K = .k_proj(stacked[:, :, :, :].reshape(batch_size, -, hidden_dim))
        V = .v_proj(stacked.reshape(batch_size, -, hidden_dim))

        
        Q = Q.view(batch_size, seq_len, .num_heads, -).transpose(, )
        K = K.view(batch_size, num_layers*seq_len, .num_heads, -).transpose(, )
        V = V.view(batch_size, num_layers*seq_len, .num_heads, -).transpose(, )

        scores = torch.matmul(Q, K.transpose(-, -)) / (hidden_dim ** )
        attn = F.softmax(scores, dim=-)
        output = torch.matmul(attn, V)

        output = output.transpose(, ).contiguous()
        output = output.view(batch_size, seq_len, hidden_dim)
         .out_proj(output)