progressive-residual-warmup

name	progressive-residual-warmup
title	Progressive Residual Warmup for Language Model Pretraining
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.05369
keywords	["Pretraining","Optimization","Residual Connections","Layer-wise Warmup","Gradient Stability"]
description	Improves LLM convergence and downstream task performance by introducing time-dependent scaling to residual connections, enabling shallow layers to learn first before deeper layers activate. Apply during model pretraining to achieve 0.4-4.86 perplexity reduction.

Progressive Residual Warmup: Improving Training Stability Through Layer-wise Activation Scheduling

Deep transformer models suffer from unstable optimization during pretraining. All layers simultaneously modify representations from initialization, causing conflicting learning signals where downstream layers process ill-formed inputs from upstream layers. This creates inefficient convergence and suboptimal feature learning across model depth.

Progressive Residual Warmup (ProRes) solves this by introducing learnable time-dependent scaling factors to residual connections. Shallow layers activate immediately while deeper layers gradually "warm up" to full capacity, forcing shallow layers to establish stable representations before downstream layers begin learning. This orchestrated activation prevents representation collapse and improves optimization trajectory.

Core Concept

Residual connections normally read: x_{l+1} = x_l + F(Norm(x_l))

ProRes modifies this to: x_{l+1} = x_l + α(l,t) · F(Norm(x_l))

The scaling factor α(l,t) is deterministic based on layer depth (l) and training step (t), following a linear schedule that gradually increases from 0 to 1. Early layers see α ≈ 1 from step 1, while later layers remain near 0 initially, gradually increasing throughout warmup.

Architecture Overview

• Scheduling Mechanism: Per-layer warmup schedules based on depth; deeper layers have longer warmup periods
• No Additional Parameters: Uses purely deterministic scheduling—no learnable α values, preserving parameter efficiency
• Gradient Flow: Shallow layers receive strong gradients from deeper frozen paths, establishing features; gradients gradually route through deeper layers
• Integration Point: Drop-in modification to standard transformer architectures at the residual addition step

Implementation Steps

The core modification integrates cleanly into existing training loops. Implement the warmup schedule as a deterministic multiplier computed once per training step.

# Compute warmup scaling factor for each layer at step t
def compute_warmup_scale(layer_idx, current_step, total_warmup_steps, num_layers):
    """
    layer_idx: 0-indexed layer position (0 = shallowest, num_layers-1 = deepest)
    current_step: current training step
    total_warmup_steps: total steps for complete warmup (e.g., 5000)
    num_layers: total number of layers in model

    Returns scalar in [0, 1]
    """
    # Deeper layers have later warmup start points
    layer_warmup_start = (layer_idx / num_layers) * total_warmup_steps
    steps_into_layer_warmup = (, current_step - layer_warmup_start)
    layer_warmup_duration = total_warmup_steps - layer_warmup_start

    
    alpha = (, steps_into_layer_warmup / layer_warmup_duration)
     alpha