simple-gpt-normalization-strategy

name	simple-gpt-normalization-strategy
title	SimpleGPT: Improving GPT via A Simple Normalization Strategy
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.01212
keywords	["Normalization Strategy","Training Stability","Learning Rate","Hessian Conditioning","Transformer Architecture"]
description	Place RMSNorm immediately after every linear layer to stabilize activation scales at O(sqrt(d)) and reduce Hessian spectral norm. Enables 3-10x larger learning rates and faster convergence without architectural changes; improves loss by 0.08 on 7B models.

SimpleGPT: Linear-Layer Normalization for Training Stability

Standard Transformer architectures place LayerNorm only before attention and MLPs, allowing activation scales to drift during training. SimpleGPT proposes a minimal change: insert RMSNorm immediately after every linear transformation (Q/K/V projections, MLPs, output projections). This simple shift dramatically improves optimization geometry, enabling much larger learning rates and faster convergence.

The key insight is that normalization right after linear projections stabilizes the activation scale that linear layers output, preventing the representation drift that limits learning rate. This is grounded in second-order optimization: the Hessian spectral norm directly limits maximum learning rate, and SimpleNorm reduces this bound significantly.

Core Concept

SimpleNorm applies a uniform pattern across the entire model:

Standard Transformer: Linear → Activation → (Attention/MLP)

SimpleGPT: Linear → RMSNorm → Activation → (Attention/MLP)

This seemingly small change has outsized impact because:

1. Stabilized activation scale: RMSNorm forces activations to remain at O(√d), preventing explosion or vanishing
2. Reduced Hessian curvature: Normalization smooths the loss landscape, reducing spectral norm of Hessian
3. Weight-scale invariance: Unlike unnormalized layers where curvature scales with ||W||₂², normalized layers maintain consistent curvature regardless of weight magnitude

Architecture Overview

• Linear projections (Q, K, V in attention; feed-forward up/down): Apply RMSNorm immediately after
• Output projections (attention out, MLP out): Apply RMSNorm after
• RMSNorm placement: Before activation functions (ReLU, GELU, etc.)
• Hyperparameter scaling: Increase learning rate 3-10× based on model size
• torch.compile integration: JIT compilation adds minimal overhead

Implementation

Step 1: Define RMSNorm Operation

Create an efficient RMSNorm that normalizes to unit RMS value.

# RMSNorm implementation
class RMSNorm(nn.Module):
    def __init__(self, hidden_dim: int, eps:  = ):
        
        ().__init__()
        .eps = eps
        .weight = nn.Parameter(torch.ones(hidden_dim))

     () -> torch.Tensor:
        
        
        rms = torch.sqrt(
            torch.mean(x ** , dim=-, keepdim=) + .eps
        )
        
        normalized = x / rms
         normalized * .weight