learnable-multipliers-lm-scaling

name	learnable-multipliers-lm-scaling
title	Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.04890
keywords	["Language Model Training","Optimization","Scaling","Weight Initialization"]
description	Adapt language model weight matrix scales automatically during training by adding learnable scalar and vector multipliers to each layer. Breaks the noise-weight-decay equilibrium that constrains scales based on hyperparameters rather than data, enabling data-driven scaling adaptation without inference cost or extensive tuning overhead.

When to Use This Skill

• Training large language models with Adam or Muon optimizers
• Scenarios where weight decay creates suboptimal equilibrium norms
• Models where μP multiplier tuning is burdensome (35+ manual values)
• Improving reasoning task performance (BBH, MATH) without increasing model size
• Attention, MLP, and SSM-based architectures

When NOT to Use This Skill

• Inference-critical applications requiring minimal parameter overhead (overhead is negligible but exists during training)
• Models already achieving saturation performance (marginal gains may not justify complexity)
• Scenarios with fixed weight matrices (non-learnable components)

Problem Summary

Weight decay in language model training creates a noise-WD equilibrium that constrains matrix layer scales based on optimization hyperparameters (η, λ) rather than data properties. This fixes weight scales as √(η/λ) regardless of task requirements, preventing data-driven scale adaptation. Current methods like μP require extensive manual tuning (35+ multiplier values) across model sizes.

Solution: Learnable Multipliers (LRM)

Introduce trainable scalar and vector multipliers that adapt layer scales during training without experiencing noise-driven expansion.

# Learnable multiplier parameterization
class LearnableMultiplier:
    def __init__(self, weight_matrix):
        # Scalar multiplier: applies uniform scale
        self.s = nn.Parameter(torch.ones(1))

        # Vector multipliers: per-row and per-column scales (selective placement)
        self.r = nn.Parameter(torch.ones(weight_matrix.shape[0]))  # Row scales
        self.c = nn.Parameter(torch.ones(weight_matrix.shape[1]))  # Column scales

        self.W = weight_matrix

    def forward(self):
        # Combine original weights with learned scales
        
         .s * (.r[:, ] * .W * .c[, :])