msign-stable-rank-restoration

name	msign-stable-rank-restoration
title	MSign: An Optimizer Preventing Training Instability via Stable Rank Restoration
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.01734
keywords	["Optimizer","Training Stability","Gradient Explosions","Matrix Decomposition","Numerical Stability"]
description	Prevent unrecoverable gradient explosions in LLM training by periodically restoring weight matrix stable rank through SVD-based matrix sign operations, eliminating sudden training failures without computational burden.

MSign: An Optimizer Preventing Training Instability via Stable Rank Restoration

Problem Context

LLM pretraining exhibits sudden, unrecoverable gradient explosions that waste significant computational resources. These failures lack obvious early warning and occur after thousands of training steps. Understanding their root cause requires analyzing weight matrix properties. Low stable rank (concentration of singular values) combined with high layer Jacobian alignment creates conditions for exponential gradient growth.

Core Concept

MSign identifies [stable rank collapse, Jacobian alignment, causal mechanisms] as precursors to training failures. The optimizer applies [periodic matrix sign operations, SVD-based restoration, lightweight updates] to restore stable rank by equalizing non-zero singular values to 1, breaking the chain that leads to gradient explosions.

Architecture Overview

• Diagnosis: Monitor stable rank of weight matrices and Jacobian alignment between layers
• Theory: Prove causal chain from rank → Jacobian → gradient explosion
• Solution: Periodic SVD to compute W_sign(W) where all singular values → 1
• Integration: Drop-in modification to existing optimizers
• Cost: <7% throughput reduction; can be applied selectively to attention layers only

Implementation

Step 1: Analyze stable rank and diagnose failure risk

Compute stable rank metrics to identify when training is at risk.

# Stable rank diagnostics
class StableRankMonitor:
    def __init__(self, check_interval=100):
        self.check_interval = check_interval
        self.step = 0
        self.stable_ranks = []
        self.jacobian_alignments = []

    def compute_stable_rank(self, weight_matrix):
        """
        Compute stable rank: (trace(M^T M))^2 / trace((M^T M)^2)
        Measures concentration of singular values.
        High stable rank = evenly distributed singular values (good).
        Low stable rank = concentrated singular values (bad).
        """
        
        gram = weight_matrix.T @ weight_matrix

        
        trace_gram = torch.trace(gram)

        
        gram_squared = gram @ gram
        trace_gram2 = torch.trace(gram_squared)

        
        stable_rank = (trace_gram ** ) / (trace_gram2 + )

         stable_rank.item()

     ():
        
        
        
        jac_i_flat = weight_matrix_i.flatten()
        jac_j_flat = weight_matrix_j.flatten()

        
        alignment = torch.nn.functional.cosine_similarity(
            jac_i_flat.unsqueeze(),
            jac_j_flat.unsqueeze()
        ).item()

         alignment

     ():
        
        .step += 

         .step % .check_interval != :
             , {}

        diagnostics = {: {}, : []}
        min_rank = ()

        
         name, param  model.named_parameters():
             (param.shape) >= :  
                sr = .compute_stable_rank(param.data)
                diagnostics[][name] = sr
                min_rank = (min_rank, sr)

        
        param_list = [p  p  model.parameters()  (p.shape) >= ]
         i  ((param_list) - ):
            alignment = .compute_jacobian_alignment(
                param_list[i], param_list[i + ]
            )
            diagnostics[].append(alignment)

        
        avg_alignment = (diagnostics[]) / (
            diagnostics[]
        )  diagnostics[]  

        is_at_risk = (min_rank < )  (avg_alignment > )

         is_at_risk, diagnostics