poet-orthogonal-llm-training

name	poet-orthogonal-llm-training
title	Reparameterized LLM Training via Orthogonal Equivalence Transformation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.08001
keywords	["LLM training","orthogonal reparameterization","spectrum-preserving","efficient optimization","generalization"]
description	Improve LLM training stability and generalization by reparameterizing weight matrices as orthogonal transformations, achieving better perplexity than AdamW with fewer trainable parameters.

Reparameterized LLM Training via Orthogonal Equivalence Transformation

Core Concept

POET reparameterizes weight matrices as products of fixed random initialization and learnable orthogonal transformations, decoupling spectral control from optimization dynamics. Rather than optimizing weights W directly, POET learns two orthogonal matrices R and P such that W = RW₀P, where W₀ is fixed. This approach preserves singular values (spectrum) while optimizing singular vectors, improving training stability and generalization without increasing total parameters.

Architecture Overview

• Spectrum-Preserving Parameterization: W = RW₀P where W₀ is frozen random, R/P are learned orthogonal transformations
• Three Training Phases: Vector probing analysis reveals distinct phases—conical-shell searching, stable learning on shell, final adjustment
• Efficient Approximations: Stochastic Primitive Optimization (SPO) factorizes large orthogonal matrices into products of smaller primitives; Cayley-Neumann parameterization avoids expensive matrix inversions
• Memory Optimization: Merge-then-reinitialize trick consolidates learned transformations periodically, reducing GPU memory by 30%
• Generalization Guarantee: Maintains small hyperspherical energy under initialization, connecting to established generalization theory

Implementation

Step 1: Implement Orthogonal Matrix Parameterization

import torch
import torch.nn as nn
from torch.linalg import qr

class OrthogonalLinear(nn.Module):
    """
    Reparameterized linear layer: W = RW₀P
    Learns R and P as orthogonal transformations of fixed random W₀.
    """

    def __init__(self, in_features, out_features):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features

        # Fixed random initialization (normalized Gaussian)
        W0 = torch.randn(out_features, in_features) / (in_features ** 0.5)
        self.register_buffer(, W0)

        
        .R_param = nn.Parameter(torch.eye(out_features))
        .P_param = nn.Parameter(torch.eye(in_features))

     ():
        
        W = .R_param @ .W0 @ .P_param
         torch.nn.functional.linear(x, W, )

 (nn.Module):
    

     ():
        ().__init__()
        .size = size
        .num_primitives = num_primitives
        .primitives = nn.ParameterList()

        
         _  (num_primitives):
            
            v = torch.randn(size)
            v = v / (torch.norm(v) + )
            .primitives.append(nn.Parameter(v))

     ():
        
        Q = torch.eye(.size, device=.primitives[].device)

         v  .primitives:
            v_norm = v / (torch.norm(v) + )
            
            H = torch.eye(.size, device=v.device) -  * torch.outer(v_norm, v_norm)
            Q = Q @ H

         Q