mano-manifold-optimization

name	mano-manifold-optimization
title	Mano: Restriking Manifold Optimization for LLM Training
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.23000
keywords	["Optimization","Manifold Methods","LLM Training","Convergence","Parameter Updates"]
description	Improve LLM training efficiency through manifold-based optimization that projects momentum onto tangent spaces and constrains updates on rotational Oblique manifolds. Achieves 1.75× faster convergence than Muon with reduced memory.

Mano: Restriking Manifold Optimization for LLM Training

Standard optimizers like AdamW rely on diagonal curvature estimates and ignore structural properties of weight matrices, while recent manifold methods like Muon sacrifice curvature information for global spectral normalization. Mano bridges this gap by projecting updates onto tangent spaces while constraining parameters to an Oblique manifold. The key innovation is rotating normalization—alternating between column-wise and row-wise normalization across iterations—that preserves curvature while enforcing geometric constraints.

The core insight is that LLM weight matrices have natural manifold structure, and respecting this geometry during optimization improves both convergence speed and stability.

Core Concept

Mano operates through three key mechanisms:

1. Manifold Projection: Updates are projected onto the tangent space of parameters, keeping the objective and solution unchanged while enforcing geometric constraints
2. Oblique Manifold Selection: Uses rotational Oblique manifold (yields shortest geodesic distance compared to alternatives)
3. Rotating Normalization: Alternates between column-wise normalization (odd iterations) and row-wise normalization (even iterations) for adaptive geometric constraints

This creates curvature-aware optimization without problem-specific assumptions.

Architecture Overview

• Momentum Computation: Standard momentum accumulation in tangent space
• Tangent Space Projection: Project accumulated momentum onto manifold surface
• Column-wise Normalization: Normalize columns (odd iterations)
• Row-wise Normalization: Normalize rows (even iterations)
• Update Application: Apply constrained update to parameters
• Learning Rate Schedule: Standard warmup and decay (compatible with existing schedules)

Implementation

The optimizer involves momentum projection, alternating normalization, and constrained updates.

Implement core Mano optimizer step:

import torch
import torch.nn as nn

class ManoOptimizer(torch.optim.Optimizer):
    """Manifold Optimization for LLM training."""

    def __init__():
        defaults = (lr=lr, betas=betas, eps=eps)
        ().__init__(params, defaults)
        .iteration = 

     ():
        
        loss = 
         closure   :
            loss = closure()

         group  .param_groups:
             p  group[]:
                 p.grad  :
                    

                grad = p.grad.data
                state = .state[p]

                
                 (state) == :
                    state[] = 
                    state[] = torch.zeros_like(p.data)
                    state[] = torch.zeros_like(p.data)

                exp_avg, exp_avg_sq = state[], state[]
                beta1, beta2 = group[]

                state[] += 
                step = state[]

                
                exp_avg.mul_(beta1).add_(grad, alpha= - beta1)

                
                exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value= - beta2)

                
                bias_correction1 =  - beta1 ** step
                bias_correction2 =  - beta2 ** step

                
                momentum_tangent = .project_to_tangent(
                    exp_avg, p.data, bias_correction1
                )

                
                 .iteration %  == :
                    
                    normalized = .row_normalize(momentum_tangent)
                :
                    
                    normalized = .column_normalize(momentum_tangent)

                
                p.data.add_(normalized, alpha=-group[])

        .iteration += 
         loss

     ():
        
        
        

        
        param_norm_sq = torch.(param ** )
        momentum_dot_param = torch.(momentum * param)

        
        tangent = momentum - (momentum_dot_param / (param_norm_sq + )) * param

         tangent / scale

     ():
        
        
        shape = tensor.shape
         (shape) > :
            tensor_2d = tensor.view(-, shape[-])
        :
            tensor_2d = tensor

        
        col_norms = torch.norm(tensor_2d, dim=, keepdim=) + 
        normalized_2d = tensor_2d / col_norms

         (shape) > :
             normalized_2d.view(shape)
         normalized_2d

     ():
        
        
        shape = tensor.shape
         (shape) > :
            tensor_2d = tensor.view(shape[], -)
        :
            tensor_2d = tensor

        
        row_norms = torch.norm(tensor_2d, dim=, keepdim=) + 
        normalized_2d = tensor_2d / row_norms

         (shape) > :
             normalized_2d.view(shape)
         normalized_2d

mano_optimizer = ManoOptimizer(model.parameters(), lr=)