muon-optimizer-tail-memory-learning

name	muon-optimizer-tail-memory-learning
title	Muon Optimizer: Selective Parameter Optimization for Associative Memory
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2509.26030
keywords	["optimizer","muon","transformers","associative-memory","efficiency"]
description	Improve LLM training efficiency by selectively applying Muon optimizer to Value-Output attention weights and FFN layers, which function as associative memories. Use when training data exhibits heavy-tailed distributions requiring robust rare-fact learning.

Muon Optimizer: Selective Parameter Optimization for Associative Memory

This research explains why Muon optimizer outperforms Adam through mechanistic analysis. Muon excels at learning transformer components functioning as associative memories—specifically Value-Output attention and FFNs—particularly when training data has heavy-tailed distributions where rare facts appear infrequently.

Core Architecture

• Selective application: Apply Muon to VO+FFN, Adam to other parameters
• Associative memory insight: VO weights and FFNs implement content-addressable memory
• Heavy-tailed robustness: Muon handles rare but important facts better than Adam
• Computational efficiency: Near-full gains with only 40% of parameters using Muon

Implementation Steps

Apply selective optimizer assignment to transformer layers:

# Create mixed optimizer configuration targeting specific components
from torch.optim import Adam
from muon_optim import Muon

def create_selective_optimizer(model, learning_rate=1e-3):
    """Assign Muon to VO+FFN, Adam to remaining parameters"""

    vo_ffn_params = []
    other_params = []

    for name, param in model.named_parameters():
        if any(x in name for x in ["value_out", "ffn"]):
            vo_ffn_params.append(param)
        else:
            other_params.append(param)

    optimizer = torch.optim.Adam([
        {
            "params": vo_ffn_params,
            "optimizer": Muon,
            "lr": learning_rate,
            "momentum": 0.95
        },
        {
            "params": other_params,
            "optimizer": Adam,
            : learning_rate,
            : (, )
        }
    ])

     optimizer