weight-norm-criticality-loss-spikes

name	weight-norm-criticality-loss-spikes
description	Weight-norm Criticality framework for understanding loss spikes in neural network training induced by normalization and weight decay interactions. Use when analyzing training instability, criticality phenomena, or optimization dynamics in deep learning models with scale-invariant components.
metadata	{"arxiv_id":"2607.21005","published":"2026-07-23","authors":"Xiaolong Li, Zhangchen Zhou, Zhi-Qin John Xu","tags":["neural-dynamics","criticality","weight-decay","normalization","loss-spikes","optimization"]}
license	Complete terms in LICENSE.txt

Weight-norm Criticality: A Mechanism for Loss Spikes

This skill provides a framework for understanding and analyzing weight-norm criticality, a mechanism that explains loss spikes in deep neural network training caused by the interaction between normalization layers and weight decay.

Core Concept

Most explanations of training instability focus on learning-rate criticality (Edge of Stability). However, this work identifies an additional and often overlooked weight-norm criticality:

• Normalization introduces scale-invariant components in the network
• Weight decay persistently shrinks parameter norms toward zero
• As weight decay coefficient increases, scale-invariant weight norms are driven toward zero
• The sharpness of the loss landscape increases rapidly, destabilizing optimization dynamics
• This results in abrupt loss spikes

Key Insights

1. Critical Boundary Phenomenon

Excessive weight decay drives scale-invariant weight norms past a critical boundary, causing training destabilization. This explains why weight penalties improve generalization but cannot be made arbitrarily strong.

2. Testable Predictions

The weight-norm criticality framework yields empirically testable predictions that have been validated in networks with scale-invariant components.

3. Mechanistic Understanding

Provides a new mechanistic understanding of loss spikes through the lens of weight-norm criticality, complementing existing learning-rate criticality theories.

When to Use This Framework

• Analyzing training instability in normalized networks (BatchNorm, LayerNorm, etc.)
• Investigating the effects of weight decay strength on optimization dynamics
• Understanding loss spike phenomena in deep learning training
• Designing stable training protocols for networks with scale-invariant components
• Researching criticality phenomena in neural network dynamics

Methodology

Analysis Steps

1. Identify scale-invariant components: Determine which parts of your network architecture exhibit scale invariance due to normalization
2. Monitor weight norms: Track the evolution of weight norms during training, particularly for scale-invariant components
3. Vary weight decay coefficient: Systematically increase weight decay strength to observe the critical transition point
4. Measure loss landscape sharpness: Compute or estimate the sharpness of the loss landscape as weight norms approach zero
5. Correlate with loss spikes: Establish the relationship between norm collapse and loss spike occurrence

Experimental Validation

• Compare training dynamics across different weight decay coefficients
• Measure the critical boundary where loss spikes begin to occur
• Validate predictions about the relationship between norm collapse and optimization instability

Pitfalls and Considerations

Common Misconceptions

• Not just learning rate: Weight-norm criticality is distinct from learning-rate criticality and can occur even with stable learning rates
• Scale-invariance requirement: The phenomenon specifically affects scale-invariant components; non-normalized networks may not exhibit this behavior
• Gradual vs abrupt: The transition is often abrupt rather than gradual, making it appear as sudden instability

Practical Implications

• Optimal weight decay: There exists an optimal range for weight decay that balances regularization benefits against criticality risks
• Architecture dependence: Different normalization schemes may exhibit different critical boundaries
• Dataset sensitivity: The critical boundary may vary depending on dataset complexity and size

Related Concepts

• Edge of Stability: Complementary framework focusing on learning rate criticality
• Loss landscape geometry: Understanding how weight norms affect loss surface properties
• Scale invariance in deep learning: Mathematical properties of normalized networks
• Optimization dynamics: How parameter evolution affects training stability

Activation Keywords

• weight-norm criticality
• loss spikes
• weight decay instability
• normalization criticality
• scale-invariant components
• training instability
• optimization dynamics
• critical boundary

References

• Original paper: arXiv:2607.21005
• Edge of Stability literature
• Scale invariance in deep learning research
• Loss landscape analysis methodologies