| name | full-stack-fp4-pretraining |
| category | ai_collection |
| description | Full-Stack FP4 pretraining framework — first complete NVFP4 LLM pretraining resolving stability bottlenecks in linear projections (LoRA-SVD), optimizers (AdamW second-moment transform, Muon Newton-Schulz), and attention (mixed-precision with forward-backward alignment). |
| tags | ["FP4","NVFP4","quantization","pretraining","low-bit-training","Muon-optimizer","attention-quantization"] |
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Core Problem
Existing NVFP4 pretraining targets only transformer linear layers, leaving optimizer states, optimizer arithmetic, and attention unexplored in 4-bit pipelines. Three modules have unique numerical failure patterns:
- Linear layers: hard quantization noise limits with dimension-propagated error amplification
- AdamW second moments: heavy-tailed non-negative values fragile to low-precision denominators
- Attention: error-prone computation paths demanding strict forward-backward quantization consistency
Methodology (Module-wise Precision Strategies)
Linear Projections
- LoRA-SVD lightweight decomposition suppresses quantization noise
- Breaks direct-quantization error ceiling
- Shrinks linear-only loss gap from 1.40% to 0.61%
Optimizers
- AdamW second-moment transformation for robust NVFP4 storage
- Native NVFP4 Newton-Schulz iterations for Root (Muon) optimizer
- Fully supports both AdamW and Muon in 4-bit
Attention
- Mixed-precision: quantizes Q/K/V and backward dS
- Guards vulnerable paths (PV, dOV^T branches) in BF16
- Unified tensor reuse sustains forward-backward alignment
Results
- 3B/64B-token pretraining: near-BF16 performance with merely 1.47% loss gap
- First feasible stable end-to-end NVFP4 LLM pretraining
- All modules are plug-and-play
When to Use
- Pretraining LLMs at scale where memory is a bottleneck
- When you need to quantize the entire training pipeline (not just weights)
- When using Muon optimizer and want to quantize Newton-Schulz iterations
- When attention quantization instability is blocking your low-bit training
Pitfalls
- Naive low-bit matrix multiplication has fast error accumulation
- PV and dOV^T attention branches are extremely sensitive — must keep in BF16
- Forward-backward quantization consistency is critical for attention stability
- AdamW second moments need special transformation (not just naive quantization)
Reference
arXiv:2607.04422 - "Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention" (Ding et al., 2026)
Activation
Full-Stack FP4, NVFP4 pretraining, 4-bit pretraining, FP4 LLM training, quantized optimizer, Muon optimizer FP4, attention quantization, low-bit LLM training, LoRA-SVD quantization
