| name | sonicmoe-training-optimization |
| title | SonicMoE: Accelerating Mixture of Experts Training with IO-Aware and Tile-Aware Optimizations |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2512.14080 |
| keywords | ["mixture-of-experts","training-efficiency","GPU-optimization","memory-efficiency","tensor-operations"] |
| description | Optimize MoE training through memory-efficient backward pass, IO-aware kernel design overlapping memory operations, and token rounding routing. Avoid caching large-scale activations, fuse operations with GEMM, implement ping-pong scheduling. Achieve 1.86× compute throughput on Hopper GPUs and 16% higher TFLOPS in sparse configurations. |
Skill Summary
SonicMoE addresses training inefficiency in fine-grained and sparse MoE models through three complementary techniques. A memory-efficient algorithm redesigns the backward pass to avoid caching activations scaling with expert granularity. IO-aware kernel design overlaps memory operations using modern GPU features (Hopper, Blackwell), fusing token gather with GEMM and implementing ping-pong scheduling. Token rounding routing aligns per-expert token counts with GEMM tile sizes, eliminating wasted padding. Combined improvements achieve 1.86× compute throughput on Hopper GPUs and 16% higher TFLOPS in sparse configurations.
When To Use
- Training large sparse mixture-of-experts models where efficiency matters
- Projects using Hopper/Blackwell GPUs supporting advanced optimization features
- Scenarios where memory and compute efficiency are both critical
- Research on GPU kernel optimization for deep learning
When NOT To Use
- Legacy GPU architectures (optimization requires Hopper/Blackwell features)
- Small MoE models where optimization overhead isn't justified
- Frameworks without custom kernel support
- Scenarios where training throughput isn't the bottleneck
Core Technique
Three complementary optimization techniques:
1. Memory-Efficient Algorithm
Redesign MoE's backward pass to avoid caching activations that scale with expert granularity. Compute router gradients without materializing certain intermediate activations, reducing memory usage by up to 45% while maintaining mathematical equivalence. This is the most impactful optimization.
2. IO-Aware Kernel Design
Overlap memory operations with computation using Hopper and Blackwell GPU features:
- Fuse token gather operations with GEMM input loading
- Merge activation functions into GEMM epilogues
- Implement ping-pong scheduling to hide IO latency behind computation
These techniques yield "1.86× compute throughput improvement on Hopper GPUs" relative to prior work.
3. Token Rounding Routing
Novel routing algorithm rounds per-expert token counts to multiples of GEMM tile sizes, eliminating wasted computations from padding. Method preserves model quality while achieving "16% higher TFLOPS" in sparse MoE configurations compared to standard token-choice routing.
Results
- Memory usage: up to 45% reduction
- Compute throughput on Hopper: 1.86× improvement
- TFLOPS in sparse configurations: 16% improvement
- Training on fewer GPUs with comparable throughput
Implementation Notes
Implement memory-efficient backward pass redesign avoiding large activation caches. Use Hopper/Blackwell GPU features for IO-aware kernel fusions. Implement token rounding routing aligning with GEMM tile sizes. Validate memory savings and throughput improvements on your sparse MoE models.
