moe-training

name	moe-training
description	Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.
category	ml-training
version	1.0.0
author	Synthetic Sciences
license	MIT
tags	["Emerging Techniques","MoE","Mixture Of Experts","Sparse Models","DeepSpeed","Expert Parallelism","Mixtral","DeepSeek","Routing","Load Balancing","Efficient Training"]
dependencies	["deepspeed","transformers","torch","accelerate"]

MoE Training: Mixture of Experts

When to Use This Skill

Use MoE Training when you need to:

• Train larger models with limited compute (5× cost reduction vs dense models)
• Scale model capacity without proportional compute increase
• Achieve better performance per compute budget than dense models
• Specialize experts for different domains/tasks/languages
• Reduce inference latency with sparse activation (only 13B/47B params active in Mixtral)
• Implement SOTA models like Mixtral 8x7B, DeepSeek-V3, Switch Transformers

Notable MoE Models: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)

Installation

# DeepSpeed with MoE support
pip install deepspeed>=0.6.0

# Megatron-DeepSpeed for large-scale training
git clone https://github.com/microsoft/Megatron-DeepSpeed
cd Megatron-DeepSpeed
pip install -r requirements.txt

# Alternative: HuggingFace Transformers
pip install transformers accelerate

Quick Start

Basic MoE Architecture

import torch
import torch.nn as nn

class MoELayer(nn.Module):
    """Sparse Mixture of Experts layer."""

    def __init__(self, hidden_size, num_experts=8, top_k=2):
        super().__init__()
        self.num_experts = num_experts
        self.top_k = top_k

        # Expert networks (FFN)
        self.experts = nn.ModuleList([
            nn.Sequential(
                nn.Linear(hidden_size, 4 * hidden_size),
                nn.GELU(),
                nn.Linear(4 * hidden_size, hidden_size)
            )
             _  (num_experts)
        ])

        
        .gate = nn.Linear(hidden_size, num_experts)

     ():
        
        batch_size, seq_len, hidden_size = x.shape

        
        x_flat = x.view(-, hidden_size)  

        
        gate_logits = .gate(x_flat)  

        
        gate_scores = torch.softmax(gate_logits, dim=-)
        topk_scores, topk_indices = torch.topk(gate_scores, .top_k, dim=-)

        
        topk_scores = topk_scores / topk_scores.(dim=-, keepdim=)

        
        output = torch.zeros_like(x_flat)

         i  (.top_k):
            expert_idx = topk_indices[:, i]
            expert_scores = topk_scores[:, i].unsqueeze(-)

            
             expert_id  (.num_experts):
                mask = (expert_idx == expert_id)
                 mask.():
                    expert_input = x_flat[mask]
                    expert_output = .experts[expert_id](expert_input)
                    output[mask] += expert_scores[mask] * expert_output

        
         output.view(batch_size, seq_len, hidden_size)