nemotron-3-nano

name	nemotron-3-nano
title	Nemotron 3 Nano: MoE Hybrid Mamba-Transformer for Agentic Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.20848
keywords	["mixture-of-experts","mamba-transformer","efficient","agentic","moe"]
description	Efficient agentic reasoning via sparse MoE activating 50% parameters per token. Combines Mamba-Transformer hybrid with 6-of-128 expert routing, three-stage post-training (SFT, verifiable RL, RLHF), and Group Relative Length Control—achieving 3.3× inference throughput of competitors while maintaining 1M token context support and superior reasoning.

Overview

Nemotron 3 Nano achieves efficiency through sparse mixture-of-experts combined with hybrid Mamba-Transformer architecture, backed by sophisticated post-training methodology.

Core Technique

Sparse MoE with Hybrid Architecture: 31.6B total parameters, 3.2B active per token (20% activation).

class NemotronNanoMoE:
    def __init__(self):
        # 6-of-128 expert selection
        self.num_experts = 128
        self.top_k = 6  # Activate only 6 experts per token
        self.router = nn.Linear(hidden_dim, 128)
        self.experts = nn.ModuleList([MambaTransformerBlock() for _ in range(128)])

    def forward(self, x):
        # Router selects top-6 experts
        router_logits = self.router(x)
        expert_indices = torch.topk(router_logits, k=6, dim=-1).indices

        # Activate selected experts only
        output = 0
        for idx in range(6):
            expert = self.experts[expert_indices[:, idx]]
            output = output + expert(x)

        return output / 6

Three-Stage Post-Training:

def nemotron_nano_training(base_model):
    # Stage 1: Supervised Fine-Tuning
    model = sft_training(base_model, reasoning_data)

    
    model = rlvr_training(model)

    
    model = rlhf_with_length_control(model)

     model