nemotron-3

name	nemotron-3
title	NVIDIA Nemotron 3: Efficient and Open Intelligence
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.20856
keywords	["language-model","mixture-of-experts","mamba-transformer","moe","efficient"]
description	Build efficient open-source LLMs via hybrid Mamba-Transformer MoE architecture with LatentMoE expert design, multi-token prediction training, FP4 precision, and multi-environment RL post-training—achieving 3.3× higher throughput than equivalently-sized models while maintaining state-of-the-art reasoning, coding, and tool-use capabilities.

Overview

Nemotron 3 combines five efficiency innovations to create high-performance open models. The core architectural change—hybrid Mamba-Transformer with MoE—reduces KV cache overhead while maintaining accuracy through careful expert design and multi-environment training.

Core Technique

Hybrid Mamba-Transformer MoE Architecture: Replace expensive self-attention with cheaper Mamba layers, retaining attention where needed.

class HybridMambaTransformerMoE:
    def __init__(self, num_layers=32, num_experts=128, top_k=6):
        self.layers = nn.ModuleList()
        for i in range(num_layers):
            if i % 2 == 0:
                # Even layers: Mamba (cheaper)
                layer = MambaLayer()
            else:
                # Odd layers: MoE (routed)
                layer = MixtureOfExpertsLayer(num_experts, top_k)
            self.layers.append(layer)

    def forward(self, x):
        """Alternate between Mamba and MoE."""
        for layer in self.layers:
            x = layer(x)
        return x

LatentMoE (Hardware-Aware Expert Design): Project tokens to latent space before routing to reduce routed parameters.

class LatentMoE(nn.Module):
    def __init__(self, input_dim=4096, latent_dim=1024, num_experts=128):
        .projection_in = nn.Linear(input_dim, latent_dim)
        .experts = nn.ModuleList([
            nn.Linear(latent_dim, latent_dim)  _  (num_experts)
        ])
        .router = Router(latent_dim, num_experts)
        .projection_out = nn.Linear(latent_dim, input_dim)

     ():
        
        
        latent = .projection_in(x)

        
        expert_idx = .router(latent)
        expert_out = .experts[expert_idx](latent)

        
        output = .projection_out(expert_out)

         output