Build hybrid Mamba-Transformer models combining efficient Mamba-2 layers with standard attention to achieve 6x higher inference throughput while maintaining reasoning accuracy on long-context tasks.
Build hybrid Mamba-Transformer models combining efficient Mamba-2 layers with standard attention to achieve 6x higher inference throughput while maintaining reasoning accuracy on long-context tasks.
Nemotron Nano 2: Hybrid Mamba-Transformer for Efficient Reasoning
Core Concept
Nemotron-Nano-9B-v2 achieves state-of-the-art accuracy with significantly improved inference throughput by replacing most self-attention layers in Transformers with Mamba-2 layers. This hybrid approach maintains the reasoning capabilities of full Transformers while gaining the linear-time complexity benefits of Mamba. The architecture processes extended reasoning traces efficiently, achieving up to 6x higher throughput on reasoning workloads while supporting 128k token context on a single GPU.
Architecture Overview
Hybrid Layer Composition: Mix of Mamba-2 and Transformer self-attention layers
Strategic Attention Placement: Preserve full attention for critical reasoning steps
Model Compression: Pre-train on 20 trillion tokens then compress to target size
FP8 Training: Optimize memory efficiency during pre-training and inference
Long-Context Support: Enable 128k token processing within single GPU memory constraints
Implementation Steps
1. Design Hybrid Layer Mixing Strategy
Determine which layers use Mamba vs. Transformer attention:
defcreate_hybrid_architecture(
total_layers: int,
mamba_ratio: float = 0.7,
attention_positions: list[int] = None) -> list[str]:
"""
Design layer composition with Mamba and Transformer layers.
Default: ~70% Mamba for efficiency, ~30% attention for critical reasoning
Attention layers placed strategically at early, middle, and late stages
"""if attention_positions isNone:
# Default: preserve attention at key positions for information bottlenecks
attention_positions = [0, total_layers // 2, total_layers - 1]
layer_types = []
for i inrange(total_layers):
i attention_positions:
layer_types.append()
i < (total_layers * mamba_ratio):
layer_types.append()
:
layer_types.append()
layer_types
if
in
"attention"
elif
int
"mamba"
else
"attention"
return
2. Implement Mamba-2 Layers
Use selective state space models for efficient sequence processing:
defcreate_mamba2_layer(
hidden_size: int,
state_size: int = 16,
expand_factor: int = 2) -> "Mamba2Layer":
"""
Create a Mamba-2 layer combining selective state space model with gating.
"""classMamba2Layer:
def__init__(self):
self.input_projection = Linear(hidden_size, hidden_size * expand_factor)
self.state_matrix = StateSpaceMatrix(hidden_size * expand_factor, state_size)
self.output_projection = Linear(hidden_size * expand_factor, hidden_size)
self.gate = Linear(hidden_size * expand_factor, hidden_size * expand_factor)
defforward(self, x):
# Project input
x_expanded = self.input_projection(x)
# Apply selective SSM
y = self.state_matrix(x_expanded)
# Apply gating mechanism
gated = y * sigmoid(self.gate(x_expanded))
# Project back to hidden size
output = self.output_projection(gated)
return output
return Mamba2Layer()
3. Configure Attention Layers
Maintain standard Transformer attention at strategic positions:
defcreate_attention_layer(
hidden_size: int,
num_heads: int = 8,
max_seq_length: int = 128000) -> "AttentionLayer":
"""
Standard multi-head attention, optimized for long sequences.
"""classAttentionLayer:
def__init__(self):
self.num_heads = num_heads
self.head_dim = hidden_size // num_heads
self.q_proj = Linear(hidden_size, hidden_size)
self.k_proj = Linear(hidden_size, hidden_size)
self.v_proj = Linear(hidden_size, hidden_size)
self.out_proj = Linear(hidden_size, hidden_size)
defforward(self, x, attention_mask=None):
batch_size, seq_len, _ = x.shape
# Project to Q, K, V
q = self.q_proj(x).view(batch_size, seq_len, self.num_heads, self.head_dim)
k = self.k_proj(x).view(batch_size, seq_len, self.num_heads, self.head_dim)
v = self.v_proj(x).view(batch_size, seq_len, self.num_heads, self.head_dim)
# Compute attention with optional long-context optimization
attention_scores = torch.matmul(q, k.transpose(-2, -1)) / sqrt(self.head_dim)
if attention_mask isnotNone:
attention_scores = attention_scores + attention_mask
attention_weights = softmax(attention_scores, dim=-1)
output = torch.matmul(attention_weights, v)
# Reshape and project
output = output.view(batch_size, seq_len, -1)
returnself.out_proj(output)
return AttentionLayer()
4. Pre-training and Compression
Implement the Minitron compression strategy:
defpretrain_and_compress(
base_config: dict,
target_size: str = "9B",
pretrain_tokens: int = 20_000_000_000,
fp8_enabled: bool = True) -> "CompressedModel":
"""
Pre-train large model then compress to target size using Minitron strategy.
"""# Step 1: Pre-train larger base modelprint(f"Pre-training on {pretrain_tokens:,} tokens...")
base_model = create_hybrid_model(
hidden_size=base_config["hidden_size"],
num_layers=base_config["num_layers"],
num_heads=base_config["num_heads"]
)
if fp8_enabled:
base_model = convert_to_fp8(base_model)
# Train on diverse data corpus
base_model = train_model(base_model, dataset, pretrain_tokens)
# Step 2: Apply Minitron compressionprint(f"Compressing to {target_size}...")
target_hidden_size = map_size_to_hidden_dim(target_size)
target_num_layers = map_size_to_layers(target_size)
compressed = compress_model(
base_model,
target_hidden_size=target_hidden_size,
target_num_layers=target_num_layers,
method="layer_dropping_and_projection"
)
return compressed
5. Configure Long-Context Inference
Enable efficient processing of extended sequences:
defconfigure_long_context_inference(
model: "HybridModel",
max_tokens: int = 128000,
device: str = "cuda") -> dict:
"""
Set up inference for long-context processing on single GPU.
"""
config = {
"max_sequence_length": max_tokens,
"use_kv_cache": True,
"kv_cache_dtype": torch.float8_e4m3fn, # FP8 for memory efficiency"attention_implementation": "flash_attention_2", # Optimized kernels"mamba_scan_mode": "hardware_optimized",
"device": device,
"gradient_checkpointing": False# Only for training
}
# Apply configuration
model.config.update(config)
model = model.to(device)
if fp8_enabled:
model = quantize_model(model, "fp8")
return config
