mamba-architecture

name	mamba-architecture
description	State-space model with O(n) complexity vs Transformers' O(n²). 5× faster inference, million-token sequences, no KV cache. Selective SSM with hardware-aware design. Mamba-1 (d_state=16) and Mamba-2 (d_state=128, multi-head). Models 130M-2.8B on HuggingFace.
category	ml-training
version	1.0.0
author	Synthetic Sciences
license	MIT
tags	["Model Architecture","Mamba","State Space Models","SSM","Linear Complexity","Long Context","Efficient Inference","Hardware-Aware","Alternative To Transformers"]
dependencies	["mamba-ssm","torch","transformers","causal-conv1d"]

Mamba - Selective State Space Models

Quick start

Mamba is a state-space model architecture achieving O(n) linear complexity for sequence modeling.

Installation:

# Install causal-conv1d (optional, for efficiency)
pip install causal-conv1d>=1.4.0

# Install Mamba
pip install mamba-ssm
# Or both together
pip install mamba-ssm[causal-conv1d]

Prerequisites: Linux, NVIDIA GPU, PyTorch 1.12+, CUDA 11.6+

Basic usage (Mamba block):

import torch
from mamba_ssm import Mamba

batch, length, dim = 2, 64, 16
x = torch.randn(batch, length, dim).to("cuda")

model = Mamba(
    d_model=dim,      # Model dimension
    d_state=16,       # SSM state dimension
    d_conv=4,         # Conv1d kernel size
    expand=2          # Expansion factor
).to("cuda")

y = model(x)  # O(n) complexity!
assert y.shape == x.shape

Common workflows

Workflow 1: Language model with Mamba-2

Complete LM with generation:

from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
from mamba_ssm.models.config_mamba import MambaConfig
import torch

# Configure Mamba-2 LM
config = MambaConfig(
    d_model=1024,           # Hidden dimension
    n_layer=24,             # Number of layers
    vocab_size=50277,       # Vocabulary size
    ssm_cfg=(
        layer=,     
        d_state=,        
        headdim=,         
        ngroups=           
    )
)

model = MambaLMHeadModel(config, device=, dtype=torch.float16)


input_ids = torch.randint(, , (, ), device=, dtype=torch.long)
output = model.generate(
    input_ids=input_ids,
    max_length=,
    temperature=,
    top_p=
)