| name | lumos-1-autoregressive-video |
| title | Lumos-1: On Autoregressive Video Generation from a Unified Model Perspective |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.08801 |
| keywords | ["Video Generation","Autoregressive Models","Positional Embeddings","Diffusion Forcing"] |
| description | Generate videos autoregressively by extending LLM architectures to spatiotemporal data. MM-RoPE balances frequency spectra across temporal and spatial dimensions, while Autoregressive Diffusion Forcing enables efficient parallel decoding. Lumos-1 (0.5B-3B variants) matches or exceeds Show-o2 and COSMOS on text-to-video with training on 48 GPUs. |
Lumos-1: Unifying Video Generation Through Autoregressive LLMs
Video generation requires capturing temporal dynamics and spatial detail across multiple frames. Standard approaches treat frames independently (diffusion) or apply sequential RNNs (slow). Lumos-1 extends LLM architectures to video by addressing two fundamental challenges: (1) positional encodings designed for 1D text perform poorly on 3D video data, causing frequency imbalances, and (2) token-by-token generation is prohibitively slow. The solution is MM-RoPE (multimodal rotary position embeddings) for balanced spatiotemporal encoding, plus Autoregressive Diffusion Forcing for efficient parallel decoding. The result is a unified LLM-style model generating competitive video quality at tractable speed.
The key insight is that video is just another modality—positional encodings need careful frequency allocation across temporal and spatial axes, and generation can be parallelized through diffusion-style masking rather than sequential decoding.
Core Concept
Lumos-1 combines two technical innovations:
- MM-RoPE: Extends 1D rotary position embeddings to 3D (time, height, width) with distributed frequency allocation preventing temporal domination of the spectrum
- Autoregressive Diffusion Forcing (AR-DF): Parallel mask-based generation where multiple frames are generated simultaneously while maintaining temporal consistency through temporal tube masking
The model shares a single backbone (Llama-style) between text understanding and video generation, eliminating separate components.
Architecture Overview
- Llama Base Architecture: Standard decoder-only transformer (0.5B to 3B variants)
- MM-RoPE Positional Encoding: 3D position embeddings balancing temporal, height, and width frequencies
- Discrete Video Tokenizer: Cosmos tokenizer compressing video into spatiotemporal patches (8×8×4 compression)
- Unified Embedding: Single token vocabulary for both text and video tokens
- QK-Norm Stabilization: Layer normalization on Q and K in attention (stabilizes large-scale training)
- Parallel Diffusion Decoder: Mask-based generation with temporal tube constraints (all frames in same spatial position share mask)
- Multi-stage Pre-training: Text→ image → image-to-video → text-to-video progression
Implementation
The following demonstrates MM-RoPE and Autoregressive Diffusion Forcing:
import torch
import torch.nn nn
torch.nn.functional F
typing ,
(nn.Module):
():
().__init__()
.hidden_dim = hidden_dim
.max_seq_len = max_seq_len
.video_height = video_height
.video_width = video_width
.video_frames = video_frames
inv_freq = / ( ** (torch.arange(, hidden_dim, ).() / hidden_dim))
.register_buffer(, inv_freq)
.time_compression =
.space_compression =
() -> torch.Tensor:
modality == :
t = seq_positions.type_as(.inv_freq)
freqs = torch.einsum(, t, .inv_freq)
modality == :
seq_positions.dim() == :
seq_len = seq_positions.shape[]
total_spatial = .video_height * .video_width
seq_positions = seq_positions.view(-, total_spatial)
T, spatial = seq_positions.shape
H, W = .video_height, .video_width
t_indices = torch.arange(T, device=seq_positions.device).()
h_indices = torch.arange(H, device=seq_positions.device).()
w_indices = torch.arange(W, device=seq_positions.device).()
inv_freq_t = .inv_freq * .time_compression
inv_freq_h = .inv_freq * .space_compression
inv_freq_w = .inv_freq * .space_compression
freqs_t = torch.einsum(, t_indices, inv_freq_t[:.hidden_dim//])
freqs_h = torch.einsum(, h_indices, inv_freq_h[:.hidden_dim//])
freqs_w = torch.einsum(, w_indices, inv_freq_w[:.hidden_dim//])
freqs_t = freqs_t.unsqueeze(-).unsqueeze(-).expand(-, -, H, W)
freqs_h = freqs_h.unsqueeze().unsqueeze(-).expand(T, -, -, W)
freqs_w = freqs_w.unsqueeze().unsqueeze().expand(T, H, -)
freqs = torch.cat([freqs_t, freqs_h, freqs_w], dim=)
freqs = freqs.view(-, .hidden_dim // )
:
ValueError()
emb = torch.cat([freqs, freqs], dim=-)
emb.cos(), emb.sin()
(nn.Module):
():
().__init__()
.vocab_size = vocab_size
.hidden_dim = hidden_dim
.video_height = video_height
.video_width = video_width
() -> torch.Tensor:
spatial_dim = seq_len // frames
h = w = (spatial_dim ** )
mask = torch.ones(batch_size, frames, h, w, dtype=torch.)
num_spatial_positions = h * w
num_to_mask = (num_spatial_positions * mask_fraction)
spatial_mask_indices = torch.randperm(num_spatial_positions)[:num_to_mask]
idx spatial_mask_indices:
i, j = idx // w, idx % w
mask[:, :, i, j] =
mask.view(batch_size, seq_len)
() -> [torch.Tensor, torch.Tensor]:
batch_size, seq_len, hidden_dim = embeddings.shape
mask = .create_temporal_tube_mask(batch_size, seq_len, frames, mask_fraction)
logits = torch.randn(batch_size, seq_len, .vocab_size)
loss_mask = ~mask
loss_mask.() > :
loss = F.cross_entropy(
logits[loss_mask],
target_tokens[loss_mask],
reduction=
)
:
loss = torch.tensor(, device=embeddings.device)
logits, loss
(nn.Module):
():
().__init__()
.vocab_size = vocab_size
.hidden_dim = hidden_dim
.embed_tokens = nn.Embedding(vocab_size, hidden_dim)
.rope = MultimodalRotaryPositionEmbedding(hidden_dim)
.layers = nn.ModuleList([
LumosTransformerLayer(hidden_dim, num_heads, ff_dim=hidden_dim*)
_ (num_layers)
])
.lm_head = nn.Linear(hidden_dim, vocab_size)
.ar_df = AutoregressiveDiffusionForcing(vocab_size, hidden_dim)
() -> torch.Tensor:
batch_size, seq_len = input_ids.shape
embeddings = .embed_tokens(input_ids)
cos, sin = .rope(
torch.arange(seq_len, device=input_ids.device),
modality=modality
)
embeddings = apply_rotary_pos_emb(embeddings, cos, sin)
layer .layers:
embeddings = layer(embeddings)
logits = .lm_head(embeddings)
logits
(nn.Module):
():
().__init__()
.norm1 = nn.LayerNorm(hidden_dim)
.norm2 = nn.LayerNorm(hidden_dim)
.attention = nn.MultiheadAttention(
hidden_dim, num_heads, batch_first=
)
.qk_norm = nn.LayerNorm(hidden_dim)
.mlp = nn.Sequential(
nn.Linear(hidden_dim, ff_dim),
nn.GELU(),
nn.Linear(ff_dim, hidden_dim)
)
() -> torch.Tensor:
x_norm = .norm1(x)
q = .qk_norm(x_norm)
attn_out, _ = .attention(q, x_norm, x_norm)
x = x + attn_out
x = x + .mlp(.norm2(x))
x
() -> torch.Tensor:
dim = embeddings.shape[-]
x1 = embeddings[..., :dim//]
x2 = embeddings[..., dim//:]
rotated = torch.cat([
x1 * cos[..., :dim//] - x2 * sin[..., :dim//],
x1 * sin[..., dim//:] + x2 * cos[..., dim//:]
], dim=-)
rotated
() -> :
optimizer.zero_grad()
input_ids = batch[]
target_ids = batch[]
frames = batch.get(, )
logits = model(input_ids, modality=modality, frames=frames)
modality == :
_, loss = model.ar_df(
model.embed_tokens(input_ids),
target_ids,
frames=frames
)
:
loss = F.cross_entropy(
logits.view(-, model.vocab_size),
target_ids.view(-)
)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), )
optimizer.step()
loss.item()
