lumos-1-autoregressive-video

name	lumos-1-autoregressive-video
title	Lumos-1: On Autoregressive Video Generation from a Unified Model Perspective
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.08801
keywords	["Video Generation","Autoregressive Models","Positional Embeddings","Diffusion Forcing"]
description	Generate videos autoregressively by extending LLM architectures to spatiotemporal data. MM-RoPE balances frequency spectra across temporal and spatial dimensions, while Autoregressive Diffusion Forcing enables efficient parallel decoding. Lumos-1 (0.5B-3B variants) matches or exceeds Show-o2 and COSMOS on text-to-video with training on 48 GPUs.

Lumos-1: Unifying Video Generation Through Autoregressive LLMs

Video generation requires capturing temporal dynamics and spatial detail across multiple frames. Standard approaches treat frames independently (diffusion) or apply sequential RNNs (slow). Lumos-1 extends LLM architectures to video by addressing two fundamental challenges: (1) positional encodings designed for 1D text perform poorly on 3D video data, causing frequency imbalances, and (2) token-by-token generation is prohibitively slow. The solution is MM-RoPE (multimodal rotary position embeddings) for balanced spatiotemporal encoding, plus Autoregressive Diffusion Forcing for efficient parallel decoding. The result is a unified LLM-style model generating competitive video quality at tractable speed.

The key insight is that video is just another modality—positional encodings need careful frequency allocation across temporal and spatial axes, and generation can be parallelized through diffusion-style masking rather than sequential decoding.

Core Concept

Lumos-1 combines two technical innovations:

1. MM-RoPE: Extends 1D rotary position embeddings to 3D (time, height, width) with distributed frequency allocation preventing temporal domination of the spectrum
2. Autoregressive Diffusion Forcing (AR-DF): Parallel mask-based generation where multiple frames are generated simultaneously while maintaining temporal consistency through temporal tube masking

The model shares a single backbone (Llama-style) between text understanding and video generation, eliminating separate components.

Architecture Overview

• Llama Base Architecture: Standard decoder-only transformer (0.5B to 3B variants)
• MM-RoPE Positional Encoding: 3D position embeddings balancing temporal, height, and width frequencies
• Discrete Video Tokenizer: Cosmos tokenizer compressing video into spatiotemporal patches (8×8×4 compression)
• Unified Embedding: Single token vocabulary for both text and video tokens
• QK-Norm Stabilization: Layer normalization on Q and K in attention (stabilizes large-scale training)
• Parallel Diffusion Decoder: Mask-based generation with temporal tube constraints (all frames in same spatial position share mask)
• Multi-stage Pre-training: Text→ image → image-to-video → text-to-video progression

Implementation

The following demonstrates MM-RoPE and Autoregressive Diffusion Forcing:

import torch
import torch.nn  nn
 torch.nn.functional  F
 typing  , 

 (nn.Module):
    
     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .max_seq_len = max_seq_len
        .video_height = video_height
        .video_width = video_width
        .video_frames = video_frames

        
        
        

        inv_freq =  / ( ** (torch.arange(, hidden_dim, ).() / hidden_dim))
        .register_buffer(, inv_freq)

        
        
        
        .time_compression =     
        .space_compression =    

     () -> torch.Tensor:
        
         modality == :
            
            t = seq_positions.type_as(.inv_freq)
            freqs = torch.einsum(, t, .inv_freq)  

         modality == :
            
             seq_positions.dim() == :
                
                seq_len = seq_positions.shape[]
                total_spatial = .video_height * .video_width
                seq_positions = seq_positions.view(-, total_spatial)

            T, spatial = seq_positions.shape
            H, W = .video_height, .video_width

            
            t_indices = torch.arange(T, device=seq_positions.device).()
            h_indices = torch.arange(H, device=seq_positions.device).()
            w_indices = torch.arange(W, device=seq_positions.device).()

            
            inv_freq_t = .inv_freq * .time_compression
            inv_freq_h = .inv_freq * .space_compression
            inv_freq_w = .inv_freq * .space_compression

            
            
            freqs_t = torch.einsum(, t_indices, inv_freq_t[:.hidden_dim//])
            freqs_h = torch.einsum(, h_indices, inv_freq_h[:.hidden_dim//])
            freqs_w = torch.einsum(, w_indices, inv_freq_w[:.hidden_dim//])

            
            
            freqs_t = freqs_t.unsqueeze(-).unsqueeze(-).expand(-, -, H, W)
            freqs_h = freqs_h.unsqueeze().unsqueeze(-).expand(T, -, -, W)
            freqs_w = freqs_w.unsqueeze().unsqueeze().expand(T, H, -)

            freqs = torch.cat([freqs_t, freqs_h, freqs_w], dim=)
            freqs = freqs.view(-, .hidden_dim // )

        :
             ValueError()

        
        emb = torch.cat([freqs, freqs], dim=-)  
         emb.cos(), emb.sin()

 (nn.Module):
    
     ():
        ().__init__()
        .vocab_size = vocab_size
        .hidden_dim = hidden_dim
        .video_height = video_height
        .video_width = video_width

     () -> torch.Tensor:
        
        spatial_dim = seq_len // frames
        h = w = (spatial_dim ** )

        mask = torch.ones(batch_size, frames, h, w, dtype=torch.)

        
        num_spatial_positions = h * w
        num_to_mask = (num_spatial_positions * mask_fraction)
        spatial_mask_indices = torch.randperm(num_spatial_positions)[:num_to_mask]

         idx  spatial_mask_indices:
            i, j = idx // w, idx % w
            mask[:, :, i, j] = 

         mask.view(batch_size, seq_len)

     () -> [torch.Tensor, torch.Tensor]:
        
        batch_size, seq_len, hidden_dim = embeddings.shape

        
        mask = .create_temporal_tube_mask(batch_size, seq_len, frames, mask_fraction)

        
        
        logits = torch.randn(batch_size, seq_len, .vocab_size)  

        
        loss_mask = ~mask
         loss_mask.() > :
            
            loss = F.cross_entropy(
                logits[loss_mask],
                target_tokens[loss_mask],
                reduction=
            )
        :
            loss = torch.tensor(, device=embeddings.device)

         logits, loss

 (nn.Module):
    
     ():
        ().__init__()
        .vocab_size = vocab_size
        .hidden_dim = hidden_dim

        
        .embed_tokens = nn.Embedding(vocab_size, hidden_dim)

        
        .rope = MultimodalRotaryPositionEmbedding(hidden_dim)

        
        .layers = nn.ModuleList([
            LumosTransformerLayer(hidden_dim, num_heads, ff_dim=hidden_dim*)
             _  (num_layers)
        ])

        
        .lm_head = nn.Linear(hidden_dim, vocab_size)

        
        .ar_df = AutoregressiveDiffusionForcing(vocab_size, hidden_dim)

     () -> torch.Tensor:
        
        batch_size, seq_len = input_ids.shape

        
        embeddings = .embed_tokens(input_ids)

        
        cos, sin = .rope(
            torch.arange(seq_len, device=input_ids.device),
            modality=modality
        )
        embeddings = apply_rotary_pos_emb(embeddings, cos, sin)

        
         layer  .layers:
            embeddings = layer(embeddings)

        
        logits = .lm_head(embeddings)

         logits

 (nn.Module):
    
     ():
        ().__init__()
        .norm1 = nn.LayerNorm(hidden_dim)
        .norm2 = nn.LayerNorm(hidden_dim)

        
        .attention = nn.MultiheadAttention(
            hidden_dim, num_heads, batch_first=
        )
        .qk_norm = nn.LayerNorm(hidden_dim)

        
        .mlp = nn.Sequential(
            nn.Linear(hidden_dim, ff_dim),
            nn.GELU(),
            nn.Linear(ff_dim, hidden_dim)
        )

     () -> torch.Tensor:
        
        x_norm = .norm1(x)
        q = .qk_norm(x_norm)  
        attn_out, _ = .attention(q, x_norm, x_norm)
        x = x + attn_out

        
        x = x + .mlp(.norm2(x))
         x

 () -> torch.Tensor:
    
    
    dim = embeddings.shape[-]
    x1 = embeddings[..., :dim//]
    x2 = embeddings[..., dim//:]

    rotated = torch.cat([
        x1 * cos[..., :dim//] - x2 * sin[..., :dim//],
        x1 * sin[..., dim//:] + x2 * cos[..., dim//:]
    ], dim=-)

     rotated

 () -> :
    
    optimizer.zero_grad()

    input_ids = batch[]
    target_ids = batch[]
    frames = batch.get(, )

    
    logits = model(input_ids, modality=modality, frames=frames)

    
     modality == :
        
        _, loss = model.ar_df(
            model.embed_tokens(input_ids),
            target_ids,
            frames=frames
        )
    :
        
        loss = F.cross_entropy(
            logits.view(-, model.vocab_size),
            target_ids.view(-)
        )

    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), )
    optimizer.step()

     loss.item()