nextflow-multimodal-generation

name	nextflow-multimodal-generation
title	NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.02204
keywords	["Multimodal Models","Sequential Modeling","Image Generation","Video Generation","Unified Architecture"]
description	Build unified decoder-only transformers for multimodal tasks using 6 trillion interleaved text-image tokens with next-scale prediction for visual content—enabling fast 1024x1024 image generation (5 seconds), image editing, and video generation while rivaling specialized diffusion models.

Overview

NextFlow is a unified, decoder-only autoregressive transformer trained on massive interleaved text-image discrete token sequences. Unlike specialized models (text models, diffusion models, video models), NextFlow handles all modalities through a single consistent architecture.

Core Innovation: Recognize that text and images have fundamentally different structure—text is strictly sequential, images are hierarchical. Use next-token prediction for text but next-scale prediction for visual generation, achieving both speed and quality.

Architecture: Unified Decoder-Only Design

Modality Integration

Single Transformer:

• Processes both text and image tokens
• Shares parameters across modalities
• Unified training procedure
• Single inference engine

Token Representation:

Input: "A sunset over the ocean" + [IMAGE_TOKENS]
             ↓
         Tokenize
             ↓
text_tokens = [A, sunset, over, ocean]
image_tokens = [discrete_token_0, ..., discrete_token_n]
             ↓
Combined: [A, sunset, over, ocean, tok_0, tok_1, ...]
             ↓
Transformer processing

Training on Interleaved Data

6 Trillion Tokens:

• Mixed text-image sequences from diverse sources
• Interleaved layout (text and images together)
• Natural multimodal reasoning emerges
• Consistent objective function

Example Training Sequence:

[text: "A cat..."] [image_tokens: descr of cat photo]
[text: "...sitting on a desk"] [image_tokens: desk photo]

Next-Token vs. Next-Scale Prediction

Critical Design Insight

Text: Next-Token Prediction

• Natural ordering (sequential)
• Standard autoregressive approach
• Token-by-token generation

Images: Next-Scale Prediction

• Images are inherently hierarchical
• Generate from coarse to fine
• Multi-scale token prediction

Next-Scale Image Generation

Generate images hierarchically from low to high resolution:

def next_scale_image_generation(model, prompt: str, max_resolution=1024):
    """Generate image at progressively finer scales."""

    
    text_tokens = tokenizer.encode(prompt)

    
    image_tokens = []

    scales = [, , , , ]  
     scale  scales:
        
        
        scale_tokens = model.generate(
            text_tokens + image_tokens,
            num_new_tokens=tokens_per_scale(scale),
            temperature=,
        )
        image_tokens.extend(scale_tokens)

    
    image = decode_image_tokens(image_tokens)
     image