tv2tv-interleaved-video-text

name	tv2tv-interleaved-video-text
title	TV2TV: Unified Framework for Interleaved Language and Video Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.05103
keywords	["video-generation","language-generation","multimodal","flow-matching","user-control"]
description	Mixture-of-Transformers jointly learning language modeling and video flow matching, enabling interleaved text-video generation where semantic decisions happen in language, pixel generation in video, and users can intervene textually at any step.

Summary

TV2TV introduces a unified framework that decomposes video generation into interleaved text and video generation stages. The approach employs a Mixture-of-Transformers architecture jointly learning language modeling and video flow matching, enabling models to "think in words" about subsequent content before "acting in pixels" to produce frames. This factorization enables dynamic user control through textual interventions at any generation step.

Core Technique

Interleaved Generation Stages:

1. Language Stage: Model reasons about what should happen next in compact text
2. Video Stage: Model generates pixels matching the language description
3. User Intervention: User can provide text feedback to steer generation

Mixture-of-Transformers: Separate but jointly-trained transformers:

• Language Transformer: Autoregressive language modeling
• Video Transformer: Flow matching for pixel generation
• Routing: Mixture gates determine which transformer to use

Dynamic User Control: At any point, user can provide text instructions modifying the generation path without restarting.

Implementation

Mixture-of-Transformers architecture:

class MixtureOfTransformers:
    def __init__(self):
        self.language_transformer = LanguageTransformer()
        self.video_transformer = VideoTransformer()
        self.router = Router()

    def forward(self, context, modality_hint='auto'):
        # Route to appropriate transformer
        gate = self.router(context)

        if gate > 0.5 or modality_hint == 'language':
            # Generate text description
            output = self.language_transformer(context)
        :
            
            output = .video_transformer(context)

         output, gate