semanticgen-video

name	semanticgen-video
title	SemanticGen: Video Generation in Semantic Space
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.20619
keywords	["video-generation","diffusion","semantic-space","efficiency","scalability"]
description	Accelerate video generation and enable long-video synthesis by decomposing into two diffusion stages: first generate compact semantic features for global planning, then generate VAE latents conditioned on semantics. Includes learnable semantic compression to improve training convergence—enabling minute-long videos with faster convergence than direct VAE modeling.

Overview

SemanticGen addresses two critical bottlenecks in video generation: slow convergence requiring hundreds of thousands of GPU-hours, and poor scaling to extended videos due to attention complexity. The key insight is that generation should occur first in semantic space for planning, then add details in pixel space.

Core Technique

The method decomposes video generation into two complementary stages operating on different feature spaces.

Two-Stage Generation Pipeline: Semantic space for global planning precedes pixel-space refinement.

# Two-stage semantic+pixel generation
class SemanticVideoGenerator:
    def __init__(self):
        self.semantic_generator = DiffusionModel()  # Compact semantic space
        self.pixel_generator = DiffusionModel()     # VAE latent space

    def generate_video(self, prompt, num_frames):
        """
        Stage 1: Semantic generation for global video planning
        Stage 2: Pixel generation conditioned on semantics
        """
        # Stage 1: Compact semantic video features
        semantic_features = self.semantic_generator.denoise(
            x_T=torch.randn(1, num_frames, semantic_dim),
            conditioning=prompt
        )
        # semantic_features: [1, num_frames, compact_semantic_dim]

        # Stage 2: VAE latents conditioned on semantics
        pixel_latents = self.pixel_generator.denoise(
            x_T=torch.randn(1, num_frames, vae_latent_dim),
            conditioning=semantic_features  # Condition on Stage 1 output
        )

        # Decode VAE latents to pixels
        video = vae_decoder(pixel_latents)
        return video

Semantic Space Compression: High-dimensional semantic representations converge slowly. A learnable MLP compresses them for faster training.