hierarchical-lvm-reasoning

name	hierarchical-lvm-reasoning
title	Chain of World: World Model Thinking in Latent Motion
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.03195
keywords	["World Models","Latent Reasoning","Structure-Motion Disentanglement","Video Understanding","Planning"]
description	Reason about long-horizon dynamics by disentangling structure and motion in video VAE latents. Learn continuous latent motion chains that preserve temporal coherence while predicting terminal keyframes, enabling efficient reasoning about multi-step scenarios.

Chain of World: Latent Motion Reasoning for Long-Horizon Planning

Traditional world models either predict dense frame sequences (computationally expensive) or use discrete action sequences (temporally discontinuous). Chain of World introduces a middle ground: learn continuous latent motion representations extracted from video VAE latents, then reason about multi-step trajectories as chains of motion in latent space. This approach preserves temporal structure while dramatically reducing computational requirements.

The core insight is to decompose each video segment into separable structure (static semantic content) and motion (how objects move). Reasoning then operates on compact continuous motion representations rather than full frames or discrete actions.

Core Concept

Chain of World implements three coordinated mechanisms:

1. Structure-Motion Disentanglement: Use pretrained video VAE to factorize scene into static structure latents and dynamic motion latents
2. Continuous Latent Motion Chains: Learn to predict chains of motion latents, which implicitly define object trajectories
3. Terminal Keyframe Prediction: Predict end-state visuals from motion chains, reconstructing the trajectory from sparse keyframes

Architecture Overview

• Input: Video segments or action sequences with visual context
• Video VAE Encoder: Decompose into structure (global semantics) and motion (directional dynamics)
• Motion Chain Generator: Predict sequences of motion latents given instruction
• Keyframe Decoder: Convert predicted motion chains to visual output
• Output: Structured representations of multi-step scenarios

Implementation Steps

Step 1: Implement structure-motion decomposition

Extract separable structure and motion from pretrained video VAE.

class VideoVAEDecomposer:
    """
    Decompose video into structure and motion latents using pretrained VAE.
    """

    def __init__(self, pretrained_vae_path):
        """Load pretrained video VAE (e.g., VideoMAE)."""
        self.vae = load_pretrained_vae(pretrained_vae_path)
        self.structure_dim = 
        .motion_dim = 

     ():
        
        
        segment_latent = .vae.encode(video_segment)  

        
        structure_latent = segment_latent[:, :.structure_dim]

        
        motion_latents = []
         t  ((video_segment) - ):
            
            frame_t = .vae.encode(video_segment[t:t+])
            frame_t1 = .vae.encode(video_segment[t+:t+])

            
            motion_t = frame_t1 - frame_t
            motion_latents.append(motion_t[:, .structure_dim:])

        motion_latents = np.stack(motion_latents, axis=).squeeze()

         structure_latent, motion_latents

     ():
        
        
        magnitude = np.linalg.norm(motion_latent)
         magnitude > :
            direction = motion_latent / magnitude
        :
            direction = np.zeros_like(motion_latent)

         direction, magnitude