video-world-models-spatial-memory

name	video-world-models-spatial-memory
title	Video World Models with Long-term Spatial Memory
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.05284
keywords	["video-generation","world-models","spatial-memory","3d-consistency","long-context"]
description	Enables long-term consistent video generation through three-tier memory architecture combining working memory, geometry-grounded point clouds, and episodic keyframes.

Video World Models with Long-term Spatial Memory

Core Concept

Video world models struggle to maintain scene consistency when revisiting previously generated locations due to limited temporal context windows. This work introduces a neuroscience-inspired memory architecture with three tiers—working memory (recent frames), spatial memory (static geometry), and episodic memory (historical keyframes)—enabling coherent generation across extended sequences. The key insight is grounding generation in persistent 3D point cloud representations that maintain physical consistency across time.

Architecture Overview

• Working Memory: Recent context frames (4-8 frames) capturing dynamic elements and temporal dependencies
• Spatial Memory: TSDF-fused point clouds representing static scene geometry, filtering out transient objects
• Episodic Memory: Sparse historical keyframes at regular intervals providing long-range context
• Static Point Cloud Rendering: Conditioning input that guides generation while preserving spatial consistency
• TSDF-Fusion Filtering: Removes dynamic elements, retaining only persistent scene structure
• Autoregressive Update: Newly generated frames' static components update spatial memory for future predictions

Implementation

The following code demonstrates the memory architecture and fusion process:

import torch
import torch.nn as nn
import numpy as np
from typing import List, Tuple, Optional
from collections import deque

class TrieredMemoryVideoModel(nn.Module):
    """
    Three-tier memory architecture for long-term consistent video generation.
    """
    def __init__(self, feature_dim: int = 768, max_keyframes: int = 16):
        super().__init__()
        self.feature_dim = feature_dim
        self.max_keyframes = max_keyframes

        # Working memory: deque of recent frames
        self.working_memory = deque(maxlen=8)

        # Episodic memory: sparse keyframes
        self.episodic_memory = deque(maxlen=max_keyframes)

        # Spatial memory: point cloud representation
        self.spatial_memory = None

    def update_working_memory(self, frame: torch.Tensor) -> None:
        """Add frame to working memory (most recent context)."""
        self.working_memory.append(frame)

    def tsdf_fusion(self, depth_map: np.ndarray,
                    camera_pose: np.ndarray,
                    voxel_size: float = ) -> np.ndarray:
        
        
        h, w = depth_map.shape
        y, x = np.meshgrid(np.arange(h), np.arange(w), indexing=)

        
        fx, fy = w / , h / 
        cx, cy = w / , h / 

        z = depth_map
        X = (x - cx) * z / fx
        Y = (y - cy) * z / fy

        
        points = np.stack([X, Y, z], axis=-).reshape(-, )

        
        points_homog = np.hstack([points, np.ones(((points), ))])
        points_world = (camera_pose @ points_homog.T)[:].T

        
        
        filtered_points = ._filter_dynamic_points(points_world)

         filtered_points

     () -> np.ndarray:
        
         scipy.spatial  cKDTree

         (points) < :
             points

        tree = cKDTree(points)
        variances = []

         i, point  (points):
            neighbors = tree.query_ball_point(point, neighbor_radius)
             (neighbors) > :
                neighbor_variance = np.var(points[neighbors], axis=).mean()
                variances.append(neighbor_variance)
            :
                variances.append(())

        
        static_mask = np.array(variances) < max_variance
         points[static_mask]

     () -> :
        
        .episodic_memory.append({
            : frame.detach().cpu(),
            : point_cloud
        })

     () -> torch.Tensor:
        
        condition_parts = []

        
         (.working_memory) > :
            working = torch.stack((.working_memory)).mean(dim=)
            condition_parts.append(working)

        
         .spatial_memory   :
            point_cloud_rendered = ._render_point_cloud(.spatial_memory)
            condition_parts.append(point_cloud_rendered)

        
         (.episodic_memory) > :
            episodic_frames = torch.stack([
                kf[]  kf  (.episodic_memory)
            ])
            episodic_cond = episodic_frames.mean(dim=)
            condition_parts.append(episodic_cond)

         torch.cat(condition_parts, dim=)  condition_parts  torch.zeros()

     () -> torch.Tensor:
        
        h, w = resolution
        rendering = np.zeros((h, w, ), dtype=np.float32)

        
         (point_cloud) > :
            x_norm = ((point_cloud[:, ] - point_cloud[:, ].()) /
                      (point_cloud[:, ].() - point_cloud[:, ].() + ))
            y_norm = ((point_cloud[:, ] - point_cloud[:, ].()) /
                      (point_cloud[:, ].() - point_cloud[:, ].() + ))

            x_px = (x_norm * (w - )).astype()
            y_px = (y_norm * (h - )).astype()

            valid = (x_px >= ) & (x_px < w) & (y_px >= ) & (y_px < h)
            rendering[y_px[valid], x_px[valid]] = [, , ]

         torch.from_numpy(rendering).permute(, , ).unsqueeze()

     () -> torch.Tensor:
        
        
        .update_working_memory(current_frame)

        
        new_points = .tsdf_fusion(depth_map, camera_pose)
         .spatial_memory  :
            .spatial_memory = new_points
        :
            .spatial_memory = np.vstack([.spatial_memory, new_points])

        
         (.working_memory) %  == :
            .add_episodic_keyframe(current_frame, .spatial_memory)

        
        memory_condition = .get_memory_condition()

         memory_condition

Practical Guidance

Memory Update Cadence: Update spatial memory every 4-8 generated frames to balance consistency with computational cost. More frequent updates improve geometric accuracy but increase overhead.

Point Cloud Density: Typical TSDF fusion produces 50K-500K points per view. Downsample to 100K points if memory becomes constrained; this minimally impacts quality.

Camera Pose Tracking: Ensure accurate camera pose estimation from the video generation model. Pose errors directly propagate to spatial memory misalignment.

Episodic Keyframe Interval: Store keyframes every 16-32 frames. This provides sufficient long-range context without excessive memory overhead.

TSDF Truncation Distance: Set truncation distance to 2-4 voxel sizes. This controls which depth variations are treated as dynamic vs. measurement noise.

Training Resolution: Train on 480×720 clips; spatial memory efficiently generalizes to 1080p at test time due to 3D geometry grounding.

Reference

The memory-augmented approach achieves improved consistency metrics:

• View Recall: Higher pixel-level consistency when revisiting scenes
• Camera Accuracy: Fewer off-trajectory hallucinations
• Temporal Coherence: More stable object and structure persistence

This method is particularly valuable for long-form video generation (100+ frames) and applications requiring geometric accuracy such as virtual environment synthesis or video editing.