voyager-3d-scene-generation

name	voyager-3d-scene-generation
title	Voyager: World-Consistent Video Diffusion for 3D Scene Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.04225
keywords	["3d-generation","video-diffusion","depth-estimation","world-caching"]
description	Generate spatially-coherent 3D point-cloud videos from single images using depth-fused diffusion with efficient world caching for infinite scene exploration.

Voyager: Long-Range and World-Consistent Video Diffusion

Core Concept

Voyager addresses a critical limitation in 3D scene generation: most methods cannot simultaneously generate RGB and depth sequences that remain spatially consistent across arbitrarily long videos. By jointly generating aligned RGB-depth pairs and implementing efficient world caching with point culling, Voyager enables direct 3D reconstruction without expensive post-processing pipelines.

Architecture Overview

• Geometry-Injected Conditioning: Use partial RGB and depth maps rather than RGB-only conditioning to reduce hallucinations in complex occlusions
• Depth-Fused Diffusion: Concatenate RGB and depth along spatial dimensions for pixel-level interaction
• World Caching System: Accumulate 3D points from generated frames; point culling removes redundancy, reducing memory ~40% while maintaining essential geometry
• Auto-Regressive Sampling: Generate arbitrary-length videos while maintaining world consistency through incremental point cloud updates
• Camera Path Control: Accept user-specified camera trajectories for controlled scene exploration

Implementation

Step 1: Prepare Dual-Modal Training Dataset

from typing import List, Dict, Tuple
import numpy as np

class DualModalDatasetBuilder:
    def __init__(self, num_training_pairs=100000):
        self.target_size = num_training_pairs
        self.auto_annotate_pipeline = self.AnnotationPipeline()

    class AnnotationPipeline:
        def estimate_metric_depth(self, single_image):
            """Use pretrained depth estimator (MiDaS, etc) to get metric depth"""
            depth_estimator = load_pretrained_depth_model('midas')
            relative_depth = depth_estimator(single_image)

            # Convert to metric scale using known scene structures
            metric_depth = self.scale_to_metric(relative_depth)
            return metric_depth

        def generate_camera_poses(self, image, num_frames=30):
            """Automatically generate camera trajectories"""
            poses = []

            # Infer scene bounds from depth and image content
            scene_bounds = self.estimate_scene_bounds(image)

            # Generate smooth camera path (circular, spiral, etc)
            for t in np.linspace(0, 1, num_frames):
                pose = self.generate_smooth_camera_motion(
                    scene_bounds, t
                )
                poses.append(pose)

             poses

         ():
            
            
            point_cloud = .image_to_point_cloud(
                single_image,
                .estimate_metric_depth(single_image)
            )

            rgb_frames = []
            depth_frames = []

             pose  camera_poses:
                
                rgb_frame = .render_rgb(point_cloud, pose)
                depth_frame = .render_depth(point_cloud, pose)

                rgb_frames.append(rgb_frame)
                depth_frames.append(depth_frame)

             rgb_frames, depth_frames

     ():
        
        dataset = []

         img  single_images:
            
            depth_map = .auto_annotate_pipeline.estimate_metric_depth(img)

            
            camera_poses = .auto_annotate_pipeline.generate_camera_poses(
                img, num_frames=
            )

            
            rgb_frames, depth_frames = (
                .auto_annotate_pipeline.render_target_frames(
                    img, camera_poses
                )
            )

            
            example = {
                : img,
                : depth_map,
                : camera_poses,
                : rgb_frames,
                : depth_frames,
                : (dataset),
            }

            dataset.append(example)

        ()
        ()

         dataset


builder = DualModalDatasetBuilder(num_training_pairs=)
training_data = builder.build_training_dataset(images)

Step 2: Implement Depth-Fused Diffusion Model

import torch
import torch.nn as nn

class DepthFusedDiffusionModel(nn.Module):
    def __init__(self, num_channels=8, num_layers=12):
        super().__init__()
        # 8 channels: 3 RGB + 1 Depth + 4 extra features
        self.num_channels = num_channels

        # Transformer blocks for spatial interaction
        self.spatial_fusion = nn.ModuleList([
            self.SpatialFusionBlock(num_channels)
            for _ in range(num_layers)
        ])

        # Temporal consistency module
        self.temporal_consistency = self.TemporalModule()

    class SpatialFusionBlock(nn.Module):
        """Enable pixel-level interaction between RGB and depth"""

        def __init__(self, channels):
            super().__init__()
            # Cross-modal attention: RGB queries, depth keys/values
            self.rgb_to_depth_attn = nn.MultiheadAttention(
                embed_dim=channels, num_heads=8
            )
            self.depth_to_rgb_attn = nn.MultiheadAttention(
                embed_dim=channels, num_heads=8
            )

        def forward(self, rgb, depth):
            """
            Args:
                rgb: [B, 3, H, W] - RGB channels
                depth: [B, 1, H, W] - Depth channel
            Returns:
                fused: [B, 4, H, W] - Interacted RGB-D
            """
            B, _, H, W = rgb.shape

            
            rgb_flat = rgb.view(B, , -).transpose(, )  
            depth_flat = depth.view(B, , -).transpose(, )  

            
            rgb_informed, _ = .depth_to_rgb_attn(
                rgb_flat, depth_flat, depth_flat
            )

            
            depth_informed, _ = .rgb_to_depth_attn(
                depth_flat, rgb_flat, rgb_flat
            )

            
            rgb_informed = rgb_informed.transpose(, ).view(B, , H, W)
            depth_informed = depth_informed.transpose(, ).view(B, , H, W)

            
            fused = torch.cat([rgb_informed, depth_informed], dim=)

             fused

     (nn.Module):
        

         ():
            ().__init__()
            .temporal_attention = nn.MultiheadAttention(
                embed_dim=, num_heads=
            )

         ():
            
            T, B, C, H, W = frame_sequence.shape

            
            frames_flat = frame_sequence.view(T, B*H*W, C)

            attended, _ = .temporal_attention(
                frames_flat, frames_flat, frames_flat
            )

             attended.view(T, B, C, H, W)

     ():
        

        
        rgb_frames = []
        depth_frames = []

        
        rgb_cond = image
        depth_cond = depth_init

         frame_idx  (num_frames):
            
             frame_idx > :
                warped_rgb = .warp_frame(
                    rgb_frames[-],
                    depth_frames[-],
                    camera_poses[:, frame_idx]
                )
            :
                warped_rgb = rgb_cond

            
            noisy_frame = .add_noise(
                torch.cat([warped_rgb, depth_cond], dim=),
                noise_level=frame_idx / num_frames  
            )

            
            denoised = noisy_frame
             fusion_block  .spatial_fusion:
                rgb_part = denoised[:, :, :, :]
                depth_part = denoised[:, :, :, :]
                denoised = fusion_block(rgb_part, depth_part)

            
            rgb_frame = denoised[:, :, :, :]
            depth_frame = denoised[:, :, :, :]

            rgb_frames.append(rgb_frame)
            depth_frames.append(depth_frame)

        
        frame_sequence = torch.stack(rgb_frames + depth_frames, dim=)
        consistent = .temporal_consistency(frame_sequence)

         rgb_frames, depth_frames

     ():
        
        
         rgb  

Step 3: Implement World Caching System

import open3d as o3d
from collections import defaultdict

class WorldCachingSystem:
    def __init__(self, memory_reduction_target=0.6):
        self.point_cloud = o3d.geometry.PointCloud()
        self.frame_points = defaultdict(list)
        self.memory_reduction_target = memory_reduction_target

    def accumulate_points_from_frame(self, rgb_frame, depth_frame,
                                    camera_pose, intrinsics):
        """Add 3D points from new frame to world cache"""

        # Project depth + camera pose to get 3D points
        points_3d = self.depth_to_world_coordinates(
            depth_frame, camera_pose, intrinsics
        )

        # Get colors from RGB frame
        colors = self.sample_colors_from_rgb(rgb_frame, points_3d)

        # Add to accumulating point cloud
        new_points = o3d.geometry.PointCloud()
        new_points.points = o3d.utility.Vector3dVector(points_3d)
        new_points.colors = o3d.utility.Vector3dVector(colors)

        self.point_cloud += new_points

        return len(points_3d)

    def point_culling(self, culling_ratio=0.4):
        """Remove redundant points to reduce memory"""

        initial_size = len(self.point_cloud.points)

        # Downsample point cloud aggressively
        culled = self.point_cloud.voxel_down_sample(
            voxel_size=0.05  
        )

        
        culled, _ = culled.remove_statistical_outlier(
            nb_neighbors=,
            std_ratio=
        )

        final_size = (culled.points)
        actual_reduction =  - (final_size / initial_size)

        ()
        ()

        .point_cloud = culled

         actual_reduction

     ():
        

        rgb_frames = []
        depth_frames = []
        culling_interval =   

         frame_batch_idx  (, (camera_poses_all), batch_size):
            batch_poses = camera_poses_all[
                frame_batch_idx : frame_batch_idx + batch_size
            ]

            
            rgb_batch, depth_batch = model(
                image, depth_init, batch_poses
            )

            rgb_frames.extend(rgb_batch)
            depth_frames.extend(depth_batch)

            
             rgb_f, depth_f, pose  (rgb_batch, depth_batch, batch_poses):
                .accumulate_points_from_frame(
                    rgb_f, depth_f, pose, intrinsics
                )

            
             (rgb_frames) % culling_interval == :
                reduction = .point_culling(culling_ratio=)
                ()

        
         rgb_frames, depth_frames, .point_cloud

Practical Guidance

1. Geometry-Aware Conditioning: Always provide depth alongside RGB conditioning. Depth prevents the diffusion model from hallucinating surfaces in occluded regions, dramatically improving consistency.

2. Joint RGB-Depth Generation: Design diffusion models to generate both modalities simultaneously with cross-attention, not as separate pipelines. Pixel-level RGB-depth interaction is crucial.

3. Camera Path Specification: Support user-defined camera trajectories (circular, spiral, free-form). This gives users control and enables reproducible generation.

4. World Caching for Infinite Video: Don't generate monolithic long videos. Instead, accumulate points, periodically cull redundancy (removing ~40% without visible quality loss), and continue generating. This enables arbitrarily long videos.

5. Point Culling Parameters: Use voxel downsampling with small voxel sizes (0.05 unit scale) and statistical outlier removal (20 neighbors, 2.0 std ratio). Tune based on target scene density.

6. Evaluation: Assess both visual quality (RGB fidelity) and geometric consistency (3D reconstruction error). A good model excels at both.

Reference

• Paper: Voyager (2506.04225)
• Architecture: Depth-fused diffusion transformer with world caching
• Dataset: 100,000+ automatic RGB-D video pairs via single-image rendering
• Key Innovation: Joint RGB-depth generation with efficient world state management