learning-4d-reasoning

name	learning-4d-reasoning
title	Learning to Reason in 4D: Dynamic Spatial Understanding for VLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.20557
keywords	["vision-language","spatial-reasoning","3d-reconstruction","dynamic","4d"]
description	Enable VLMs to perform dynamic spatial reasoning (DSR) by extracting 4D priors from videos and using Geometry Selection Modules (GSM) for selective injection. Provides DSR-Train dataset (50K QA pairs) and benchmark with six reasoning types, balancing geometric specialization with general video understanding—improving VLM 4D reasoning without degradation on general tasks.

Overview

Learning to Reason in 4D addresses a critical VLM limitation: difficulty with tasks requiring understanding of how objects move and relate spatially over time in 3D space. This framework combines automated 4D dataset generation with lightweight geometric-knowledge injection.

Core Technique

DSR Suite Dataset Pipeline: Transform in-the-wild videos into structured 4D reasoning training data.

# Automated 4D dataset generation from video
class DSRDatasetGenerator:
    def __init__(self):
        self.vision_foundation = VisionFoundationModel()
        self.qa_generator = QAGenerator()

    def create_dsr_dataset(self, video_collection):
        """
        Extract 4D priors and generate reasoning questions.
        """
        dataset = []

        for video in video_collection:
            # Extract 3D/4D information
            camera_poses = self.vision_foundation.extract_camera_poses(video)
            point_clouds = self.vision_foundation.extract_point_clouds(video)
            object_masks = self.vision_foundation.segment_objects(video)
            orientations = self.vision_foundation.estimate_orientations(video)
            trajectories = self.vision_foundation.track_trajectories(video)

            # Store 4D metadata
            video_4d = {
                'camera_poses': camera_poses,
                'point_clouds': point_clouds,
                'object_masks': object_masks,
                'orientations': orientations,
                'trajectories': trajectories
            }

            # Generate multiple-choice questions
            # Six reasoning types: distance, direction, orientation, speed, etc.
            questions = .qa_generator.generate_questions(
                video, video_4d,
                num_questions=,
                reasoning_types=[, , , , , ]
            )

             question_data  questions:
                dataset.append({
                    : video,
                    : question_data[],
                    : question_data[],
                    : question_data[],
                    : question_data[],
                    : video_4d
                })

         dataset