roboscape-physics-world-model

name	roboscape-physics-world-model
title	RoboScape: Physics-informed Embodied World Model
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.23135
keywords	["World Models","Robotic Vision","Physics-aware Video Generation","Keypoint Tracking","Embodied AI"]
description	Generate physically plausible robot manipulation videos by jointly learning RGB generation, temporal depth prediction, and keypoint dynamics. Enables training manipulation policies on synthetic data with strong correlation to simulator performance.

RoboScape: Physics-Informed World Models for Robotic Manipulation

Existing video diffusion models excel at generating visually realistic content, but they lack physical awareness needed for robotic manipulation tasks. A model might generate a video where a robot arm smoothly glides through an object, violating physics constraints, or where keypoints shift inconsistently across frames. This visual plausibility without physical correctness makes generated videos unsuitable for training robotic policies—policies trained on physically implausible demonstrations learn implausible behaviors.

RoboScape solves this by integrating physics into the video generation process through multi-task learning. Instead of generating RGB frames in isolation, the model simultaneously predicts depth, tracks physical keypoints, and generates RGB, with these tasks constraining each other. The result is a world model that generates physically consistent robotic videos suitable for both visual understanding and policy training.

Core Concept

RoboScape's innovation is treating video generation as a multi-task problem where physical constraints emerge from joint learning. The key insight is that:

1. Depth prediction enforces 3D geometric consistency across frames
2. Keypoint tracking ensures object deformations follow physical laws
3. RGB generation becomes implicitly constrained by the geometry and keypoints it must align with

By fusing intermediate features between the RGB and depth branches during decoding, the model learns that realistic videos have consistent geometry. Keypoint trajectories with high temporal continuity tell the model that abrupt position changes violate physics. This turns physics enforcement from an explicit loss term into an implicit emergent property of multi-task learning.

Architecture Overview

The RoboScape architecture uses an autoregressive Transformer with dual processing branches:

• RGB Branch: Decodes color information with causal temporal attention (current frame depends on previous frames) and bidirectional spatial attention within each frame
• Depth Branch: Predicts per-pixel depth maps with the same spatial-temporal attention structure, enabling 3D reconstruction
• Feature Fusion: Intermediate representations from the depth branch are additively fused with RGB features at each decoding layer, enforcing geometric consistency
• Keypoint Head: A lightweight predictor identifies and tracks high-motion points across frames, learning which visual elements have physical significance
• Temporal Consistency: Cross-frame attention mechanisms ensure depth and keypoints maintain smooth trajectories across time

Implementation

Step 1: Prepare training data with physics annotations