dream2flow-robotic-manipulation

name	dream2flow-robotic-manipulation
title	Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.24766
keywords	["robotic manipulation","video generation","3D object flow","embodiment gap","zero-shot learning","trajectory optimization"]
description	Convert video generation model outputs into executable robotic manipulation by extracting 3D object flow trajectories as an intermediate representation. Enables zero-shot manipulation of diverse object types (rigid, articulated, deformable, granular) without task-specific training. Use when pre-trained video models capture plausible manipulation patterns but need grounding in low-level robot control.

When to Use This Skill

• Zero-shot robotic manipulation from video models without task-specific training
• Open-world scenarios with diverse object types and materials
• Problems where desired manipulation can be visualized as a video
• Systems with pre-trained video generation models available
• Tasks requiring adaptation to object variations without retraining

When NOT to Use This Skill

• Real-time control requiring <100ms latency (video generation is slow)
• Manipulation requiring precise collision avoidance (flow-based approximation may miss fine contacts)
• Scenarios without visual groundtruth for flow validation
• Tasks with hidden or occluded state critical to manipulation
• Highly stochastic domains (uncertainty in flow prediction hurts execution)

The Embodiment Gap Problem

Video generation models excel at predicting plausible human-like manipulation. But they generate sequences of images, not robot commands:

Video Model Output:
Frame 0: Hand above object
Frame 1: Hand grasps object
Frame 2: Hand moves object right
Frame 3: Object at goal location

Needed for Robot:
Joint angles α, β, γ → Joint velocity dα/dt, dβ/dt, dγ/dt

The gap: How do we get from visual frames to motor commands?

Dream2Flow bridges this gap using 3D object flow as an intermediate representation.

Core Approach: 3D Object Flow as Interface

Instead of trying to reverse-engineer gripper positions and object poses from video frames, Dream2Flow:

1. Extracts 3D flow: Track object motion trajectories across video frames in 3D space
2. Represents motion: Encode as dense optical flow (where each 3D point moves to)
3. Plans trajectory: Convert flow to continuous object trajectory in 3D
4. Executes via control: Either trajectory optimization or RL to achieve the target trajectory

This intermediate representation is:

• Visual-to-motor bridge: Connects visual understanding (video) to motor control (robot)
• Object-centric: Focuses on what matters (object motion) not robot kinematics
• Generalizable: Same flow extraction works for rigid, articulated, deformable, or granular objects

Architecture Pattern

# Dream2Flow pipeline for robotic manipulation
class Dream2FlowManipulator:
     ():
        .video_model = video_model  
        .flow_extractor = flow_extractor  
        .controller = controller  

     ():
        
        
        video_sequence = .video_model.generate(
            start_frame=current_image,
            goal_frame=goal_image,
            num_frames=  
        )

        
        
        flow_trajectory = .flow_extractor.extract_3d_flow(
            video_sequence,
            object_mask=object_mask,
            num_trajectories=  
        )

        
        object_trajectory = .flow_extractor.aggregate_trajectory(flow_trajectory)
        

        
        
         .controller.mode == :
            actions = .controller.optimize_trajectory(
                current_state=.robot_state(),
                target_trajectory=object_trajectory,
                constraints=[, ]
            )

        
         .controller.mode == :
            state = .robot_state()
             t  ((object_trajectory)):
                action = .controller.get_action(
                    current_state=state,
                    target_position=object_trajectory[t]
                )
                state = .execute_action(action)

         actions  trajectory_log

     ():
        
        flow_3d = []
         t  ((video_frames) - ):
            
            frame_t = video_frames[t]
            frame_t1 = video_frames[t + ]
            optical_flow_2d = .compute_optical_flow(frame_t, frame_t1)

            
            depth = .estimate_depth(frame_t)
            flow_3d_frame = .lift_to_3d(
                optical_flow_2d,
                depth,
                camera_intrinsics=.camera_K
            )
            flow_3d.append(flow_3d_frame)

         flow_3d