vla-4d-spatiotemporal

name	vla-4d-spatiotemporal
title	VLA-4D: Embedding 4D Awareness into VLA Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.17199
keywords	["Vision-Language-Action","Robotics","Spatiotemporal Reasoning","4D Perception","Manipulation"]
description	Enhance VLA models with spatiotemporal awareness by embedding both 3D spatial coordinates and temporal sequences: predict actions that include temporal parameters (duration) alongside spatial movements, achieving 97.4% robotic manipulation success by grounding reasoning in coherent 4D representations.

VLA-4D: Spatiotemporal Aware Robotic Manipulation

Standard vision-language-action models predict spatial movements but lack temporal understanding, making robots execute incoherent sequences of actions with temporal discontinuities. This skill demonstrates how to extend VLAs with 4D awareness—explicitly reasoning about both 3D space and 1D time—enabling robots to perform spatiotemporally coherent manipulation requiring fine-grained timing and smooth motion trajectories.

The core innovation is augmenting action spaces to include temporal parameters and grounding visual representations in explicit 4D coordinates, enabling models to understand when actions should occur, not just where.

Core Concept

VLA-4D implements 4D awareness through:

1. 4D Visual Representation: Encodes both 3D spatial positions and temporal sequences from video
2. Spatiotemporal Action Space: Actions include temporal parameters (duration) alongside spatial control
3. Temporal Grounding: Time-aware visual features enabling smooth action execution
4. Cross-Attention Fusion: Integrates spatial and temporal information via attention mechanisms

Architecture Overview

• Video Encoder: Extracts spatiotemporal features from observation sequences
• 4D Feature Embedding: Maps 3D coordinates + time into unified representation space
• Spatial Action Head: Predicts movement vectors (Δx, Δy, Δz), rotation (Δθ), gripper state
• Temporal Action Head: Predicts action duration (Δt) and timing
• Coherence Decoder: Ensures temporal continuity between sequential actions

Implementation Steps

The system extends standard VLA architectures with temporal reasoning.

1. Build Video Encoder for Spatiotemporal Features

Extract both spatial and temporal information from observation sequences.

class SpatiotemporalVideoEncoder(torch.nn.Module):
    """
    Encodes video sequences into spatiotemporal features.
    Captures both spatial structure and temporal dynamics.
    """
    def __init__(self, hidden_dim=768, num_frames=8):
        super().__init__()

        .hidden_dim = hidden_dim
        .num_frames = num_frames

        
        .spatial_encoder = torchvision.models.resnet50(
            pretrained=, replace_stride_with_dilation=[, , ]
        )

        
        .temporal_encoder = torch.nn.Sequential(
            torch.nn.Conv3d(, hidden_dim, kernel_size=(, , ), padding=(, , )),
            torch.nn.ReLU(),
            torch.nn.Conv3d(hidden_dim, hidden_dim, kernel_size=(, , ), padding=(, , ))
        )

        
        .feature_projection = torch.nn.Linear(hidden_dim, hidden_dim)

     ():
        
        batch_size, num_frames, _, height, width = video_frames.shape

        
        spatial_feats = []

         t  (num_frames):
            frame = video_frames[:, t]
            spatial_feat = .spatial_encoder(frame)  
            spatial_feats.append(spatial_feat)

        spatial_feats = torch.stack(spatial_feats, dim=)  

        
        temporal_features = .temporal_encoder(
            spatial_feats.permute(, , , , )  
        )

        
        temporal_features = temporal_features.permute(, , , , )

        
        batch_s, frames, h, w, channels = temporal_features.shape
        temporal_features = temporal_features.reshape(-, channels)
        temporal_features = .feature_projection(temporal_features)
        temporal_features = temporal_features.reshape(batch_s, frames, h, w, -)

         temporal_features