spatial-forcing-vla

name	spatial-forcing-vla
title	Spatial Forcing: Implicit Spatial Representation Alignment for VLA Model
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.12276
keywords	["vla","spatial-understanding","representation-alignment","3d-geometry","embodied-ai"]
description	Align intermediate visual embeddings in vision-language-action models with 3D geometric representations from pretrained foundation models. Improves spatial understanding and enables faster training (3.8x speedup) without explicit 3D inputs.

Spatial Forcing: Implicit Spatial Representation Learning for VLAs

Vision-Language-Action models need spatial understanding for embodied tasks, but explicit 3D inputs (depth maps, point clouds) introduce complexity and sensor noise. Spatial Forcing implicitly teaches spatial comprehension by aligning visual embeddings with 3D representations during training.

Core insight: strong spatial understanding emerges when visual processing layers learn to align with geometric structure. By forcing alignment with lightweight 3D foundation models, VLAs learn spatial reasoning without needing explicit 3D inputs at inference time.

Core Concept

Intermediate Alignment: Rather than modifying final outputs, align intermediate visual embeddings with 3D geometric representations. This implicitly teaches spatial structure at multiple processing stages.

Geometric Guidance: Use pretrained 3D foundation models (3D vision transformers) to provide geometric signals without requiring explicit depth sensors or point cloud inputs.

Efficient Training: The alignment process accelerates convergence, achieving 3.8x speedup over baseline VLA training.

Architecture Overview

• Visual Encoder: Standard Vision Transformer processing raw images
• 3D Foundation Model: Frozen pretrained model providing geometric signals
• Alignment Loss: Compares intermediate embeddings to 3D representations
• Geometric Projection: Lightweight layer mapping visual to spatial embeddings

Implementation Steps

Stage 1: Set Up 3D Foundation Model

Initialize frozen 3D feature extractor:

import torch
import torch.nn as nn
from transformers import AutoModel

class SpatialForcingVLA(nn.Module):
    def __init__(self, vla_model_name, geometric_model_name):
        super().__init__()

        # Load VLA visual encoder
        self.visual_encoder = AutoModel.from_pretrained(
            vla_model_name,
            trust_remote_code=True
        )

        
        .geometric_encoder = AutoModel.from_pretrained(
            geometric_model_name,  
            trust_remote_code=
        )

        
         param  .geometric_encoder.parameters():
            param.requires_grad = 

        
        .alignment_heads = nn.ModuleDict()
         layer_idx  [, , ]:  
            .alignment_heads[] = nn.Linear(
                .visual_encoder.config.hidden_size,
                .geometric_encoder.config.hidden_size
            )

     ():
        

        batch_size = images.shape[]

        
        visual_outputs = .visual_encoder(
            images,
            output_hidden_states=
        )

        
         depth_maps     .training:
             torch.no_grad():
                
                geometric_signal = ._depth_to_geometric(depth_maps)
                geometric_embeddings = .geometric_encoder(
                    geometric_signal
                )
        :
            geometric_embeddings = 

         visual_outputs, geometric_embeddings

     ():
        

        batch_size, height, width = depth_maps.shape
        points_list = []

         b  (batch_size):
            
            yy, xx = torch.meshgrid(
                torch.linspace(-, , height),
                torch.linspace(-, , width),
                indexing=
            )

            
            z = depth_maps[b] / depth_maps[b].()

            
            points = torch.stack([xx, yy, z], dim=-)  
            points = points.view(-, )  
            points_list.append(points)

        
        geometric_signal = torch.stack(points_list)  

         geometric_signal