srpo-vla-policy

name	srpo-vla-policy
title	SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.15605
keywords	["Embodied AI","Vision-Language-Action Models","Policy Optimization","Reward Sparsity","Self-Supervision"]
description	Train VLA models for robotic manipulation by using the model's own successful trajectories as self-reference for reward—enable progress-based feedback for failed attempts without external rewards or demonstrations.

Train VLA Models with Self-Referential Rewards for Robotic Manipulation

Vision-Language-Action (VLA) models for robotics face extreme reward sparsity: tasks succeed or fail, with little feedback for in-between attempts. SRPO (Self-Referential Policy Optimization) breaks this bottleneck by using the model's own successful trajectories as self-reference. Failed attempts are measured against the model's successful ones from the same training batch, enabling dense progress-based rewards without external reward models or demonstrations.

This achieves 99.2% success on LIBERO (103% relative improvement from supervised baseline) by leveraging the model's latent world representation to assess behavioral progress robustly.

Core Concept

Standard VLA training for manipulation suffers from:

1. Reward Sparsity: Most trajectories fail; binary success/failure provides no gradient signal for intermediate progress
2. Sparse Demonstrations: Expert trajectories are expensive; RL without them requires dense rewards
3. Domain Shift: Reward models trained on one set of tasks fail on new objects/scenes

SRPO addresses all three by enabling self-comparison: rather than comparing failed trajectories to fixed rewards or external demonstrations, the model compares its current attempt to its own successful trajectories (from the same batch). A latent world model captures progress via compressed state representations, enabling robust progress estimation without task-specific fine-tuning.

Architecture Overview

• Vision-Language-Action Model: Encoder (vision + language) → action decoder; generates manipulation actions
• Latent World Model: Encodes trajectories into latent state representations capturing progress
• Self-Referential Comparison: Compare failed trajectory's latent progression to successful trajectories' latent states
• Progress-Based Rewards: Measure behavioral advancement using latent space distances, not task-specific metrics
• Batch-Based Training: Successful and failed trajectories in same batch enable direct comparison

Implementation Steps

Step 1: World Model for State Representation. Encode trajectories into latent progress space.

import torch
import torch.nn as nn

class LatentWorldModel(nn.Module):
    """
    Encodes manipulation trajectories into latent state representations.
    Captures progress toward goals without task-specific knowledge.
    """
     ():
        ().__init__()

        
        .encoder = nn.Sequential(
            nn.Linear(observation_dim + action_dim, ),
            nn.ReLU(),
            nn.Linear(, ),
            nn.ReLU(),
            nn.Linear(, latent_dim)
        )

        
        .predictor = nn.Sequential(
            nn.Linear(latent_dim + action_dim, ),
            nn.ReLU(),
            nn.Linear(, latent_dim)
        )

        .latent_dim = latent_dim

     ():
        
        latent_states = []

         t  ((observations)):
            obs_action = torch.cat([observations[t], actions[t]], dim=)
            latent = .encoder(obs_action)
            latent_states.append(latent)

         torch.stack(latent_states)  

     ():
        
        input_combined = torch.cat([latent_state, action], dim=-)
        next_latent = .predictor(input_combined)
         next_latent

     ():
        
        
        min_len = ((failed_latent_traj), (success_latent_traj))

        distances = []
         t  (min_len):
            
            dist = torch.norm(
                failed_latent_traj[t] - success_latent_traj[t],
                p=
            )
            distances.append(dist)

        
        avg_distance = torch.stack(distances).mean()

        
        progress = -avg_distance
         progress