wmpo-world-model-vla-training

name	wmpo-world-model-vla-training
title	WMPO: World Model-based Policy Optimization for VLA Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.09515
keywords	["Reinforcement Learning","World Models","Vision-Language-Action","Embodied AI","Robot Learning"]
description	Train Vision-Language-Action models for robotic control through world model simulation without real-world interaction—using pixel-based world models aligned with VLA features to enable self-correction and robust policy optimization.

Train Vision-Language-Action Robots Through Simulated World Models

Training robots with reinforcement learning typically requires expensive real-world trial-and-error. WMPO (World Model Policy Optimization) enables effective RL by learning a world model (pixel-based simulator) that accurately predicts VLA features trained on web-scale vision data. The robot then learns from simulated trajectories, avoiding real-world costs while gaining robust control behaviors.

The approach achieves improved sample efficiency, stronger overall performance, emergent self-correction abilities, and robust generalization—all critical for practical robotic manipulation.

Core Concept

WMPO addresses a fundamental challenge: vision-language-action models are strong visual reasoners but lack direct RL training signals. The system creates a closed-loop learning environment:

1. Pixel-Based World Model - Predicts next video frames given actions, ensuring alignment with visual domain
2. Feature Alignment - World model outputs are regularized to match VLA features, bridging vision and action
3. On-Policy RL - Uses GRPO on simulated trajectories to optimize robot behaviors
4. Self-Correction - Robot learns to recover from failures through iterative refinement

This approach replaces real-world risk with model-based simulation risk, dramatically reducing training cost.

Architecture Overview

• Vision-Language-Action Model (VLA): Perceives images, reasons about tasks, outputs actions
• Pixel-Based World Model: Predicts next frames [s_t, a_t] → s_{t+1}
• Feature Alignment Layer: Ensures world model outputs match VLA feature space
• Trajectory Simulator: Rolls out imagined episodes using world model
• Policy Optimizer (GRPO): Updates VLA parameters based on simulated rewards
• Real-World Validator: Periodic real validation to detect model divergence

Implementation Steps

Step 1: Train Pixel-Based World Model

Build a generative model predicting next frames from states and actions.

import torch
import torch.nn as nn
from torchvision.models import resnet50

class PixelWorldModel(nn.Module):
    

     ():
        
        ().__init__()
        .action_dim = action_dim
        .latent_dim = latent_dim

        
        .encoder = nn.Sequential(
            nn.Conv2d(input_channels, , kernel_size=, stride=, padding=),
            nn.ReLU(),
            nn.Conv2d(, , kernel_size=, stride=, padding=),
            nn.ReLU(),
            nn.Conv2d(, latent_dim, kernel_size=, stride=, padding=),
            nn.ReLU(),
        )

        
        .action_embed = nn.Sequential(
            nn.Linear(action_dim, ),
            nn.ReLU(),
            nn.Linear(, latent_dim)
        )

        
        .transition = nn.ModuleList([
            nn.ConvTranspose2d(latent_dim + latent_dim, latent_dim,
                              kernel_size=, padding=)
             _  (num_blocks)
        ])

        
        .decoder = nn.Sequential(
            nn.ConvTranspose2d(latent_dim, , kernel_size=, stride=, padding=),
            nn.ReLU(),
            nn.ConvTranspose2d(, , kernel_size=, stride=, padding=),
            nn.ReLU(),
            nn.ConvTranspose2d(, input_channels, kernel_size=, stride=, padding=),
            nn.Sigmoid()  
        )

     () -> torch.Tensor:
        
        
        latent = .encoder(current_frame)

        
        action_emb = .action_embed(action)
        action_emb = action_emb.unsqueeze(-).unsqueeze(-)
        action_emb = action_emb.expand(-, -, latent.shape[-], latent.shape[-])

        combined = torch.cat([latent, action_emb], dim=)

        
        next_latent = combined
         transition  .transition:
            next_latent = transition(next_latent)
            next_latent = torch.relu(next_latent)

        
        next_frame = .decoder(next_latent)

         next_frame

 ():
    
    optimizer = torch.optim.Adam(model.parameters(), lr=)
    criterion = nn.MSELoss()

     epoch  (num_epochs):
        total_loss = 

         batch_idx, (frame_t, action, frame_t1)  (train_dataloader):
            
            pred_frame = model(frame_t, action)

            
            loss = criterion(pred_frame, frame_t1)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            total_loss += loss.item()

        avg_loss = total_loss / (train_dataloader)
        ()