world-compass-rl-world-models

name	world-compass-rl-world-models
title	WorldCompass: RL for Long-Horizon World Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.09022
keywords	["World Models","Reinforcement Learning","Video Generation","Action Following","Long-Horizon"]
description	Improve long-horizon world model fidelity using RL with clip-level rollouts and complementary reward functions for action accuracy and visual quality. Breaks computational constraints by evaluating candidate clips incrementally rather than full sequences, enabling efficient multi-objective optimization.

WorldCompass: RL for Long-Horizon World Models

Training diffusion-based world models for long-horizon video generation requires balancing two conflicting objectives: following user actions precisely and maintaining visual quality. Standard supervised fine-tuning struggles because errors compound across timesteps, and determining correct action interpretation is difficult without reward signals.

WorldCompass applies RL with three key innovations: clip-level rollouts reduce computation from O(N·G) to O(N+G), complementary reward functions act as mutual constraints preventing single-objective collapse, and negative-aware fine-tuning makes multi-step optimization feasible for diffusion models.

Core Concept

Standard approach: generate full video, evaluate once. Expensive and provides weak learning signals for early timesteps.

WorldCompass: generate clips at specific positions, score them independently with dual rewards (action accuracy + visual quality), use scores to update diffusion model. Clip-level granularity makes computation tractable and provides per-segment learning signals.

Architecture Overview

• Clip-Level Rollout: Generate candidate video clips at timesteps k within sequence, not full sequences
• Dual Rewards: Action-following accuracy AND visual quality, balancing competing objectives
• Efficient Sampling: Use best-of-N selection and timestep subsampling to reduce training cost
• Diffusion Model Update: Apply RL gradients to diffusion model parameters via policy gradient
• Error Mitigation: Negative-aware fine-tuning prevents model collapse on hard negative examples

Implementation

Implement clip-level rollout and dual-reward evaluation:

import torch
import torch.nn as nn
from diffusion_model import DiffusionWorldModel

class ClipLevelRollout:
    """Generates and evaluates video clips at specific timesteps."""

    def __init__(self, world_model, action_detector, quality_scorer):
        """
        Args:
            world_model: Diffusion-based world model
            action_detector: Model to evaluate action-following accuracy
            quality_scorer: Model to evaluate visual quality
        """
        .world_model = world_model
        .action_detector = action_detector
        .quality_scorer = quality_scorer

     ():
        
        clips = []
         _  (num_candidates):
            
            clip = .world_model.generate_clip(
                prefix=prefix_video,
                actions=action_sequence,
                steps=  
            )
            clips.append(clip)

         torch.stack(clips)

     ():
        
        
        detected_actions = .action_detector(generated_clip)

        
        action_accuracy = (detected_actions == action_sequence).().mean()

         action_accuracy

     ():
        
        
        
        
        

        quality_score = .quality_scorer(generated_clip)
         quality_score

     ():
        
        action_rewards = torch.stack([
            .compute_action_accuracy_reward(clip, action_sequence)
             clip  clips
        ])

        quality_rewards = torch.stack([
            .compute_visual_quality_reward(clip)
             clip  clips
        ])

        
        action_rewards = (action_rewards - action_rewards.mean()) / (action_rewards.std() + )
        quality_rewards = (quality_rewards - quality_rewards.mean()) / (quality_rewards.std() + )

        combined_rewards = action_weight * action_rewards + quality_weight * quality_rewards

         combined_rewards, action_rewards, quality_rewards