rwml-reinforcement-world-models

name	rwml-reinforcement-world-models
title	Reinforcement World Model Learning for LLM-based Agents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.05842
keywords	["World Models","Self-Supervised Learning","GRPO","Agent Adaptation","Embedding Space"]
description	Train LLM agents to anticipate environment consequences by learning world models through reinforcement learning with embedding-space similarity rewards, avoiding task-specific labels while enabling robust environment adaptation.

Reinforcement World Model Learning for LLM-based Agents

Problem Context

LLM-based agents excel at language tasks but struggle to anticipate action consequences and adapt to environment dynamics. Standard pretraining emphasizes next-token prediction over semantic understanding of state transitions. This misalignment means agents cannot reliably model how actions transform environments, degrading performance in interactive tasks requiring consequence-aware planning.

Core Concept

RWML (Reinforcement World Model Learning) uses [GRPO, embedding-space rewards, self-supervised learning] to train agents to predict environment transitions without explicit task rewards or expert annotations. The key insight is measuring prediction quality through semantic similarity in embedding space rather than token-level fidelity, preventing collapse while enabling robust learning.

Architecture Overview

• Data collection: Rollout target model in environments; store interaction traces
• World model: Predict next state embedding from current state + action
• Reward function: Cosine similarity between predicted and actual next-state embeddings
• Training: GRPO optimization on embedding-space rewards; filter challenging samples
• No task rewards: Scaling without expert data; world model knowledge transfers to task RL

Implementation

Step 1: Collect interaction rollouts

Generate experience by running the base agent in environments. Store state-action-next_state triples with optional natural language annotations.

# Interaction data collection
def collect_rollouts(env, agent, num_episodes=1000, max_steps=50):
    trajectories = []
    for episode in range(num_episodes):
        state = env.reset()
        traj = []
        for step in range(max_steps):
            # Agent generates action
            action = agent.generate_action(state)
            next_state, reward, done = env.step(action)
            traj.append({
                'state': state,
                'action': action,
                : next_state,
                : reward
            })
             done:
                
            state = next_state
        trajectories.append(traj)
     trajectories