dreamgym-experience-synthesis-rl

name	dreamgym-experience-synthesis-rl
title	Scaling Agent Learning via Experience Synthesis
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.03773
keywords	["Experience Synthesis","Reinforcement Learning","World Models","Agent Training","Simulation"]
description	Scale agent learning by synthesizing diverse experiences using reasoning-based models instead of costly real-world rollouts, maintaining replay buffers with both real and synthetic interactions while using adaptive curriculum to focus on challenging tasks.

Title: Train Agents Efficiently With Synthesized Experience From World Models

Reinforcement learning typically requires millions of real-world interactions, which is expensive and slow. DreamGym solves this by training a reasoning-based experience model that generates synthetic rollouts: step-by-step predictions of state transitions and rewards. These synthetic experiences augment offline real-world data, enabling RL to converge faster with fewer real interactions.

The approach combines three components: experience model (synthesizes dynamics), replay buffer management (balances real and synthetic), and curriculum learning (focuses on useful tasks).

Core Concept

Reasoning-Based Experience Generation for RL:

• Experience Model: Uses step-by-step reasoning to predict state transitions
• Synthetic Rollout Generation: Creates full trajectory sequences without environment interaction
• Hybrid Replay Buffer: Initialized with real data, continuously enriched with synthetic experiences
• Adaptive Curriculum: Actively generates challenging tasks matched to current agent capability
• Efficient RL: Train on diverse synthetic experiences, occasionally validate on real environment

Architecture Overview

• Experience Model: Reasoning-based predictor of (state, action) → (next_state, reward)
• Replay Buffer Manager: Maintains real and synthetic experience ratio
• Curriculum Controller: Determines which synthetic tasks to generate
• Policy Network: Standard RL agent trained on combined data
• Validation Loop: Periodic real-world evaluation to ensure transfer

Implementation Steps

1. Build Reasoning-Based Experience Model

Create a model that generates realistic synthetic experiences using step-by-step reasoning.

class ReasoningBasedExperienceModel(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=512):
        self.state_encoder = nn.Linear(state_dim, hidden_dim)
        self.action_encoder = nn.Linear(action_dim, hidden_dim)

        # Transformer for reasoning
        .reasoning_transformer = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=, batch_first=),
            num_layers=
        )

        
        .next_state_head = nn.Linear(hidden_dim, state_dim)
        .reward_head = nn.Linear(hidden_dim, )
        .done_head = nn.Linear(hidden_dim, )

     ():
        
        
        state_emb = .state_encoder(state)
        action_emb = .action_encoder(action)

        
        combined = state_emb + action_emb
        reasoning = .reasoning_transformer(combined.unsqueeze()).squeeze()

        
        next_state = .next_state_head(reasoning)
        reward = .reward_head(reasoning)
        done = torch.sigmoid(.done_head(reasoning))

         next_state, reward, done

     ():
        
        states = [initial_state]
        actions = []
        rewards = []
        dones = []

        state = initial_state
         step  (max_steps):
            
            action = policy.select_action(state)
            actions.append(action)

            
            next_state, reward, done = .predict_step(state, action)

            states.append(next_state.detach())
            rewards.append(reward.item())
            dones.append(done.item() > )

             done.item() > :
                

            state = next_state

         {
            : torch.stack(states),
            : torch.stack(actions),
            : torch.tensor(rewards),
            : torch.tensor(dones)
        }

     ():
        
        optimizer = torch.optim.Adam(.parameters(), lr=learning_rate)

         state, action, next_state, reward, done  real_transitions:
            pred_next, pred_reward, pred_done = .predict_step(state, action)

            
            loss_state = F.mse_loss(pred_next, next_state)
            loss_reward = F.mse_loss(pred_reward, reward.unsqueeze(-))
            loss_done = F.binary_cross_entropy(pred_done, done.unsqueeze(-))

            total_loss = loss_state + loss_reward + loss_done

            optimizer.zero_grad()
            total_loss.backward()
            optimizer.step()