| name | dreamgym-experience-synthesis-rl |
| title | Scaling Agent Learning via Experience Synthesis |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2511.03773 |
| keywords | ["Experience Synthesis","Reinforcement Learning","World Models","Agent Training","Simulation"] |
| description | Scale agent learning by synthesizing diverse experiences using reasoning-based models instead of costly real-world rollouts, maintaining replay buffers with both real and synthetic interactions while using adaptive curriculum to focus on challenging tasks. |
Title: Train Agents Efficiently With Synthesized Experience From World Models
Reinforcement learning typically requires millions of real-world interactions, which is expensive and slow. DreamGym solves this by training a reasoning-based experience model that generates synthetic rollouts: step-by-step predictions of state transitions and rewards. These synthetic experiences augment offline real-world data, enabling RL to converge faster with fewer real interactions.
The approach combines three components: experience model (synthesizes dynamics), replay buffer management (balances real and synthetic), and curriculum learning (focuses on useful tasks).
Core Concept
Reasoning-Based Experience Generation for RL:
- Experience Model: Uses step-by-step reasoning to predict state transitions
- Synthetic Rollout Generation: Creates full trajectory sequences without environment interaction
- Hybrid Replay Buffer: Initialized with real data, continuously enriched with synthetic experiences
- Adaptive Curriculum: Actively generates challenging tasks matched to current agent capability
- Efficient RL: Train on diverse synthetic experiences, occasionally validate on real environment
Architecture Overview
- Experience Model: Reasoning-based predictor of (state, action) → (next_state, reward)
- Replay Buffer Manager: Maintains real and synthetic experience ratio
- Curriculum Controller: Determines which synthetic tasks to generate
- Policy Network: Standard RL agent trained on combined data
- Validation Loop: Periodic real-world evaluation to ensure transfer
Implementation Steps
1. Build Reasoning-Based Experience Model
Create a model that generates realistic synthetic experiences using step-by-step reasoning.
class ReasoningBasedExperienceModel(nn.Module):
def __init__(self, state_dim, action_dim, hidden_dim=512):
self.state_encoder = nn.Linear(state_dim, hidden_dim)
self.action_encoder = nn.Linear(action_dim, hidden_dim)
.reasoning_transformer = nn.TransformerEncoder(
nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=, batch_first=),
num_layers=
)
.next_state_head = nn.Linear(hidden_dim, state_dim)
.reward_head = nn.Linear(hidden_dim, )
.done_head = nn.Linear(hidden_dim, )
():
state_emb = .state_encoder(state)
action_emb = .action_encoder(action)
combined = state_emb + action_emb
reasoning = .reasoning_transformer(combined.unsqueeze()).squeeze()
next_state = .next_state_head(reasoning)
reward = .reward_head(reasoning)
done = torch.sigmoid(.done_head(reasoning))
next_state, reward, done
():
states = [initial_state]
actions = []
rewards = []
dones = []
state = initial_state
step (max_steps):
action = policy.select_action(state)
actions.append(action)
next_state, reward, done = .predict_step(state, action)
states.append(next_state.detach())
rewards.append(reward.item())
dones.append(done.item() > )
done.item() > :
state = next_state
{
: torch.stack(states),
: torch.stack(actions),
: torch.tensor(rewards),
: torch.tensor(dones)
}
():
optimizer = torch.optim.Adam(.parameters(), lr=learning_rate)
state, action, next_state, reward, done real_transitions:
pred_next, pred_reward, pred_done = .predict_step(state, action)
loss_state = F.mse_loss(pred_next, next_state)
loss_reward = F.mse_loss(pred_reward, reward.unsqueeze(-))
loss_done = F.binary_cross_entropy(pred_done, done.unsqueeze(-))
total_loss = loss_state + loss_reward + loss_done
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
