gorl-generative-online-rl

name	gorl-generative-online-rl
title	GoRL: Algorithm-Agnostic Online RL with Generative Policies
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.02581
keywords	["reinforcement-learning","generative-models","policy-learning","online-rl","actor-critic"]
description	Separates generative policy optimization through latent encoder (standard RL algorithms) and conditional decoder (frozen then refined), using two-timescale alternating schedule to eliminate gradient instability from direct generative policy optimization.

Summary

GoRL introduces Generative Online Reinforcement Learning, which addresses instability of expressive generative policies in online RL through structural decoupling. The method separates policy optimization into a learnable latent encoder optimized via standard RL (PPO/GRPO) and a conditional generative decoder refined on improved rollouts, using a two-timescale alternating schedule.

Core Technique

Structural Decoupling: Rather than treating the entire generative policy as a single learnable object, decompose it:

1. Latent Encoder π_θ(ε|s): Maps state to latent sample, optimized via standard RL
2. Conditional Decoder g_φ(s,ε): Generates actions from latent samples, refined on good trajectories

This avoids backpropagating through complex sampling chains, which causes gradient instability.

Two-Timescale Alternating: Use alternating optimization:

Phase 1: Optimize encoder via standard RL (PPO)
Phase 2: Refine decoder on trajectories with good returns
Phase 3: Reset encoder, keep improved decoder
Repeat

Gaussian Prior Anchoring: During decoder refinement, anchor the latent prior to a fixed N(0,I) to prevent distribution shift.

Implementation

Encoder-decoder architecture:

class GoRL_Policy:
    def __init__(self):
        self.encoder = mlp(state_dim -> latent_dim)  # π_θ
        self.decoder = mlp((state_dim + latent_dim) -> action_dim)  # g_φ

    def sample_action(self, state):
        epsilon = randn(latent_dim)  # Sample from N(0,I)
        action = self.decoder(concat(state, epsilon))
        return action

Phase 1 - Encoder RL optimization:

# Standard policy gradient with encoder
def encoder_loss(states, actions, returns):
     state, action, ret  (states, actions, returns):
        epsilon, log_prob = .encoder.sample_and_log_prob(state)
        loss = -log_prob * (ret - baseline(state))
     loss

encoder_optimizer.step(encoder_loss())