exgrpo-learning-from-experience

name	exgrpo-learning-from-experience
title	ExGRPO: Learning to Reason from Experience
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.02245
keywords	["experience-replay","RL-reasoning","GRPO","sample-efficiency","reasoning-training"]
description	Improve LLM reasoning efficiency by systematically reusing past rollouts through experience replay. ExGRPO organizes training data by success and diversity, applying a mixed-policy objective that prioritizes high-quality examples while maintaining exploration, achieving 3.5-7.6 point gains over on-policy methods.

ExGRPO: Experiential Group Relative Policy Optimization

Standard reinforcement learning for LLM reasoning discards data after a single gradient step. This is wasteful because good rollouts—ones that succeeded or showed diverse reasoning—are valuable for multiple training iterations. The challenge is that off-policy learning introduces distribution shift: if you trained on old data, current predictions diverge from that distribution, causing unstable gradients.

ExGRPO addresses this by intelligently organizing and reusing past rollouts. Rather than treating all experiences equally, it prioritizes successful completions and diverse outputs, then applies a mixed-policy objective that balances exploitation of good examples with exploration of new reasoning paths.

Core Concept

ExGRPO organizes experience replay around two signals:

1. Success signal: Did the rollout solve the problem?
2. Diversity signal: How different is this rollout's reasoning from others?

High-quality examples (successful and diverse) are retained and replayed multiple times across training. A mixed-policy objective prevents overexploitation: gradient updates are weighted by both the policy's likelihood (exploitation) and a uniform baseline (exploration), creating a natural curriculum where early training explores broadly, then focuses on promising directions.

Architecture Overview

• Rollout collector: Generate reasoning trajectories via forward sampling
• Experience scorer: Rate rollouts on success and diversity
• Replay buffer: Organize high-quality examples by tier (top 25%, 50%, etc.)
• Mixed-policy optimizer: Balance on-policy and off-policy updates
• Curriculum scheduler: Gradually shift focus from exploration to exploitation

Implementation Steps

Start by implementing the experience scoring system:

import torch
import numpy as np
from collections import defaultdict

class ExperienceScorer:
    """
    Score rollouts on success and diversity.
    """
    def __init__(self, verifier_model, embedding_model):
        self.verifier = verifier_model  
        .embedder = embedding_model  

     ():
        
        
        success = (.verifier.check(rollout[], problem))

        
        rollout_embedding = .embedder.encode(rollout[])

         previous_rollouts:
            previous_embeddings = [
                .embedder.encode(r[])
                 r  previous_rollouts
            ]

            
            distances = [
                torch.nn.functional.cosine_similarity(
                    rollout_embedding.unsqueeze(),
                    pe.unsqueeze()
                ).item()
                 pe  previous_embeddings
            ]

            
            diversity =  - np.mean(distances)
        :
            diversity =   

        
        quality_score =  * success +  * diversity

         {
            : quality_score,
            : success,
            : diversity
        }

     ():
        
        scores = []
         rollout  rollouts:
            score = .score_rollout(rollout, problem, previous_all)
            scores.append(score)

         scores