vcrl-variance-curriculum-rl

name	vcrl-variance-curriculum-rl
title	VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2509.19803
keywords	["curriculum learning","reinforcement learning","reward variance","LLM training","sample difficulty","policy gradient"]
description	VCRL improves LLM mathematical reasoning by dynamically adjusting training sample difficulty based on group reward variance. Uses variance-based sampling with memory replay to focus on moderately-difficult samples where models succeed ~50% of the time, achieving 4.67-point improvement over GRPO on 8B models.

Variance-based Curriculum RL: Focus training on samples with optimal difficulty

The Problem with Standard RL Training

Rollout-based reinforcement learning methods like GRPO have a fundamental issue: they treat all samples equally during training, regardless of difficulty. The model might waste effort on trivial problems it already solves perfectly, or struggle without improvement on impossibly hard ones. Neither scenario drives learning forward. The sweet spot for learning exists where the model succeeds about 50% of the time—where uncertainty is maximal and every success or failure provides genuine signal.

The challenge is identifying which samples occupy that sweet spot without explicit labels. VCRL solves this by observing an implicit difficulty signal already present in training data: the variance of rewards across multiple rollouts of the same prompt.

Core Concept

VCRL recognizes a key insight from statistical learning: when you generate multiple rollout attempts on a single prompt, the distribution of successes and failures reveals the prompt's difficulty for the current model. Easy prompts succeed almost every time (low variance). Hard prompts fail almost every time (low variance). But moderately difficult prompts—where success is about 50/50—show high variance across rollouts.

The algorithm leverages this variance signal in two mechanisms. First, it continuously samples rewards from rollout groups and calculates normalized variance scores for each prompt. Prompts with high variance scores are deemed valuable for training. Second, it maintains a replay buffer (memory bank) of high-variance prompts and substitutes them into training batches whenever low-variance samples appear, ensuring the model always learns from optimally challenging examples.

Architecture Overview

• Variance Calculator: Computes normalized group reward variance p = σ²/σ²_max for each training sample based on k successes out of G rollouts
• Threshold Filter: Compares variance score p against threshold κ to identify low-value samples for replacement
• Memory Bank: Priority queue storing high-variance samples with their variance scores, keyed by prompts
• Batch Sampler: During training, replaces low-variance samples in batches with highest-variance samples from memory
• Policy Optimizer: Standard AdamW optimization over selected batch, unchanged from baseline

Implementation

Step 1: Initialize Memory Bank and Set Hyperparameters

The memory bank is a priority queue that accumulates high-value samples during training. Initialize it at the start of training and configure the key hyperparameters that control curriculum behavior.

from collections  defaultdict
 heapq  heappush, heappop
 numpy  np

 :
    

     ():
        .max_size = max_size
        .samples = []  
        .prompt_to_score = {}  

     ():
        
         variance_score > .prompt_to_score.get(prompt, ):
            .prompt_to_score[prompt] = variance_score
            heappush(.samples,
                    (variance_score, prompt, rollouts))

            
             (.samples) > .max_size:
                _, old_prompt, _ = heappop(.samples)
                 old_prompt  .prompt_to_score:
                     .prompt_to_score[old_prompt]

     ():
        
          .samples:
             []
        
         (.samples, reverse=)[:k]


memory_bank = VCRLMemoryBank(max_size=)


VARIANCE_THRESHOLD =   
GROUP_SIZE =             
BATCH_SIZE =            
MEMORY_SAMPLE_RATE =