fastcurl-curriculum-rl-efficient-reasoning

name	fastcurl-curriculum-rl-efficient-reasoning
title	FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2503.17287
keywords	["Curriculum Learning","Reinforcement Learning","Context Scaling","Reasoning Models","GRPO","CoT Optimization"]
description	Train efficient reasoning models using stage-wise context scaling and complexity-aware data selection. Achieves 49.6% accuracy on AIME 2024 while reducing training steps by 50% through alternating compress-extend cycles that progressively refine reasoning quality.

Core Concept

FastCuRL addresses the training inefficiency of large reasoning models by jointly optimizing context length and training data complexity through a curriculum learning framework. The key insight is that controlling context length and selecting data based on problem complexity can significantly improve RL training efficiency while generating more concise Chain-of-Thought (CoT) outputs. The approach uses a cyclical compress-extend strategy that iteratively refines reasoning outputs.

Architecture Overview

FastCuRL integrates three main components:

• Group Relative Policy Optimization (GRPO): A resource-efficient RL algorithm that eliminates the need for a critic model by computing advantages from group-level scores rather than individual baseline scores
• Complexity-Aware Data Selection: Divides training data into three categories (L1, L2, L3) based on input prompt length correlation with output complexity
• Stage-wise Context Scaling: Alternates between compress phases (reducing context length) and extend phases (increasing context length) across multiple training stages

Implementation Steps

1. Group Relative Policy Optimization (GRPO) Algorithm

The GRPO objective maximizes policy improvements with KL divergence regularization and optional entropy bonus:

# Simplified GRPO loss computation
import torch
import torch.nn.functional as F

def compute_grpo_loss(policy_logits, old_policy_logits, rewards,
                      kl_coefficient=0.02, entropy_coeff=0.01, epsilon=0.2):
    """
    Compute Group Relative Policy Optimization loss.

    Args:
        policy_logits: logits from current policy
        old_policy_logits: logits from reference policy
        rewards: shape [batch, group_size]
        kl_coefficient: KL penalty weight
        entropy_coeff: entropy bonus weight
        epsilon: clipping threshold
    """
    # Normalize advantages across group
    mean_reward = rewards.mean(dim=-1, keepdim=True)
    std_reward = rewards.std(dim=-1, keepdim=True) + 1e-8
    advantages = (rewards - mean_reward) / std_reward

    
    log_ratio = policy_logits - old_policy_logits
    ratio = torch.exp(log_ratio)

    
    surr1 = ratio * advantages
    surr2 = torch.clamp(ratio, -epsilon, +epsilon) * advantages
    policy_loss = -torch.(surr1, surr2).mean()

    
    kl_loss = kl_coefficient * ((ratio - ) - log_ratio).mean()

    
    entropy = -torch.(torch.softmax(policy_logits, -) *
                         torch.log_softmax(policy_logits, -), dim=-)
    entropy_loss = -entropy_coeff * entropy.mean()

     policy_loss + kl_loss + entropy_loss