reinforce-ada-adaptive-sampling-rl-llms

name	reinforce-ada-adaptive-sampling-rl-llms
title	Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.04996
keywords	["adaptive sampling","non-linear RL","gradient estimation","RL for reasoning","signal recovery"]
description	Recover learning signals in RL for LLM reasoning by dynamically allocating sampling budget based on prompt difficulty. Use log-objective weighting (1/p for pass rate p) to prioritize challenging examples, achieving 2x convergence speedup versus uniform sampling while maintaining identical compute budgets across math, coding, and general benchmarks.

Reinforce-Ada: Adaptive Sampling Framework for Non-Linear RL Objectives

Core Concept

Standard group-based RL (GRPO) suffers signal loss when small groups produce identical rewards—advantages normalize to zero, causing gradient collapse. The root cause is undersampling difficult prompts, not model limitations. Reinforce-Ada recovers learning signals by dynamically allocating more samples to harder prompts, weighted by an implicit importance function derived from optimizing non-linear reward objectives.

Architecture Overview

• Non-Linear Objective Optimization: Frame RL as maximizing f(p_θ(x)) where f is log, power, or other monotonic function and p_θ(x) is per-prompt pass rate
• Implicit Weighting: Optimal gradient estimator naturally prioritizes difficult prompts with weight proportional to f'(p)
• Two Implementation Strategies: (1) Estimation-based: explicit reweighting with value network estimates; (2) Sequential: successive elimination of solved prompts
• Budget Efficiency: Allocate inference compute dynamically rather than uniformly across all prompts

Implementation Steps

1. Problem Formulation and Theoretical Framework

Define weighted RL objective:

J_f(θ) = E_x[f(p_θ(x))]

where f is a non-linear function of per-prompt pass rate. The gradient naturally acquires prompt-dependent weights:

# Non-linear objective framework
def weighted_rl_objective(pass_rates, f_func, f_prime_func):
    """
    J_f(θ) = E_x[f(p_θ(x))]
    ∇J_f = E_x[f'(p_θ(x)) · ∇p(x)]
    """

    # Log-objective: f(p) = log(p), f'(p) = 1/p
    # Weights difficult prompts (low p) more heavily
    def log_objective():
        weights = 1.0 / (pass_rates + 1e-8)  # Avoid division by zero
        return weights

    # Power objective: f(p) = p^α, f'(p) = α · p^(α-1)
    # Softer weighting scheme
    def power_objective(alpha=0.5):
        weights = alpha * (pass_rates ** (alpha - ))
         weights

    
     log_objective()