coba-rl-budget-allocation

name	coba-rl-budget-allocation
title	CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.03048
keywords	["Budget Allocation","Reinforcement Learning","Curriculum Learning","Sample Selection","Training Efficiency"]
description	Dynamically allocate training budget across samples using a capability-oriented value function that measures per-sample training importance based on model capability evolution. Reduces training time via greedy heap-based allocation optimizing exploration-exploitation tradeoff.

CoBA-RL: Capability-Aware Budget Allocation

Standard RL training allocates compute uniformly across all training samples, but samples have asymmetric value as the model evolves. Early training requires diverse exploration; later stages benefit from focused improvement on difficult problems. CoBA-RL automatically balances this by dynamically allocating computational budget—training iterations, gradient updates, or rollouts—to high-value samples.

The approach uses a capability-oriented value function that scores samples based on the model's current capability level, then uses efficient greedy allocation to assign budget where it matters most. This trades per-sample training cost for dramatically faster overall convergence.

Core Concept

CoBA-RL decomposes budget allocation into two components:

1. Capability-Oriented Value Function: Maps (sample, model_capability) to training importance using a Beta distribution. Samples marked as failing become high-priority when the model's global failure rate is high (exploration phase); successful samples gain priority when failure rate drops (exploitation phase).

2. Greedy Heap-Based Allocation: Instead of solving expensive dynamic programming (O(M·B·range)), use a max-heap to iteratively select the B highest-value samples in O(B·log M) time.

This separates the conceptual problem (what should we prioritize?) from the computational method (how do we find it efficiently?).

Architecture Overview

• Capability Monitor: Tracks global task failure rate across the training set
• Value Function: Beta distribution computing per-sample importance given model capability
• Heap Allocator: Maintains priority queue of samples, returns top-B samples efficiently
• Training Loop: Standard GRPO/PPO with variable iteration counts per sample
• Feedback Cycle: Updated failure rate from each epoch refines next allocation

Implementation

Step 1: Define the Capability-Oriented Value Function

Create a function that scores sample importance based on current model capability.

# Capability-oriented value function
import numpy as np
from scipy.stats import beta

class CapabilityValueFunction:
    def ():
        
        .alpha = alpha
        .beta_param = beta_param

     () -> :
        
        
        
         sample_failed:
            
             beta.pdf(global_failure_rate, .alpha, .beta_param)
        :
            
              - beta.pdf(global_failure_rate, .alpha, .beta_param)

 :
     ():
        .samples = samples
        .value_fn = value_fn
        .values = [] * (samples)

     ():
        
         i, sample  (.samples):
            sample_id = sample[]
            sample_failed = last_results.get(sample_id, )
            .values[i] = .value_fn.compute_value(
                sample_failed,
                global_failure_rate
            )

     () -> []:
        
        
        top_indices = np.argsort(-np.array(.values))[:k]
         [.samples[i]  i  top_indices]