demystifying-rl-tool-agents

name	demystifying-rl-tool-agents
title	Demystifying RL for Long-Horizon Tool-Using Agents
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.21972
keywords	["Reinforcement Learning","Long-Horizon Agents","Tool Use","Reward Design","GRPO"]
description	Comprehensive recipe for RL-training tool-using agents spanning reward design, data synthesis, model scaling, and algorithm selection. Seven ranked findings: scale-dependent rewards (curriculum for 1.5B–3B; dense for 7B), semi-sparse 'Macro' rewards balance specialization/transfer, 1K-sample sweet spot with 4:3:3 difficulty mix. Achieves SOTA on TravelPlanner with smaller models than leading proprietary systems.

Ranked Findings

1. Scale-Dependent Reward Design (Critical)

Different model scales require fundamentally different reward structures.

For smaller models (1.5B–3B):

• Curriculum rewards transition from dense to sparse signals
• Dense-only rewards fail to provide learning signal
• Sparse-only rewards cause catastrophic exploration collapse

For larger models (7B):

• Simple dense sum rewards outperform curriculum approaches
• Additional complexity adds overhead without benefit
• Larger models handle dense reward surfaces more robustly

2. Overly Dense Rewards Create Alignment Tax

Task-specific dense rewards maximize in-domain performance but significantly degrade out-of-domain generalization.

Trade-off spectrum:

• Full dense: Best in-domain performance; poorest transfer
• Semi-sparse "Macro": Balanced performance and transferability (recommended)
• Sparse: Best transfer; weak in-domain learning signal

Finding: Macro rewards (task-level rather than step-level) offer optimal generalization-specialization tradeoff.

3. Consistent Scaling Benefits

Model capacity improvements yield substantial gains across all conditions.

1.5B → 7B transition: Substantial improvements in success rates across all reward signals—no reward design fully compensates for model capacity limitations.

4. Data Sweet Spot at ~1K Samples (Critical)

Approximately 1,000 training examples with balanced difficulty provide optimal trade-offs.

Behavior:

• <1K: Insufficient signal; underfitting
• ~1K: Peak generalization; Goldilocks zone
• >2K: Over-optimization on training distribution; degraded generalization despite marginal in-domain gains

5. Balanced Difficulty Prevents Reward Sparsity

Mixed easy:medium:hard ratios (4:3:3) maintain sufficient reward signals while teaching complex constraint satisfaction.

Alternative distributions:

• Homogeneous easy: Too much positive feedback; agents never learn hard constraints
• Homogeneous hard: Catastrophic collapse—sparse reward signal
• 4:3:3 mix: Optimal—sustains learning throughout training

6. Exploration Necessity Inversely Correlates with Model Capability

Sophisticated exploration algorithms (ARPO, DAPO) help smaller models but add computational overhead without benefit for larger models.

Recommendation:

• 1.5B–3B: Use DAPO/ARPO for better exploration
• 7B+: Standard GRPO sufficient; skip advanced exploration

7. Environmental Stability Matters

Agents tolerate up to 5% tool failure rates but experience noticeable degradation above 10%.

Impact: High environment instability hinders reward signal reliability, overriding other optimizations.

Implementation STAR Pipeline

Stage 1: Data Synthesis

Generate feasible queries with controlled difficulty via back-translation and validate in sandbox.

def synthesize_training_data(base_queries, target_count=1000, difficulty_split=[0.4, 0.3, 0.3]):
    """
    Back-translate: generate diverse queries by paraphrasing and difficulty-aware sampling.
    Validate queries are solvable before inclusion.
    """
    easy = []
    medium = []
    hard = []

    for query in base_queries:
        # Paraphrase for diversity
        variants = back_translate(query, n_variants=5)

        for variant in variants:
            # Check solvability in sandbox
            trajectory = execute_in_sandbox(variant, tool_budget=60)

            if trajectory.success:
                difficulty = estimate_difficulty(variant, trajectory)
                if difficulty == "easy":
                    easy.append((variant, trajectory))
                elif difficulty == "medium":
                    medium.append((variant, trajectory))
                else:
                    hard.append((variant, trajectory))

    # Balance to 4:3:3 ratio targeting total=1K
    n_easy = int(target_count * 0.4)
    n_medium = int(target_count * 0.3)
    n_hard = int(target_count * 0.3)

    return easy[:n_easy] + medium[:n_medium] + hard[:n_hard]

Stage 2: Supervised Fine-Tuning

Filter trajectories for success; use rejection sampling for quality control.

def supervised_finetune(training_data, model, epochs=5):
    """
    SFT on successful trajectories only; rejection sampling for quality.
    Filter teacher trajectories before training.
    """
    successful_trajectories = [
        (query, traj) for query, traj in training_data
        if traj.success
    ]

    # Rejection sampling: accept top-quality trajectories
    filtered = sorted(
        successful_trajectories,
        key=lambda x: compute_trajectory_quality(x[1]),
        reverse=True
    )
    filtered = filtered[:int(len(filtered) * 0.8)]  # Top 80%

    # Fine-tune on filtered data
    for epoch in range(epochs):
        for query, trajectory in filtered:
            loss = model.compute_sft_loss(query, trajectory)
            loss.backward()
    model.optimizer.step()

Stage 3: Reinforcement Learning

GRPO optimization with spectrum of reward signals matched to model scale.

def reinforcement_learning(model, training_data, model_scale_billion=7):
    """
    GRPO training with reward design matched to model capacity.
    Curriculum rewards for small models; dense for large models.
    """
    if model_scale_billion <= 3:
        reward_fn = curriculum_reward  # Dense → sparse transition
        algo = "DAPO"
    else:
        reward_fn = dense_sum_reward  # Simple dense sum
        algo = "GRPO"

    for step in range(10000):
        # Generate diverse rollouts
        rollouts = []
        for query in training_data:
            trajectories = model.sample(query, group_size=8)  # G=8
            rollouts.extend(trajectories)

        # Compute rewards
        rewards = [reward_fn(traj) for traj in rollouts]

        # GRPO/DAPO optimization
        if algo == "DAPO":
            loss = compute_dapo_loss(rollouts, rewards)
        else:
            loss = compute_grpo_loss(rollouts, rewards)

        loss.backward()
        model.optimizer.step()

Decision Checklist

• Model capacity: 7B+ → use dense reward; 1.5B–3B → use curriculum reward
• Data synthesis: Generate ~1,000 examples via back-translation with 4:3:3 difficulty split
• SFT filtering: Include successful trajectories only; reject-sample for top 80% quality
• Reward function: Semi-sparse "Macro" rewards (task-level, not step-level)
• Training data: Exactly ~1,000 examples; more causes over-optimization
• GRPO configuration: G=8 group size, 5 training epochs, KL-free training with increased clip bounds
• Tool budget: 60 calls per episode (tunable; affects task complexity ceiling)
• Format protocol: Enforce strict protocol; format errors receive reward=0
• Environment stability: Validate tool failure rate <5%; degrade gracefully at 5–10%
• Algorithm selection: 7B+ use standard GRPO; smaller models use DAPO/ARPO
• Context windows: 30K training, 32K inference
• Benchmark: Validate on TravelPlanner; compare against proprietary system baselines

Conditions

Effective Scenarios

• Long-horizon tasks requiring sequential tool invocation (planning, math, code generation, travel booking)
• Tasks with clearly definable success criteria and reward signals
• Models with sufficient scale (1.5B+) to support long context and complex reasoning
• Environments with manageable tool failure rates (<5% ideal, <10% acceptable)

Environmental Requirements

• Sandbox execution environment with zero-cost tool simulation
• Solvable training queries (at least 80% success on best-effort attempts)
• Tool APIs with consistent interfaces and failure modes
• Reward functions computable from trajectory completion

When to Reduce Complexity

• Model <1.5B: Consider smaller tool sets or shorter horizons; RL becomes less sample-efficient
• Tool failure >10%: Environment instability dominates; improve tool reliability before RL
• Sparse reward signal: Increase curriculum steps for smaller models; add task-level signal design
• Over 2K training samples: Performance plateaus; reallocate resources elsewhere