| name | demystifying-rl-tool-agents |
| title | Demystifying RL for Long-Horizon Tool-Using Agents |
| version | 0.0.3 |
| engine | skillxiv-v0.0.3-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.21972 |
| keywords | ["Reinforcement Learning","Long-Horizon Agents","Tool Use","Reward Design","GRPO"] |
| description | Comprehensive recipe for RL-training tool-using agents spanning reward design, data synthesis, model scaling, and algorithm selection. Seven ranked findings: scale-dependent rewards (curriculum for 1.5B–3B; dense for 7B), semi-sparse 'Macro' rewards balance specialization/transfer, 1K-sample sweet spot with 4:3:3 difficulty mix. Achieves SOTA on TravelPlanner with smaller models than leading proprietary systems. |
Ranked Findings
1. Scale-Dependent Reward Design (Critical)
Different model scales require fundamentally different reward structures.
For smaller models (1.5B–3B):
- Curriculum rewards transition from dense to sparse signals
- Dense-only rewards fail to provide learning signal
- Sparse-only rewards cause catastrophic exploration collapse
For larger models (7B):
- Simple dense sum rewards outperform curriculum approaches
- Additional complexity adds overhead without benefit
- Larger models handle dense reward surfaces more robustly
2. Overly Dense Rewards Create Alignment Tax
Task-specific dense rewards maximize in-domain performance but significantly degrade out-of-domain generalization.
Trade-off spectrum:
- Full dense: Best in-domain performance; poorest transfer
- Semi-sparse "Macro": Balanced performance and transferability (recommended)
- Sparse: Best transfer; weak in-domain learning signal
Finding: Macro rewards (task-level rather than step-level) offer optimal generalization-specialization tradeoff.
3. Consistent Scaling Benefits
Model capacity improvements yield substantial gains across all conditions.
1.5B → 7B transition: Substantial improvements in success rates across all reward signals—no reward design fully compensates for model capacity limitations.
4. Data Sweet Spot at ~1K Samples (Critical)
Approximately 1,000 training examples with balanced difficulty provide optimal trade-offs.
Behavior:
- <1K: Insufficient signal; underfitting
- ~1K: Peak generalization; Goldilocks zone
- >2K: Over-optimization on training distribution; degraded generalization despite marginal in-domain gains
5. Balanced Difficulty Prevents Reward Sparsity
Mixed easy:medium:hard ratios (4:3:3) maintain sufficient reward signals while teaching complex constraint satisfaction.
Alternative distributions:
- Homogeneous easy: Too much positive feedback; agents never learn hard constraints
- Homogeneous hard: Catastrophic collapse—sparse reward signal
- 4:3:3 mix: Optimal—sustains learning throughout training
