simpletir-multi-turn-tool-reasoning-rl

name	simpletir-multi-turn-tool-reasoning-rl
title	SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2509.02479
keywords	["reinforcement learning","tool integration","multi-turn reasoning","GRPO","trajectory filtering","distributional drift","gradient stability","mathematical reasoning"]
description	Train LLMs for multi-turn tool-integrated reasoning end-to-end using RL without supervised pretraining. SimpleTIR stabilizes training by filtering void turns (responses lacking code blocks or final answers) to prevent gradient explosion from distributional drift, enabling discovery of emergent reasoning patterns like self-correction and cross-validation on mathematical benchmarks.

SimpleTIR: Stable End-to-End RL for Multi-Turn Tool-Integrated Reasoning

Outcome

Enable base language models to learn multi-turn tool-integrated reasoning (code generation, execution, reflection, correction) end-to-end through reinforcement learning without requiring supervised fine-tuning or domain-specific pretraining, achieving state-of-the-art performance on mathematical reasoning benchmarks while maintaining stable training dynamics.

Problem Context

When training language models for multi-turn tool-integrated reasoning—where models generate code, execute it, observe results, and iterate—the training process becomes fundamentally unstable. The root cause lies in distributional drift: external tool outputs (execution results, error messages) exist far outside the model's pretraining distribution. When the model generates a response that leads to unexpected tool feedback, subsequent token generations fall into extremely low-probability regions of the learned distribution.

These low-probability tokens compound across turns. After three or four incorrect generations followed by tool feedback, the model assigns vanishingly small probabilities to valid continuations. During backpropagation, this produces catastrophic gradient norm explosions. Importance ratio terms explode, gradient magnitudes spike orders of magnitude above normal RL training, and the policy diverges into nonsensical behavior. The naive approach crashes entirely—gradient explosion destroys the learned policy.

Existing solutions impose external structure: supervised fine-tuning on human examples, careful curriculum learning, or complex reward shaping. These sacrifice the core promise of end-to-end learning: discovering reasoning strategies naturally from task feedback alone.

Core Concept

SimpleTIR introduces a minimal but surgically effective filtering mechanism targeting the symptom of distributional drift: "void turns"—individual model responses that contain neither a complete code block nor a final answer. These represent generation failures caused by the model assigning high probability to out-of-distribution token sequences.

By filtering entire trajectories containing void turns before computing policy losses, SimpleTIR prevents the high-magnitude gradients associated with problematic sequences while preserving credit assignment for successful reasoning chains. The mechanism is algorithm-agnostic, works as a plug-and-play wrapper around existing policy optimization methods, and requires no threshold tuning.

The theoretical grounding reveals that gradient norm depends inversely on token probabilities and is exacerbated by two compounding factors: unbounded importance ratios for low-probability tokens and sustained high gradient magnitudes when the policy assigns low probability to sampled sequences. Filtering void turns blocks the worst offenders without requiring complex annealing or uncertainty estimation.

Architecture Overview

SimpleTIR operates within a hierarchical MDP framework: