The Research Question
What question is this paper answering?
When large language models are fine-tuned via reinforcement learning (RLVR), how do they improve reasoning? Do all tokens change equally, or do gains concentrate at specific positions? Are new reasoning tokens invented, or are existing tokens reordered?
Why practitioners care:
Understanding RLVR mechanisms has implications for:
- Which parts of the generation to focus optimization effort on
- Whether sparse fine-tuning (updating only critical tokens) could be more efficient
- Whether RLVR teaches genuinely new reasoning or reshuffles existing knowledge
- How to diagnose failures (are specific decision points bottlenecked?)
What do people commonly believe?
Conventional assumption: RL fine-tuning distributes changes across the entire generation, gradually improving the overall reasoning process through global updates. Model learns to rewrite entire sequences better.
The Analytical Approach: Token-Level Divergence Analysis
Measurement technique:
The paper uses Kullback-Leibler (KL) divergence at the token level:
def token_level_divergence_analysis(base_model, rl_model, prompts):
"""
For each position in generated sequences:
1. Sample from base model, compute log probabilities P(token | context)
2. Sample from RL model, compute log probabilities Q(token | context)
3. Compute KL(P || Q) at each position
4. Identify positions with high divergence (RL-sensitive)
"""
divergences = []
for prompt in prompts:
base_sequence = base_model.generate(prompt)
rl_sequence = rl_model.generate(prompt)
for position in range(len(sequence)):
kl_div = kl_divergence(
base_model.logits(prompt, up_to=position),
rl_model.logits(prompt, up_to=position)
)
divergences.append((position, kl_div))
return divergences
Why this approach:
- Directly measures what changed in the model (probability distributions, not just outputs)
- Token-level granularity identifies which decisions are targeted by RL
- Captures both large changes (new tokens) and subtle reordering (existing tokens reweighted)
- Replicable: any model with logits can be analyzed this way
What it measures:
Token-level divergence quantifies how much the model's confidence changed for each token position. High divergence = RL substantially rewrote that decision. Low divergence = RL left that token choice largely unchanged.
Controls & Confounds: Isolating RL Effects
Key confound: Position bias
Early tokens in reasoning might naturally be more important (setup context) vs late tokens (conclusions). Divergence could correlate with position, not with RL impact.
Control: Compare divergence distribution to random shuffled positions. If high-divergence positions cluster at specific reasoning steps (not just early/late), it confirms RL targets particular decisions.
Key confound: Task difficulty
Harder reasoning tasks might show more distributed changes. Easier tasks might show sparse changes.
Control: Segment analysis by task difficulty. If sparsity pattern holds across difficulty levels, it suggests sparsity is fundamental, not task-dependent.
Key confound: Model size
Different models might exhibit different sparsity patterns (large models sparse vs small models distributed).
Control: Analyze multiple model sizes (7B to 70B parameters). The paper finds sparsity >83% across all scales, confirming it's not size-dependent.
Key confound: Sampling variability
Stochastic sampling could obscure true divergences. Multiple samples from the same prompt might show different divergence patterns.
Control: Report divergence statistics (mean, std, percentiles) and verify findings are robust to sampling variance.
Core Findings: Sparse, Targeted Refinement
Finding 1: >83% of positions exhibit near-zero divergence
Across reasoning tasks, more than 83% of token positions show minimal KL divergence between base and RL models.
Distribution of divergences:
- 83%+ positions: KL < 0.01 (near-zero change)
- 10-15% positions: KL = 0.01-0.1 (moderate change)
- 2-5% positions: KL > 0.1 (large change)
This reveals that RL fine-tuning is highly sparse—most token choices remain unchanged; refinement concentrates at critical decision points.
Finding 2: Cross-Sampling Validation (Forward Direction)
When researchers inject RL-selected tokens into otherwise base-model sequences:
- Inserting 1.5-7.8% of high-divergence RL tokens into base generation
- Result: Recovers full RL performance gains (near-complete improvement)
This demonstrates that a small fraction of strategically positioned tokens drives the entire performance improvement. The RL model doesn't need to rewrite most of the sequence—just specific bottleneck positions.
Finding 3: Cross-Sampling Validation (Reverse Direction)
When researchers revert RL tokens to base model choices:
- Reverting just 5-10% of high-divergence RL tokens back to base distribution
- Result: Collapses RL performance back to baseline
This asymmetric sensitivity proves that RL-selected tokens are both necessary and sufficient for gains. Remove them and reasoning breaks.
Finding 4: Mechanism is Probability Reordering, Not Invention
Examining token candidates at high-divergence positions:
- 80% average overlap in top-k tokens between base and RL models
- RL doesn't invent novel tokens at critical positions
- Instead: reorders existing plausible candidates higher
This reveals the mechanism: RL doesn't teach the model to generate unprecedented reasoning tokens. Instead, it learns to prioritize tokens that were already available but low-probability under the base model.
Surprising finding: SimpleRL exceeds 98% sparsity (near-zero divergence at 98% of positions) while maintaining strong performance. This suggests even extremely aggressive sparsity can preserve reasoning gains, challenging assumptions about distributed learning in RL fine-tuning.
Practitioner Implications
What should practitioners do differently given these findings?
1. Sparse Fine-Tuning is Viable
If only 1.5-7.8% of tokens drive gains, targeted fine-tuning could be more efficient than dense updates:
def sparse_rl_finetuning(base_model, reward_signal):
high_impact_positions = identify_divergent_positions(base_model, reward_signal)
for example in training_data:
loss = rl_loss(base_model.logits[high_impact_positions], reward_signal)
update_logits[high_impact_positions] += gradient(loss)
Benefit: Reduces compute and memory for fine-tuning; faster adaptation.
2. Debugging Reasoning Failures
If a reasoning model fails, token-level analysis identifies bottlenecks:
divergence_profile = token_level_divergence(failing_model, reference_model)
critical_positions = positions_with_divergence > threshold
Implication: Focus debugging effort on high-divergence positions, not entire sequences.
3. Data Selection & Curriculum Learning
High-divergence positions likely correspond to difficult reasoning steps. Use this for curriculum:
difficult_examples = [
ex for ex in data
if any(divergence[pos] > threshold for pos in ex)
]
4. Efficiency Gains via Selective Adaptation
When adapting to new tasks, only refine parameters affecting high-divergence positions:
- Identify which model layers contribute to high-divergence tokens
- Fine-tune only those layers
- Freeze rest of model
Implication: Faster, cheaper adaptation with less catastrophic forgetting.
Methodology You Can Reuse
Core analytical pattern: Perturbation-based localization
To test what's important in a system:
- Identify a measurement (divergence, activation magnitude, attention weight)
- Measure it across the system (all positions, all neurons, all heads)
- Identify outliers (high-divergence positions)
- Remove outliers and measure system performance
- Compare: was removal impactful? (If yes, you've found critical components)
Application to new questions:
- Are certain attention heads critical? Measure attention divergence across heads, ablate high-divergence heads
- Do early layers differ from late layers? Compare layer-wise divergence distributions
- Is there redundancy in the model? Try ablating multiple high-divergence components simultaneously; if performance is preserved, there's redundancy
To apply to your own models:
def analyze_importance_via_divergence(model, reference, data, dimension):
"""
dimension could be: token position, layer, attention head, neuron, etc.
"""
divergences = {}
for item in dimension:
div = measure_divergence(model, reference, data, item)
divergences[item] = div
important = [item for item, div in divergences.items() if div > threshold]
for item in important:
ablated_model = remove_component(model, item)
performance_drop = measure_performance_drop(ablated_model, reference)
return important, performance_drops
Limitations & Caveats
What this methodology can't tell us:
- Whether position-level changes are necessary vs sufficient. We know they're sufficient (cross-sampling shows 1.5-7.8% tokens recover gains). But does the model need those precise tokens, or would alternatives work?
- Whether divergence correlates with human-interpretable reasoning. High divergence at position i might reflect what humans would call "critical reasoning," but analysis doesn't prove causation.
- Whether sparsity holds across all tasks and models. Findings here are on reasoning benchmarks; other domains (summarization, translation) might show different patterns.
Caveats:
- Cross-sampling relies on inference-time substitution. At training time, RL loss gradients flow through all tokens, even if divergence is sparse. Fine-tuning might be more distributed than inference-time analysis suggests.
- Token position is task-dependent. High-divergence positions identified on reasoning tasks don't transfer to other domains.
Related Mechanistic Questions
Insights this analysis enables:
- Why does RL select certain tokens over alternatives? Follow-up analysis of what makes selected tokens preferable (semantic content, linguistic properties)
- Do different reasoning tasks have different critical positions? Comparative analysis of divergence patterns across task types
- How does model capacity affect sparsity? Do larger models show sparser or denser divergence?
- Does sparsity hold for other fine-tuning objectives (SFT, DPO)? Generalize methodology to test other RL variants
When to Apply Token-Level Analysis
| Scenario | Should I Use This? | Why / Why Not? |
|---|
| Optimizing model fine-tuning efficiency | Yes, absolutely | Identifies critical positions for targeted updates |
| Debugging reasoning failures | Yes, as first step | Locates which tokens are problematic |
| Understanding model behavior | Yes, with caveats | Identifies what changed but not why |
| Comparing fine-tuning methods | Yes, useful | Token divergence profiles reveal different optimization strategies |
| Improving interpretability | Partially | Locates critical decisions but doesn't explain them |
| Real-time model monitoring | Partially | Divergence analysis is compute-intensive; approximate heuristics might be needed |
Decision Table: When to Use Cross-Sampling
| Question | Use Cross-Sampling? | Why |
|---|
| "Which tokens matter?" | Yes | Direct evidence of necessity and sufficiency |
| "How to optimize fine-tuning?" | Yes | Guides selective update strategies |
| "Why did this token change?" | No | Cross-sampling shows what changed, not why |
| "How many tokens need to change?" | Yes | Empirical minimum identified via forward cross-sampling |
Reference
Paper: https://arxiv.org/abs/2603.22446
Related mechanistic analyses: Lottery Ticket Hypothesis (sparse subnetworks), Activation Patching (causal tracing), Attention Dissection
Related work: RLVR methodology papers, mechanistic interpretability in LLMs
