Analyzes how Reinforcement Learning from Verification Rewards (RLVR) improves reasoning by examining token-level probability distributions. Finds that >83% of token positions exhibit near-zero divergence—RL operates through sparse, targeted refinements. Cross-sampling experiments show 1.5-7.8% RL-selected tokens recover full gains, while reverting 5-10% of RL tokens collapses performance. Reveals that RL primarily reallocates probability within existing candidates (80% overlap in top-k tokens), not inventing novel tokens. Trigger: When analyzing LLM reasoning improvements, apply token-level divergence analysis and cross-sampling to identify which positions drive gains and whether changes are sparse or distributed.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Analyzes how Reinforcement Learning from Verification Rewards (RLVR) improves reasoning by examining token-level probability distributions. Finds that >83% of token positions exhibit near-zero divergence—RL operates through sparse, targeted refinements. Cross-sampling experiments show 1.5-7.8% RL-selected tokens recover full gains, while reverting 5-10% of RL tokens collapses performance. Reveals that RL primarily reallocates probability within existing candidates (80% overlap in top-k tokens), not inventing novel tokens. Trigger: When analyzing LLM reasoning improvements, apply token-level divergence analysis and cross-sampling to identify which positions drive gains and whether changes are sparse or distributed.
category
Mechanistic Analysis
The Research Question
What question is this paper answering?
When large language models are fine-tuned via reinforcement learning (RLVR), how do they improve reasoning? Do all tokens change equally, or do gains concentrate at specific positions? Are new reasoning tokens invented, or are existing tokens reordered?
Why practitioners care:
Understanding RLVR mechanisms has implications for:
Which parts of the generation to focus optimization effort on
Whether sparse fine-tuning (updating only critical tokens) could be more efficient
Whether RLVR teaches genuinely new reasoning or reshuffles existing knowledge
How to diagnose failures (are specific decision points bottlenecked?)
What do people commonly believe?
Conventional assumption: RL fine-tuning distributes changes across the entire generation, gradually improving the overall reasoning process through global updates. Model learns to rewrite entire sequences better.
The Analytical Approach: Token-Level Divergence Analysis
Measurement technique:
The paper uses Kullback-Leibler (KL) divergence at the token level:
# Measure how much each token's distribution changes under RLVRdeftoken_level_divergence_analysis(base_model, rl_model, prompts):
"""
For each position in generated sequences:
1. Sample from base model, compute log probabilities P(token | context)
2. Sample from RL model, compute log probabilities Q(token | context)
3. Compute KL(P || Q) at each position
4. Identify positions with high divergence (RL-sensitive)
"""
divergences = []
for prompt in prompts:
base_sequence = base_model.generate(prompt)
rl_sequence = rl_model.generate(prompt)
for position inrange(len(sequence)):
# Divergence at this position
kl_div = kl_divergence(
base_model.logits(prompt, up_to=position),
rl_model.logits(prompt, up_to=position)
)
divergences.append((position, kl_div))
return divergences
Why this approach:
Directly measures what changed in the model (probability distributions, not just outputs)
Token-level granularity identifies which decisions are targeted by RL
Captures both large changes (new tokens) and subtle reordering (existing tokens reweighted)
Replicable: any model with logits can be analyzed this way
What it measures:
Token-level divergence quantifies how much the model's confidence changed for each token position. High divergence = RL substantially rewrote that decision. Low divergence = RL left that token choice largely unchanged.
Controls & Confounds: Isolating RL Effects
Key confound: Position bias
Early tokens in reasoning might naturally be more important (setup context) vs late tokens (conclusions). Divergence could correlate with position, not with RL impact.
Control: Compare divergence distribution to random shuffled positions. If high-divergence positions cluster at specific reasoning steps (not just early/late), it confirms RL targets particular decisions.
Key confound: Task difficulty
Harder reasoning tasks might show more distributed changes. Easier tasks might show sparse changes.
Control: Segment analysis by task difficulty. If sparsity pattern holds across difficulty levels, it suggests sparsity is fundamental, not task-dependent.
Key confound: Model size
Different models might exhibit different sparsity patterns (large models sparse vs small models distributed).
Control: Analyze multiple model sizes (7B to 70B parameters). The paper finds sparsity >83% across all scales, confirming it's not size-dependent.
Key confound: Sampling variability
Stochastic sampling could obscure true divergences. Multiple samples from the same prompt might show different divergence patterns.
Control: Report divergence statistics (mean, std, percentiles) and verify findings are robust to sampling variance.
Core Findings: Sparse, Targeted Refinement
Finding 1: >83% of positions exhibit near-zero divergence
Across reasoning tasks, more than 83% of token positions show minimal KL divergence between base and RL models.
When researchers inject RL-selected tokens into otherwise base-model sequences:
Inserting 1.5-7.8% of high-divergence RL tokens into base generation
Result: Recovers full RL performance gains (near-complete improvement)
This demonstrates that a small fraction of strategically positioned tokens drives the entire performance improvement. The RL model doesn't need to rewrite most of the sequence—just specific bottleneck positions.
This reveals the mechanism: RL doesn't teach the model to generate unprecedented reasoning tokens. Instead, it learns to prioritize tokens that were already available but low-probability under the base model.
Surprising finding: SimpleRL exceeds 98% sparsity (near-zero divergence at 98% of positions) while maintaining strong performance. This suggests even extremely aggressive sparsity can preserve reasoning gains, challenging assumptions about distributed learning in RL fine-tuning.
Practitioner Implications
What should practitioners do differently given these findings?
1. Sparse Fine-Tuning is Viable
If only 1.5-7.8% of tokens drive gains, targeted fine-tuning could be more efficient than dense updates:
# Fine-tuning focused on high-impact token positions# rather than updating all model parametersdefsparse_rl_finetuning(base_model, reward_signal):
# Identify high-divergence positions (via divergence analysis)
high_impact_positions = identify_divergent_positions(base_model, reward_signal)
# Fine-tune only logits at those positions# (don't update rest of model)for example in training_data:
loss = rl_loss(base_model.logits[high_impact_positions], reward_signal)
update_logits[high_impact_positions] += gradient(loss)
Benefit: Reduces compute and memory for fine-tuning; faster adaptation.
2. Debugging Reasoning Failures
If a reasoning model fails, token-level analysis identifies bottlenecks:
# Diagnostic: which token positions predict failure?
divergence_profile = token_level_divergence(failing_model, reference_model)
critical_positions = positions_with_divergence > threshold
# These positions likely contain the reasoning errors# Can then inspect what tokens the model generates at those positions
Implication: Focus debugging effort on high-divergence positions, not entire sequences.
3. Data Selection & Curriculum Learning
High-divergence positions likely correspond to difficult reasoning steps. Use this for curriculum:
# Prioritize training examples where divergence is high# (model struggles at these positions)
difficult_examples = [
ex for ex in data
ifany(divergence[pos] > threshold for pos in ex)
]
# Focus training on difficult examples; easier examples are already learned
4. Efficiency Gains via Selective Adaptation
When adapting to new tasks, only refine parameters affecting high-divergence positions:
Identify which model layers contribute to high-divergence tokens
Fine-tune only those layers
Freeze rest of model
Implication: Faster, cheaper adaptation with less catastrophic forgetting.
Identify a measurement (divergence, activation magnitude, attention weight)
Measure it across the system (all positions, all neurons, all heads)
Identify outliers (high-divergence positions)
Remove outliers and measure system performance
Compare: was removal impactful? (If yes, you've found critical components)
Application to new questions:
Are certain attention heads critical? Measure attention divergence across heads, ablate high-divergence heads
Do early layers differ from late layers? Compare layer-wise divergence distributions
Is there redundancy in the model? Try ablating multiple high-divergence components simultaneously; if performance is preserved, there's redundancy
To apply to your own models:
# General pattern: localization via divergence analysisdefanalyze_importance_via_divergence(model, reference, data, dimension):
"""
dimension could be: token position, layer, attention head, neuron, etc.
"""
divergences = {}
for item in dimension:
div = measure_divergence(model, reference, data, item)
divergences[item] = div
# Identify important items (high divergence)
important = [item for item, div in divergences.items() if div > threshold]
# Validate by ablation: remove important items, measure impactfor item in important:
ablated_model = remove_component(model, item)
performance_drop = measure_performance_drop(ablated_model, reference)
# If performance_drop > 0, item is truly importantreturn important, performance_drops
Limitations & Caveats
What this methodology can't tell us:
Whether position-level changes are necessary vs sufficient. We know they're sufficient (cross-sampling shows 1.5-7.8% tokens recover gains). But does the model need those precise tokens, or would alternatives work?
Whether divergence correlates with human-interpretable reasoning. High divergence at position i might reflect what humans would call "critical reasoning," but analysis doesn't prove causation.
Whether sparsity holds across all tasks and models. Findings here are on reasoning benchmarks; other domains (summarization, translation) might show different patterns.
Caveats:
Cross-sampling relies on inference-time substitution. At training time, RL loss gradients flow through all tokens, even if divergence is sparse. Fine-tuning might be more distributed than inference-time analysis suggests.
Token position is task-dependent. High-divergence positions identified on reasoning tasks don't transfer to other domains.
Related Mechanistic Questions
Insights this analysis enables:
Why does RL select certain tokens over alternatives? Follow-up analysis of what makes selected tokens preferable (semantic content, linguistic properties)
Do different reasoning tasks have different critical positions? Comparative analysis of divergence patterns across task types
How does model capacity affect sparsity? Do larger models show sparser or denser divergence?
Does sparsity hold for other fine-tuning objectives (SFT, DPO)? Generalize methodology to test other RL variants
When to Apply Token-Level Analysis
Scenario
Should I Use This?
Why / Why Not?
Optimizing model fine-tuning efficiency
Yes, absolutely
Identifies critical positions for targeted updates
Debugging reasoning failures
Yes, as first step
Locates which tokens are problematic
Understanding model behavior
Yes, with caveats
Identifies what changed but not why
Comparing fine-tuning methods
Yes, useful
Token divergence profiles reveal different optimization strategies
Improving interpretability
Partially
Locates critical decisions but doesn't explain them
Real-time model monitoring
Partially
Divergence analysis is compute-intensive; approximate heuristics might be needed
Decision Table: When to Use Cross-Sampling
Question
Use Cross-Sampling?
Why
"Which tokens matter?"
Yes
Direct evidence of necessity and sufficiency
"How to optimize fine-tuning?"
Yes
Guides selective update strategies
"Why did this token change?"
No
Cross-sampling shows what changed, not why
"How many tokens need to change?"
Yes
Empirical minimum identified via forward cross-sampling
Reference
Paper: https://arxiv.org/abs/2603.22446
Related mechanistic analyses: Lottery Ticket Hypothesis (sparse subnetworks), Activation Patching (causal tracing), Attention Dissection
Related work: RLVR methodology papers, mechanistic interpretability in LLMs
