| name | stapo-spurious-token-silencing |
| title | STAPO: Stabilizing RL for LLMs by Silencing Rare Spurious Tokens |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2602.15620 |
| keywords | ["reinforcement learning","LLM training stability","gradient normalization","policy optimization","mathematical reasoning"] |
| description | Stabilize RL training in LLMs by detecting and masking gradient contributions from spurious tokens that comprise 0.01% of output but cause disproportionate instability. Identifies tokens with low probability, low entropy, and positive advantage, then suppresses their gradients during optimization to maintain stable policy entropy and improve reasoning performance by 7%+ across model scales. |
STAPO: Suppressing Spurious Token Gradients for Stable LLM Policy Learning
Large language models trained with reinforcement learning on reasoning tasks often experience training instability characterized by erratic policy entropy and performance collapse. This instability stems from spurious tokens—statistically rare outputs that the model assigns high advantage values to, causing them to dominate gradient updates despite representing <0.01% of tokens. These overconfident mistakes can reverse optimization progress and prevent the model from discovering diverse, correct reasoning paths.
The challenge lies in identifying which tokens genuinely contribute harmful signal versus which represent valid exploratory behavior. Spurious tokens share three properties simultaneously: very low generation probability, very low entropy (the model is overconfident), and positive estimated advantage (incorrectly valued as beneficial). Standard RL algorithms treat all non-optimal tokens identically, allowing this small spurious fraction to monopolize optimization.
Core Concept
STAPO introduces the Silencing Spurious Tokens (S2T) mechanism, which operates as a selective gradient masking layer during policy optimization. Rather than redesigning the learning algorithm, S2T surgically removes gradient perturbations originating from problematic tokens, allowing the optimization process to focus on signal-bearing transitions.
The mechanism works by identifying tokens matching three criteria:
- Probability below threshold (typically τ_p = 0.002)
- Entropy below dynamic quantile (typically τ_h = 80th percentile)
- Positive advantage value
Only tokens satisfying all three conditions are masked, preserving legitimate exploratory signals.
Architecture Overview
- Detection Stage: For each generated token in a batch, compute generation probability p(y|x), entropy H(π), and advantage estimate A(y|x) from the training batch
- Masking Decision: Flag token for masking if p(y|x) < τ_p AND H(π) < quantile(H, τ_h) AND A > 0
- Loss Recalibration: Suppress gradient flow for flagged tokens by zeroing their loss contributions
- Normalization Adjustment: Recompute loss normalization over only non-masked tokens to avoid numerical instability
- Monitoring: Track percentage of masked tokens per batch; expect 0.01%-0.1% in healthy training
Implementation
The detection logic integrates into the loss computation phase of any policy gradient algorithm. For each training batch of (state, action, advantage) tuples, compute a binary mask before backpropagation:
def ():
probs = torch.nn.functional.softmax(logits, dim=-)
token_probs = probs.gather(-, targets.unsqueeze(-)).squeeze(-)
entropy = -(probs * torch.log(probs + )).(dim=-)
entropy_quantile = torch.quantile(entropy, )
low_prob = token_probs < prob_threshold
low_entropy = entropy < entropy_quantile
high_advantage = advantages >
spurious_mask = low_prob & low_entropy & high_advantage
spurious_mask
