| name | stable-drl |
| title | Stabilizing Reinforcement Learning for Diffusion Language Models |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.06743 |
| keywords | ["Reinforcement Learning","Discrete Diffusion","Policy Optimization","Training Stability","GRPO"] |
| description | Fixes training instability in Group Relative Policy Optimization for discrete language models by replacing conditional clipping with strict importance ratio bounds and self-normalized advantages. Prevents gradient spikes and policy collapse. |
StableDRL: Fixing GRPO Training for Discrete Diffusion Language Models
Group Relative Policy Optimization (GRPO) applied to discrete diffusion language models causes catastrophic training instability. The problem: discrete models require noisy estimation of importance ratios (since exact computation is intractable), and GRPO's conditional clipping bypasses bounds when advantages are negative, allowing noise-induced outliers to generate massive unclipped gradients. This creates a self-reinforcing loop: large updates increase policy divergence, which amplifies future ratio variance, causing exponential gradient growth.
StableDRL breaks this loop through two mechanisms: unconditional clipping that always bounds ratios regardless of advantage sign, and self-normalized advantages that constrain updates within the convex hull of per-sample gradients.
Core Concept
Standard GRPO gradient update is:
∇θ 𝒥 = 𝔼[1/|G| · Σ clip(ρ_i, 1-ε, 1+ε) · A_i · ∇_θ log π_θ(y_i | x)]
where importance ratios ρ_i = π_θ(y_i | x) / π_ref(y_i | x) are clipped only when advantages are positive. In discrete models with noisy ratio estimation, this creates two failure modes:
- Variance in Ratios: Exponential mapping maps symmetric noise to long-tailed distributions with extreme outliers
- Gradient Spikes: Conditional clipping misses these outliers when advantage < 0, allowing unclipped gradients
- Policy Drift: Large updates increase KL divergence, amplifying future ratio variance
StableDRL eliminates this loop by:
- Unconditional Clipping: Bound ratios always: clip(ρ_i, 1-ε, 1+ε)
- Self-Normalization: Replace group-size normalization with sum of clipped ratios
New gradient:
∇θ 𝒥 = 𝔼[1/Σ clip(ρ_i) · Σ clip(ρ_j) · A_j · ∇_θ log π_θ(y_j | x)]
This ensures magnitude stays bounded independent of group-level fluctuations.
Architecture Overview
- Unconditional Importance Ratio Clipping: Apply bounds regardless of advantage sign
- Self-Normalized Advantage Weighting: Use ratio sum rather than group size for normalization
- Gradient Magnitude Guarantee: Theoretical proof that gradient norm is bounded by max clipped ratio value
- Policy Divergence Control: Maintain stable KL divergence despite noisy ratio estimates
Implementation Steps
Modify GRPO's advantage computation and gradient step. Replace the standard clipping logic with unconditional bounds and self-normalization.
Standard GRPO vs StableDRL Comparison
