メニュー
Fixes training instability in Group Relative Policy Optimization for discrete language models by replacing conditional clipping with strict importance ratio bounds and self-normalized advantages. Prevents gradient spikes and policy collapse.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/ADu2021/skillXiv --skill stable-drlコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
SOC 職業分類に基づく
SKILL.md を表示中
| name | stable-drl |
| title | Stabilizing Reinforcement Learning for Diffusion Language Models |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.06743 |
| keywords | ["Reinforcement Learning","Discrete Diffusion","Policy Optimization","Training Stability","GRPO"] |
| description | Fixes training instability in Group Relative Policy Optimization for discrete language models by replacing conditional clipping with strict importance ratio bounds and self-normalized advantages. Prevents gradient spikes and policy collapse. |
Group Relative Policy Optimization (GRPO) applied to discrete diffusion language models causes catastrophic training instability. The problem: discrete models require noisy estimation of importance ratios (since exact computation is intractable), and GRPO's conditional clipping bypasses bounds when advantages are negative, allowing noise-induced outliers to generate massive unclipped gradients. This creates a self-reinforcing loop: large updates increase policy divergence, which amplifies future ratio variance, causing exponential gradient growth.
StableDRL breaks this loop through two mechanisms: unconditional clipping that always bounds ratios regardless of advantage sign, and self-normalized advantages that constrain updates within the convex hull of per-sample gradients.
Standard GRPO gradient update is:
∇θ 𝒥 = 𝔼[1/|G| · Σ clip(ρ_i, 1-ε, 1+ε) · A_i · ∇_θ log π_θ(y_i | x)]
where importance ratios ρ_i = π_θ(y_i | x) / π_ref(y_i | x) are clipped only when advantages are positive. In discrete models with noisy ratio estimation, this creates two failure modes:
StableDRL eliminates this loop by:
New gradient:
∇θ 𝒥 = 𝔼[1/Σ clip(ρ_i) · Σ clip(ρ_j) · A_j · ∇_θ log π_θ(y_j | x)]
This ensures magnitude stays bounded independent of group-level fluctuations.
Modify GRPO's advantage computation and gradient step. Replace the standard clipping logic with unconditional bounds and self-normalization.
Standard GRPO vs StableDRL Comparison
import torch
import torch.nn as nn
# ============ Standard GRPO (unstable) ============
def standard_grpo_step(log_probs, log_probs_ref, rewards, advantages):
"""
log_probs: [batch_size, seq_len] - log probs from policy
log_probs_ref: [batch_size, seq_len] - log probs from reference model
rewards: [batch_size] - group-relative rewards
advantages: [batch_size] - group-relative advantages
PROBLEMATIC: conditional clipping allows outliers when A < 0
"""
# Estimate importance ratios
log_ratio = log_probs - log_probs_ref # [batch_size, seq_len]
ratio = torch.exp(log_ratio.sum(dim=1)) # [batch_size]
# CONDITIONAL clipping: only clips when advantage > 0
eps = 0.2
clipped_ratio = torch.where(
advantages > 0,
torch.clamp(ratio, 1 - eps, 1 + eps),
ratio # NO CLIPPING when advantage <= 0
)
# Gradient calculation
group_size = len(advantages)
weighted_advantages = clipped_ratio * advantages
loss = -weighted_advantages.mean() # No normalization by sum of clipped ratios
return loss
# ============ StableDRL (stable) ============
def stable_drl_step(log_probs, log_probs_ref, advantages):
"""
log_probs: [batch_size, seq_len] - log probs from policy
log_probs_ref: [batch_size, seq_len] - log probs from reference model
advantages: [batch_size] - computed advantages
SOLUTION: unconditional clipping + self-normalization
"""
# Estimate importance ratios
log_ratio = log_probs - log_probs_ref # [batch_size, seq_len]
# In discrete models, sum log-ratios (for tractable computation)
# Alternative: for continuous, use exp(log_ratio)
ratio = torch.exp(log_ratio.sum(dim=1)) # [batch_size]
# UNCONDITIONAL clipping: always bound ratios
eps = 0.2
clipped_ratio = torch.clamp(ratio, 1 - eps, 1 + eps)
# Self-normalization: divide by sum of clipped ratios instead of group size
# This bounds gradient magnitude by max clipped ratio value
ratio_sum = clipped_ratio.sum() # Scalar
weighted_advantages = clipped_ratio * advantages
loss = -(weighted_advantages / ratio_sum).sum()
return loss
Integration into RL Training Loop
class DiscreteLanguageModelWithStableDRL:
def __init__(self, model, ref_model, optimizer, eps=0.2):
self.model = model
self.ref_model = ref_model
self.optimizer = optimizer
self.eps = eps
def compute_advantages(self, rewards):
"""
Compute group-relative advantages.
Args:
rewards: [batch_size] scores for each sample
Returns:
advantages: [batch_size] centered and normalized
"""
mean = rewards.mean()
std = rewards.std() + 1e-8
advantages = (rewards - mean) / std
return advantages
def training_step(self, prompts, generated_sequences, reward_scores):
"""
Single training step with StableDRL.
Args:
prompts: list of input prompts
generated_sequences: [batch_size, seq_len] token indices
reward_scores: [batch_size] reward values from reward model
Returns:
loss: scalar loss value
"""
batch_size = len(generated_sequences)
# Forward pass: compute log probabilities
policy_outputs = self.model(
input_ids=prompts,
generated_tokens=generated_sequences
)
log_probs_policy = policy_outputs['log_probs'] # [batch_size, seq_len]
# Reference model inference (no_grad for efficiency)
with torch.no_grad():
ref_outputs = self.ref_model(
input_ids=prompts,
generated_tokens=generated_sequences
)
log_probs_ref = ref_outputs['log_probs'] # [batch_size, seq_len]
# Compute advantages
advantages = self.compute_advantages(reward_scores)
# Sum log-ratios (discrete model case)
log_ratio_sum = (log_probs_policy - log_probs_ref).sum(dim=1) # [batch_size]
ratio = torch.exp(log_ratio_sum)
# Unconditional clipping
clipped_ratio = torch.clamp(ratio, 1 - self.eps, 1 + self.eps)
# Self-normalized advantage weighting
ratio_sum = clipped_ratio.sum() # Scalar denominator
weighted_advantages = clipped_ratio * advantages
# Gradient: self-normalized advantages
loss = -(weighted_advantages / ratio_sum).sum()
# Backward pass
self.optimizer.zero_grad()
loss.backward()
# Optional: gradient clipping for additional stability
torch.nn.utils.clip_grad_norm_(self.model.parameters(), max_norm=1.0)
self.optimizer.step()
return loss.item()
def training_epoch(self, data_loader, num_epochs=1):
"""
Full training epoch.
Args:
data_loader: yields (prompts, generated_sequences, rewards)
num_epochs: number of passes over data
"""
losses = []
for epoch in range(num_epochs):
for prompts, sequences, rewards in data_loader:
loss = self.training_step(prompts, sequences, rewards)
losses.append(loss)
return {
'mean_loss': sum(losses) / len(losses),
'max_loss': max(losses),
'num_steps': len(losses)
}
Hyperparameters:
When to Apply:
When NOT to Apply:
Key Pitfalls:
Integration Notes: Drop-in replacement for standard GRPO gradient computation; requires reference model (frozen policy); works with any discrete language model architecture.
Evidence: Completely eliminates gradient spikes observed in standard GRPO on dLLMs; achieves stable training with comparable or better final performance; enables 2-3x longer stable training before collapse on challenging reward functions.
Reference: https://arxiv.org/abs/2603.06743