| name | gdpo-multi-reward-optimization |
| title | GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2601.05242 |
| keywords | ["Reinforcement Learning","Multi-objective Optimization","LLM Alignment","Policy Optimization"] |
| description | Optimize language models against multiple reward signals simultaneously by decoupling reward normalization. GDPO prevents reward combination collapse that undermines training signal quality when aligning models to multiple human preferences like accuracy, safety, efficiency, and format compliance. |
When to Use This Skill
- Training LLMs to satisfy multiple objectives (tool calling accuracy + format compliance)
- Multi-reward RL settings where reward signals have different scales or difficulties
- Aligning models where early rewards (e.g., correctness) must be satisfied before optimizing secondary objectives (e.g., length constraints)
- Extending GRPO-based training to handle >2 reward objectives
When NOT to Use This Skill
- Single-objective reward optimization (use standard GRPO)
- When rewards are naturally on identical scales without collapse risk
- Scenarios with fully independent reward signals requiring no prioritization
Problem Summary
Applying Group Relative Policy Optimization (GRPO) directly to multiple rewards causes advantage value collapse: distinct reward combinations map to identical normalized advantages. For example, with two binary rewards, reward combinations (0,1), (0,2), and (1,2) produce identical advantages despite representing fundamentally different satisfaction levels. This collapses six distinct signal types into two advantage groups, degrading training efficiency and model performance.
Solution: GDPO Three-Step Algorithm
Decouple normalization at the reward level before aggregation:
normalized_rewards = []
for reward_signal in reward_signals:
mean = reward_signal.mean()
std = reward_signal.std()
norm_reward = (reward_signal - mean) / (std + eps)
normalized_rewards.append(norm_reward)
aggregated_advantage = sum(normalized_rewards)
final_advantage = (aggregated_advantage - aggregated_advantage.mean()) / (aggregated_advantage.std() + eps)
Instead of: A = norm(r₁ + r₂ + ... + rₙ)
Use: A = norm(norm(r₁) + norm(r₂) + ... + norm(rₙ))
Key Implementation Details
Handling Reward Difficulty Gaps:
When objectives differ substantially in achievability (e.g., correctness is harder than format compliance), condition lower-priority rewards on higher-priority success:
length_reward = 1 if (length <= L and correctness == )
