gdpo-multi-reward-optimization

name	gdpo-multi-reward-optimization
title	GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.05242
keywords	["Reinforcement Learning","Multi-objective Optimization","LLM Alignment","Policy Optimization"]
description	Optimize language models against multiple reward signals simultaneously by decoupling reward normalization. GDPO prevents reward combination collapse that undermines training signal quality when aligning models to multiple human preferences like accuracy, safety, efficiency, and format compliance.

When to Use This Skill

• Training LLMs to satisfy multiple objectives (tool calling accuracy + format compliance)
• Multi-reward RL settings where reward signals have different scales or difficulties
• Aligning models where early rewards (e.g., correctness) must be satisfied before optimizing secondary objectives (e.g., length constraints)
• Extending GRPO-based training to handle >2 reward objectives

When NOT to Use This Skill

• Single-objective reward optimization (use standard GRPO)
• When rewards are naturally on identical scales without collapse risk
• Scenarios with fully independent reward signals requiring no prioritization

Problem Summary

Applying Group Relative Policy Optimization (GRPO) directly to multiple rewards causes advantage value collapse: distinct reward combinations map to identical normalized advantages. For example, with two binary rewards, reward combinations (0,1), (0,2), and (1,2) produce identical advantages despite representing fundamentally different satisfaction levels. This collapses six distinct signal types into two advantage groups, degrading training efficiency and model performance.

Solution: GDPO Three-Step Algorithm

Decouple normalization at the reward level before aggregation:

# Step 1: Normalize each reward independently using group statistics
normalized_rewards = []
for reward_signal in reward_signals:
    mean = reward_signal.mean()
    std = reward_signal.std()
    norm_reward = (reward_signal - mean) / (std + eps)
    normalized_rewards.append(norm_reward)

# Step 2: Aggregate normalized rewards
aggregated_advantage = sum(normalized_rewards)

# Step 3: Stabilize via batch-wise normalization
final_advantage = (aggregated_advantage - aggregated_advantage.mean()) / (aggregated_advantage.std() + eps)

Instead of: A = norm(r₁ + r₂ + ... + rₙ) Use: A = norm(norm(r₁) + norm(r₂) + ... + norm(rₙ))

Key Implementation Details

Handling Reward Difficulty Gaps: When objectives differ substantially in achievability (e.g., correctness is harder than format compliance), condition lower-priority rewards on higher-priority success:

# Force format compliance to activate only when correctness is met
length_reward = 1 if (length <= L and correctness == )