dppo-divergence-policy

name	dppo-divergence-policy
title	Rethinking the Trust Region in LLM Reinforcement Learning: DPPO
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.04879
keywords	["Reinforcement Learning","PPO","Policy Divergence","Trust Regions","LLM Training"]
description	Replace PPO's heuristic ratio-based clipping with Divergence Proximal Policy Optimization (DPPO) that directly constrains policy divergence using either Total Variation or KL, enabling lightweight approximations (Binary, Top-K) for vocabulary-scale computations while improving stability and efficiency.

DPPO: Direct Divergence Constraints for Stable LLM RL

PPO's probability ratio clipping creates problematic asymmetries in LLM training: low-probability tokens trigger aggressive clipping despite negligible distributional impact, while high-probability tokens shift substantially without penalty. DPPO replaces heuristic ratio clipping with direct divergence constraints, measuring actual policy divergence rather than noisy single-sample estimates. Lightweight approximations enable vocabulary-scale divergence computation.

Core Concept

The key insight is that PPO's mechanism is fundamentally misaligned with LLM training objectives. PPO uses token-level probability ratios as proxies for policy divergence, but these are noisy single-sample estimates that don't reflect true distributional differences. DPPO directly measures divergence (Total Variation or KL) between old and new policies, constraining whether the entire distribution has shifted too far rather than penalizing individual tokens.

Architecture Overview

• Direct Divergence Measurement: Compute actual policy divergence rather than probability ratios
• Lightweight Approximations: Binary (Bernoulli) and Top-K approximations for efficient divergence estimation
• Vocabulary-Scale Feasibility: Avoid expensive full-vocabulary KL computation via smart approximations
• Asymmetry-Free Updates: Uniform penalty structure regardless of token probability
• Joint Optimization: Train both policy and reward model with aligned divergence constraints

Implementation

Step 1: Understand PPO's Ratio-Based Clipping Problem

Analyze why probability ratios are problematic for policy divergence control.

def analyze_ppo_asymmetry(old_logits, new_logits, sampled_tokens):
    """
    Demonstrate PPO's asymmetry: ratio clipping poorly controls actual divergence.
    """
    batch_size, vocab_size = old_logits.shape
    
    # Compute probability ratios for sampled tokens
    old_probs = torch.softmax(old_logits, dim=-1)
    new_probs = torch.softmax(new_logits, dim=-1)
    
    # Get sampled token probabilities
    sampled_old_probs = old_probs[torch.arange(batch_size), sampled_tokens]
    sampled_new_probs = new_probs[torch.arange(batch_size), sampled_tokens]
    
    # Probability ratios
    prob_ratios = sampled_new_probs / (sampled_old_probs + )
    
    
    ()
    ()
    
    
    actual_kl = torch.nn.functional.kl_div(
        torch.log_softmax(new_logits, dim=-),
        torch.softmax(old_logits, dim=-),
        reduction=
    )
    
     prob_ratios, actual_kl