clipping-free-policy-optimization

name	clipping-free-policy-optimization
title	Clipping-Free Policy Optimization for Large Language Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.22801
keywords	["Policy Optimization","RL for LLMs","Trust Region","PPO Alternatives","Alignment"]
description	Replace hard clipping in policy gradients with smooth quadratic penalties derived from Total Variation divergence constraints. Eliminates zero-gradient regions and training instability while maintaining stable policy evolution.

Clipping-Free Policy Optimization for Large Language Models

Policy gradient methods like PPO and GRPO use hard clipping to enforce trust regions, but this creates discontinuous gradients that cause zero-gradient regions, reward hacking, and training instability at scale. Models exploit superficial reward correlates like verbosity and degrade rapidly. CFPO replaces hard clipping with smooth convex penalties derived from Total Variation divergence constraints, providing everywhere-differentiable gradients that smoothly pull the policy toward the trust region without artificial boundaries.

The key insight is that TV divergence permits larger policy improvements than KL while remaining tractable, and smooth penalty-based enforcement is more stable than clipping.

Core Concept

CFPO replaces the clipped objective with a smooth penalty:

Traditional GRPO (with hard clip):

L_GRPO = r * min(ratio, clip(ratio, 1-ε, 1+ε))

CFPO (with smooth penalty):

L_CFPO = r * ratio - |ratio_advantage| / (2ε) * (ratio - 1)²

The penalty term is a quadratic function that smoothly constrains the ratio while providing everywhere-nonzero gradients. This avoids the cliff-like behavior of clipping while maintaining strong trust region enforcement.

Architecture Overview

• Advantage Computation: Standard advantage estimation (group-relative or RLOO)
• Policy Ratio Computation: log-probability ratio between current and reference policy
• Smooth Penalty Computation: Quadratic TV-constrained penalty
• Objective Combination: Reward signal + smooth penalty constraint
• Gradient-Based Update: Standard SGD/Adam on smooth objective
• Compatibility: Works with any advantage estimator (reasoning, alignment)

Implementation

The method involves computing policy ratios and applying the smooth penalty objective.

Compute policy ratios and advantages:

import torch
import torch.nn.functional as F

def compute_policy_ratios(logprobs_new, logprobs_ref, logprobs_old):
    """Compute probability ratios for policy gradient."""
    # Log probability differences
    log_ratio = logprobs_new - logprobs_old
    ratio = torch.exp(log_ratio)

     ratio, log_ratio

 ():
    
    
    deltas = rewards - values
    advantages = []
    gae = 

     delta  (deltas):
        gae = delta + gamma * gae_lambda * gae
        advantages.insert(, gae)

     torch.tensor(advantages)


 ():
    
    group_mean = rewards.mean()
    group_std = rewards.std() + 

    advantages = (rewards - group_mean) / group_std
     advantages