entropy-ratio-clipping-stable-rl

name	entropy-ratio-clipping-stable-rl
title	Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.05591
keywords	["reinforcement learning","policy stability","entropy ratio","LLM post-training","training stability"]
description	Stabilize LLM post-training by constraining global distributional shifts in policy exploration. Entropy Ratio Clipping supplements local clipping mechanisms with global entropy constraints—essential when PPO alone produces unstable gradients and distribution shifts.

Overview

Entropy Ratio Clipping (ERC) addresses training instabilities in LLM post-training by introducing a global metric measuring relative policy exploration changes. While PPO-Clip helps locally, it overlooks the global distributional shift of actions. ERC provides bidirectional constraints on entropy ratio to stabilize policy updates globally.

When to Use

• Training language models with reinforcement learning post-training (DAPO, GPPO)
• Entropy fluctuations and unstable gradients appear during off-policy training
• Probability shifts of unsampled actions cause performance degradation
• Local clipping (PPO-Clip) alone is insufficient for stability

When NOT to Use

• Models already achieving stable training with PPO-Clip
• On-policy training scenarios without distribution shift concerns
• Applications where some degree of exploration variance is beneficial
• Tasks where entropy constraints reduce desired diversity

Core Technique

Entropy Ratio Clipping uses entropy ratio between successive policies as a constraint mechanism:

# Entropy Ratio Clipping for stable RL
class EntropyRatioClipping:
    def __init__(self, epsilon_lower=0.9, epsilon_upper=1.1):
        self.epsilon_lower = epsilon_lower
        self.epsilon_upper = epsilon_upper

    def compute_entropy_ratio(self, current_policy, previous_policy):
        """
        Measures relative exploration changes between policies.
        Quantifies distributional shifts at the aggregate level.
        """
        # Entropy of current policy
        current_entropy = self.compute_policy_entropy(current_policy)
        # Entropy of previous policy
        prev_entropy = self.compute_policy_entropy(previous_policy)

        # Entropy ratio
        ratio = current_entropy / (prev_entropy + 1e-8)
        return ratio

     ():
        
        
        clipped_ratio = torch.clamp(
            ratio,
            .epsilon_lower,
            .epsilon_upper
        )

        
        erc_penalty = torch.mean((ratio - clipped_ratio) ** )
        total_loss = base_loss +  * erc_penalty

         total_loss

     ():
        
        probs = torch.softmax(logits, dim=-)
        entropy = -torch.(probs * torch.log(probs + ), dim=-)
         entropy.mean()