f-grpo-divergence-alignment

name	f-grpo-divergence-alignment
title	f-GRPO and Beyond: Divergence-Based RL for General LLM Alignment
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.05946
keywords	["f-Divergence","RL Alignment","GRPO Generalization","Policy Improvement","Verifiable Rewards"]
description	Unify LLM alignment methods through f-divergence theory. f-GRPO extends GRPO to handle any divergence measure (KL, Jensen-Shannon, Hellinger), enabling tailored alignment objectives. f-HAL combines on-policy and off-policy preference learning to prevent reward hacking while maintaining safety alignment.

f-GRPO: Divergence-Based RL for LLM Alignment

Popular LLM alignment methods optimize divergence between aligned and unaligned distributions, yet lack a unified framework. f-GRPO grounds GRPO in f-divergence theory, enabling selection of divergence measures matching your alignment objectives. For verifiable rewards (correct/incorrect), f-GRPO concentrates probability on high-reward responses. For preference-based alignment, f-HAL balances on-policy exploration with off-policy preference learning to prevent reward hacking.

Core Concept

f-GRPO generalizes GRPO by parameterizing the divergence: min_π f(π || π_ref) where f is any f-divergence (KL, Jensen-Shannon, Hellinger, etc.). Different f-divergences yield different concentration behaviors:

• KL divergence: Punishes low-probability modes in π_ref; stays close to reference
• Jensen-Shannon: Symmetric; balances exploration and exploitation
• Hellinger: More aggressive concentration on high-reward modes
• Reverse KL: Avoids modes with low rewards in π_ref

For verifiable rewards: f-GRPO estimates divergence between above-average and below-average reward distributions. For preference-based: f-HAL uses on-policy RL (exploration) combined with off-policy preference signals (exploitation).

Architecture Overview

• f-Divergence Family: Choose divergence matching alignment goals (concentration vs. exploration)
• Reward-Based f-GRPO: For verifiable rewards (math, code); converges to max-reward policies
• f-HAL: Hybrid on-policy + off-policy for preference alignment; prevents reward hacking
• Monotonic Improvement: All variants guarantee reward improvement until convergence
• Theoretical Guarantees: Alignment consistency proofs for each divergence choice

Implementation

Implement f-GRPO for verifiable rewards:

import torch
import torch.nn.functional as F

def compute_f_divergence(p_logits, q_logits, divergence_type='kl'):
    """
    Compute f-divergence between two distributions.
    Args:
        p_logits: Logits from policy distribution [batch, vocab_size]
        q_logits: Logits from reference distribution [batch, vocab_size]
        divergence_type: 'kl', 'js', 'hellinger', etc.
    Returns:
        divergence: Scalar divergence value
    """
    p = F.softmax(p_logits, dim=-)
    q = F.softmax(q_logits, dim=-)

     divergence_type == :
        
         (p * (torch.log(p) - torch.log(q))).(dim=-).mean()

     divergence_type == :
        
        m =  * (p + q)
          * (p * (torch.log(p) - torch.log(m))).(dim=-).mean() + \
                * (q * (torch.log(q) - torch.log(m))).(dim=-).mean()

     divergence_type == :
        
         (torch.sqrt(p * q + )).(dim=-).mean()

    :
         ValueError()

 ():
    
    
    mean_reward = rewards.mean()
    above_avg = (rewards > mean_reward).()

    
    advantage = rewards - mean_reward

     advantage * divergence_strength

 ():
    
    
    divergence = compute_f_divergence(policy_logits, reference_logits, divergence_type)

    
    advantages = f_grpo_advantage(rewards, divergence_strength)

    
    log_probs = F.log_softmax(policy_logits, dim=-)
    policy_loss = -(log_probs.detach() * advantages.unsqueeze(-)).mean()

    
    entropy = -(log_probs * F.softmax(policy_logits, dim=-)).(dim=-).mean()

    
    loss = policy_loss + divergence + entropy_coef * entropy

     loss

 ():
    
    inputs, rewards = batch

    
    policy_logits = policy(inputs).logits
     torch.no_grad():
        ref_logits = reference_model(inputs).logits

    
    loss = f_grpo_loss(policy_logits, ref_logits, rewards, divergence_type=divergence_type)

    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

     loss.item()