f-grpo-focal-policy-optimization

name	f-grpo-focal-policy-optimization
title	F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.06717
keywords	["Policy Optimization","Diversity","Rare Solution Mining","RL Verifiable Rewards","Focal Weighting"]
description	Prevent policy collapse onto common solutions during reinforcement learning by applying difficulty-aware focal weighting to gradient contributions, maintaining diversity across solution modes while preserving performance.

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Problem Context

Reinforcement learning with verifiable rewards (RLVR) for language models exhibits distribution sharpening at intermediate group sizes. Policies improve pass@1 (best single solution) but degrade pass@256 (solution diversity), indicating concentration onto a narrow set of solutions. This erases rare but valuable solutions, reducing the value of the learned model for exploring multiple reasoning paths.

Core Concept

F-GRPO applies [focal weighting, difficulty-aware scaling, group-relative advantage] to address this phenomenon. The insight is that distribution collapse peaks at intermediate group sizes due to how probability mass redistributes. A single scalar multiplier per prompt down-weights high-success cases where concentration pressure peaks, preserving rare solutions without additional compute.

Architecture Overview

• Theoretical foundation: Closed-form tail-miss probability showing non-monotonic dependence on group size
• Focal weight function: g(x) = (1 − μ̂_pos(x))^γ scaling advantage contributions
• Group-relative optimization: Asymmetric pressure preserving diversity at specific group sizes
• Hyperparameter: Single γ (focal strength, typically 0.5-1.0) with intuitive effects
• Integration: Drop-in modification to GRPO, DAPO, CISPO without new networks

Implementation

Step 1: Estimate success probability per prompt

Compute empirical pass rate (probability of getting any correct solution) for each prompt during training.

# Estimate success probability
def estimate_success_probability(
    prompt_ids, group_results, smoothing_alpha=0.5
):
    """
    Estimate μ_pos(x): probability of success for each prompt.
    Smooth with alpha for stability on small groups.
    """
    success_probs = {}

    for prompt_id in set(prompt_ids):
        # Get all results for this prompt
        prompt_results = [
            r for r, p in zip(group_results, prompt_ids)
            if p == prompt_id
        ]

        # Count correct solutions (non-zero reward)
        num_correct = (  r  prompt_results  r > )
        total = (prompt_results)

        
        raw_prob = num_correct / (total, )
        smoothed_prob = (num_correct + smoothing_alpha) / (total +  * smoothing_alpha)

        success_probs[prompt_id] = smoothed_prob

     success_probs