sample-more-think-less-gfpo

name	sample-more-think-less-gfpo
title	Sample More Think Less: Group Filtered Policy Optimization
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.09726
keywords	["policy-optimization","rl-training","token-efficiency","length-penalization","inference-optimization"]
description	Group Filtered Policy Optimization (GFPO) reduces inference-time computation by sampling larger groups during training and filtering responses based on length and token efficiency to teach models efficient reasoning.

Sample More Think Less: Group Filtered Policy Optimization

Core Concept

Group Filtered Policy Optimization (GFPO) addresses a critical problem in RL-trained language models: length inflation, where models generate unnecessarily verbose outputs during inference. Rather than optimizing purely for correctness, GFPO trades training-time computation for inference efficiency by sampling multiple candidate responses and filtering them based on efficiency metrics.

The key insight is that larger sample groups at training time enable models to learn more efficient reasoning patterns, reducing computation needed at test time.

Architecture Overview

• Group Sampling: Sample multiple candidate responses (groups) per problem during RL training
• Dual Filtering Criteria: Evaluate each response on two metrics: (1) output length, (2) reward-per-token efficiency ratio
• Efficiency-Aware Selection: Train only on responses that achieve good performance without excessive verbosity
• Adaptive Variant: Dynamically allocate more samples to harder problems for targeted efficiency gains
• Inference Speedup: Resulting models require fewer tokens at test time while maintaining accuracy

Implementation Steps

1. Prepare RL Training Setup

Start with a base language model and define your reward function (e.g., correctness on reasoning tasks). Set up PPO or similar RL algorithm infrastructure.

# Hugging Face Transformers + TRL style setup
from trl import PPOTrainer
from transformers import AutoTokenizer, AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")

def compute_reward(response, reference_answer):
    """Reward function: 1.0 for correct, 0.0 for incorrect"""
    is_correct = check_correctness(response, reference_answer)
    return 1.0 if is_correct else 0.0

2. Sample Groups and Compute Metrics

For each training problem, generate multiple candidate responses. Compute both correctness reward and efficiency metrics for each.