| name | sample-more-think-less-gfpo |
| title | Sample More Think Less: Group Filtered Policy Optimization |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2508.09726 |
| keywords | ["policy-optimization","rl-training","token-efficiency","length-penalization","inference-optimization"] |
| description | Group Filtered Policy Optimization (GFPO) reduces inference-time computation by sampling larger groups during training and filtering responses based on length and token efficiency to teach models efficient reasoning. |
Sample More Think Less: Group Filtered Policy Optimization
Core Concept
Group Filtered Policy Optimization (GFPO) addresses a critical problem in RL-trained language models: length inflation, where models generate unnecessarily verbose outputs during inference. Rather than optimizing purely for correctness, GFPO trades training-time computation for inference efficiency by sampling multiple candidate responses and filtering them based on efficiency metrics.
The key insight is that larger sample groups at training time enable models to learn more efficient reasoning patterns, reducing computation needed at test time.
Architecture Overview
- Group Sampling: Sample multiple candidate responses (groups) per problem during RL training
- Dual Filtering Criteria: Evaluate each response on two metrics: (1) output length, (2) reward-per-token efficiency ratio
- Efficiency-Aware Selection: Train only on responses that achieve good performance without excessive verbosity
- Adaptive Variant: Dynamically allocate more samples to harder problems for targeted efficiency gains
- Inference Speedup: Resulting models require fewer tokens at test time while maintaining accuracy
Implementation Steps
1. Prepare RL Training Setup
Start with a base language model and define your reward function (e.g., correctness on reasoning tasks). Set up PPO or similar RL algorithm infrastructure.
from trl import PPOTrainer
from transformers import AutoTokenizer, AutoModelForCausalLM
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")
def compute_reward(response, reference_answer):
"""Reward function: 1.0 for correct, 0.0 for incorrect"""
is_correct = check_correctness(response, reference_answer)
return 1.0 if is_correct else 0.0
2. Sample Groups and Compute Metrics
For each training problem, generate multiple candidate responses. Compute both correctness reward and efficiency metrics for each.
