two-grpo-contrastive-efficiency

name	two-grpo-contrastive-efficiency
title	2-GRPO: Contrastive Learning for Efficient Group Relative Policy Optimization
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.00977
keywords	["RLVR","GRPO","efficiency","contrastive-learning","DPO"]
description	Reduce GRPO training cost by 87.5% using only 2 rollouts instead of 16 while achieving 98.1% of baseline performance. Leverage the insight that GRPO's group mechanism serves contrastive learning rather than advantage estimation.

2-GRPO: Contrastive Learning for Efficient Group Relative Policy Optimization

This work reveals that Group Relative Policy Optimization (GRPO) functions fundamentally as implicit contrastive learning, connecting it to Direct Preference Optimization (DPO). By using only 2 rollouts instead of 16, models achieve 98.1% of full GRPO performance while reducing computation by 87.5%.

Core Architecture

• Contrastive interpretation: GRPO's group mechanism creates contrastive signals rather than accurate advantage estimates
• Minimal rollouts: 2-rollout configuration sufficient for effective contrastive pairs
• Theoretical equivalence: Mathematical connection to DPO framework
• Cross-domain validation: Improvements consistent across math, geometry, code reasoning

Implementation Steps

Modify standard GRPO training to use 2-rollout configuration:

# Setup 2-GRPO trainer with contrastive objective
from grpo_trainer import GRPOTrainer, ContrastiveGRPOConfig

config = ContrastiveGRPOConfig(
    num_rollouts=2,            # reduce from 16
    temperature=1.0,
    advantage_normalization=True,
    contrastive_scaling=1.0
)

trainer = GRPOTrainer(
    model=your_llm,
    config=config,
    algorithm="contrastive_grpo"
)

Execute training with 2-rollout sampling:

# Training loop using minimal rollout strategy
for step, batch in enumerate(dataloader):
    prompts = batch["prompt"]

    # Generate only 2 rollouts per prompt
    rollout_1 = model.generate(prompts, max_length=512, temperature=1.0)
    rollout_2 = model.generate(prompts, max_length=512, temperature=1.0)

    # Compute rewards (verification for reasoning tasks)
    rewards_1 = verifier.evaluate(prompts, rollout_1)
    rewards_2 = verifier.evaluate(prompts, rollout_2)

    # Contrastive learning objective (implicit in group comparison)
    advantages = rewards_1 - rewards_2

    
    loss = trainer.compute_contrastive_loss(
        rollouts=[rollout_1, rollout_2],
        advantages=advantages,
        log_probs_1=model.compute_log_probs(rollout_1),
        log_probs_2=model.compute_log_probs(rollout_2),
        kl_coefficient=
    )

    loss.backward()
    optimizer.step()
    optimizer.zero_grad()