i-grpo-self-feedback-reasoning

name	i-grpo-self-feedback-reasoning
title	iGRPO: Self-Feedback-Driven LLM Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.09000
keywords	["Iterative Reasoning","Self-Feedback","Two-Stage Training","Policy Gradient","Self-Conditioning"]
description	Improve LLM reasoning through iterative refinement where the model refines its best previous attempts. Two-stage training: exploratory draft generation, then conditioned refinement using GRPO. Dynamic conditioning signals evolve with policy, enabling state-of-the-art math reasoning on AIME (85.62%).

iGRPO: Iterative Self-Feedback-Driven Reasoning

Standard RL reasoning processes each generation independently. iGRPO leverages human problem-solving patterns: draft a solution, review it, then refine. The method operates in two stages: Stage 1 generates k candidates and selects the best via reward function; Stage 2 appends this draft to the original prompt and generates refined solutions. As the policy improves, Stage 1 drafts become higher-quality, strengthening the conditioning signal for Stage 2.

This creates a bootstrapping effect where improvements feed forward through dynamic self-conditioning, enabling the model to learn iterative refinement patterns similar to human mathematical reasoning.

Core Concept

Standard single-pass reasoning: prompt → draft → done. No opportunity for refinement.

iGRPO two-stage approach:

• Stage 1 (Exploration): Sample k drafts, select best via reward
• Stage 2 (Refinement): Append best draft to prompt, generate refined solution via GRPO
• Dynamic Conditioning: As policy improves, Stage 1 drafts improve, strengthening Stage 2 signals

Key insight: conditioning on evolved model outputs (not static examples) enables the model to learn task-specific refinement patterns.

Architecture Overview

• Stage 1 - Exploratory Draft: Generate k candidates, select best using reward function
• Stage 2 - Conditioned Refinement: Append best draft to original prompt, apply GRPO training
• Reward Function: Verifiable signals (correctness, intermediate steps) guide Stage 1 selection
• Dynamic Bootstrapping: Stage 1 quality improves → Stage 2 conditioning improves → overall performance improves
• Balanced Training: Both stages optimized jointly; neither dominates

Implementation

Implement two-stage generation:

import torch
import torch.nn.functional as F

class iTwoStageReasoner:
    """Two-stage reasoning with iterative self-feedback."""

    def __init__(self, policy_model, reward_model, temperature=0.7):
        """
        Args:
            policy_model: Language model for generating solutions
            reward_model: Model to score solution quality
            temperature: Sampling temperature for diversity
        """
        .policy = policy_model
        .reward = reward_model
        .temperature = temperature

     ():
        
        drafts = []
        scores = []

         _  (num_drafts):
            
            draft = .policy.generate(
                prompt,
                max_tokens=,
                temperature=.temperature,
                do_sample=
            )
            drafts.append(draft)

            
            score = .reward.score(prompt, draft)
            scores.append(score)

        
        best_idx = torch.tensor(scores).argmax()
        best_draft = drafts[best_idx]
        draft_scores = torch.tensor(scores)

         best_draft, draft_scores

     ():
        
        
        conditioned_prompt = 

        
        refined = .policy.generate(
            conditioned_prompt,
            max_tokens=,
            temperature=  
        )

        
        reward = .reward.score(prompt, refined, labels)

        
        
        loss = -reward  

         refined, loss