critique-grpo-reasoning

name	critique-grpo-reasoning
title	Critique-GRPO: Advancing LLM Reasoning with Dual Feedback
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.03106
keywords	["reinforcement-learning","reasoning","feedback","critique","policy-optimization"]
description	Improve LLM reasoning by combining numerical and natural language critique feedback in online RL for policy refinement.

Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback

Core Concept

Standard reinforcement learning for LLM reasoning using only numerical rewards plateaus in performance and fails to enable effective self-reflection. Critique-GRPO addresses these limitations by integrating both natural language critiques and scalar rewards for policy optimization. This dual-feedback approach enables models to learn from initial failures, refine responses via critique-guided self-improvement, and achieve consistent 15-21% Pass@1 improvements.

Architecture Overview

• Three-Step Framework: Initial response sampling, critique-guided self-refinement, online policy optimization
• Natural Language Feedback: Text-based critiques identifying reasoning failures enable verbal credit assignment
• Numerical Rewards: Scalar rewards combined with critiques for principled policy gradient updates
• Critique-Conditioned Refinement: In-context learning on question-response-critique triplets enables self-improvement
• Dual Objective: Train on both initial and refined responses with reward shaping that amplifies successful refinements
• Advantage: Maintains higher policy entropy and enables better exploration compared to numerical-only methods

Implementation

Step 1: Design Critique Generation

import torch
from typing import Dict, List, Tuple

class CritiqueGenerator:
    """Generate natural language critiques of reasoning responses"""

    def __init__(self, critique_model_name='gpt-4'):
        self.critique_model = load_model(critique_model_name)

    def generate_critique(self, question: str,
                         response: str,
                         ground_truth: str = None) -> Tuple[str, float]:
        """
        Generate detailed critique of response.
        Also provide numerical confidence score.
        """

        critique_prompt = f"""Evaluate this reasoning response:

Question: {question}

Response: {response}

{f"Ground Truth: {ground_truth}" if ground_truth else ""}

Provide a detailed critique covering:
1. Correctness: Is the final answer correct?
2. Reasoning: Are the logical steps sound?
3. Clarity: Is the explanation clear?
4. Completeness: Are there missing steps?
5. Efficiency: Is there a simpler approach?

Also provide a confidence score (0.0-1.0) indicating how likely this response is correct."""

        critique_response = self.critique_model.generate(critique_prompt)

        # Parse response
        critique_text = critique_response
        confidence = self.extract_confidence_score(critique_response)

        return critique_text, confidence

    def () -> :
        

        
         re

        
         = re.search(, critique_text.lower())
         :
             (.group()) / 

        
         = re.search(, critique_text.lower())
         :
             (.group())

        
         

     () -> :
        

        error_types = {
            : [, , ],
            : [, , ],
            : [, , ],
            : [, , ],
            : [, ],
        }

        critique_lower = critique.lower()

         error_type, keywords  error_types.items():
             (kw  critique_lower  kw  keywords):
                 error_type

         

Step 2: Implement Critique-Guided Self-Refinement

class CritiqueGuidedRefinement:
    """Enable models to improve responses based on critiques"""

    def __init__(self, model):
        self.model = model
        self.refinement_history = []

    def refine_response(self, question: str,
                       initial_response: str,
                       critique: str,
                       max_refinement_attempts: int = 3) -> List[str]:
        """
        Use critique to guide response refinement.
        Create in-context learning examples showing how to improve.
        """

        refinement_attempts = [initial_response]

        for attempt in range(max_refinement_attempts):
            # Build refinement prompt with few-shot examples
            refinement_prompt = self.build_refinement_prompt(
                question, initial_response, critique, attempt
            )

            # Generate refined response
            refined = self.model.generate(refinement_prompt)

            refinement_attempts.append(refined)

            # Stop if refinement converges
            if self.has_converged(refinement_attempts[-2:]):
                break

        return refinement_attempts

    def build_refinement_prompt(self, question: str,
                               current_response: str,
                               critique: str,
                               attempt: ) -> :
        

        few_shot_examples = .get_refinement_examples()

        prompt = 

         prompt

     () -> :
        

        examples = 

         examples

     () -> :
        

         (recent_attempts) < :
             

        
        similarity = .compute_similarity(
            recent_attempts[-],
            recent_attempts[-]
        )

         similarity > similarity_threshold

     () -> :
        

        words1 = (text1.lower().split())
        words2 = (text2.lower().split())

        intersection = (words1 & words2)
        union = (words1 | words2)

         intersection / union  union >   

Step 3: Implement Critique-GRPO Training Loop

class CritiqueGRPO:
    """GRPO training with dual numerical and natural language feedback"""

    def __init__(self, model, critique_generator: CritiqueGenerator,
                 refinement_module: CritiqueGuidedRefinement):
        self.model = model
        self.critique_gen = critique_generator
        self.refiner = refinement_module
        self.optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)

    def training_step(self, batch: Dict) -> Dict[str, float]:
        """
        Single training step combining initial and refined responses.

        Args:
            batch: {
                'questions': List[str],
                'ground_truths': List[str],
            }

        Returns:
            losses: {
                'initial_loss': float,
                'refinement_loss': float,
                'total_loss': float,
            }
        """

        questions = batch['questions']
        ground_truths = batch['ground_truths']

        total_loss = 0
        all_losses = {}

        for question, ground_truth in zip(questions, ground_truths):
            # Step 1: Initial response sampling
            initial_response = self.model.generate(question, temperature=1.0)

            # Step 2: Generate critique and reward
            critique, confidence = self.critique_gen.generate_critique(
                question, initial_response, ground_truth
            )

            # Compute numerical reward
            is_correct = self.check_correctness(initial_response, ground_truth)
            initial_reward = (is_correct)

            
            refined_responses = .refiner.refine_response(
                question, initial_response, critique
            )
            refined_response = refined_responses[-]  

            refined_is_correct = .check_correctness(
                refined_response, ground_truth
            )
            refined_reward = (refined_is_correct)

            
            initial_log_prob = .model.log_probability(
                question, initial_response
            )

            refined_log_prob = .model.log_probability(
                question, refined_response
            )

            
            advantage = refined_reward - initial_reward

            
            pg_loss = 
             initial_reward < refined_reward:
                
                pg_loss -= refined_log_prob * (refined_reward - ) * 
            :
                
                pg_loss -= initial_log_prob * (initial_reward - )

            total_loss += pg_loss
            all_losses[] = pg_loss.item()

        
        average_loss = total_loss / (questions)

        
        .optimizer.zero_grad()
        average_loss.backward()
        torch.nn.utils.clip_grad_norm_(.model.parameters(), )
        .optimizer.step()

        all_losses[] = average_loss.item()
        all_losses[] = np.mean([
            (.check_correctness(
                .model.generate(q), gt
            ))
             q, gt  (questions[:], ground_truths[:])
        ])

         all_losses

     () -> :
        

        
        predicted = .extract_answer(response)
        expected = .extract_answer(ground_truth)

         predicted == expected

     () -> :
        

         re

        
         = re.search(,
                         text, re.IGNORECASE)

         :
             .group().lower()

         text.strip().lower()

Step 4: Integrate with Training Pipeline

class CritiqueGRPOTrainer:
    """Full training pipeline with dual feedback"""

    def __init__(self, model, num_epochs: int = 5):
        self.model = model
        self.num_epochs = num_epochs

        self.critique_gen = CritiqueGenerator()
        self.refiner = CritiqueGuidedRefinement(model)
        self.grpo = CritiqueGRPO(model, self.critique_gen, self.refiner)

    def train(self, train_questions: List[str],
             train_answers: List[str],
             val_questions: List[str],
             val_answers: List[str]) -> Dict:
        """Train with critiques and refinement"""

        history = {
            'train_losses': [],
            'val_pass_at_1': [],
            'val_pass_at_4': [],
        }

        for epoch in range(self.num_epochs):
            print(f"\n=== Epoch {epoch + 1}/{self.num_epochs} ===")

            # Training step
            epoch_losses = []

            for question, answer in zip(train_questions, train_answers):
                batch = {
                    : [question],
                    : [answer],
                }

                losses = .grpo.training_step(batch)
                epoch_losses.append(losses[])

            avg_train_loss = np.mean(epoch_losses)
            history[].append(avg_train_loss)

            ()

            
            pass_at_1 = .evaluate_pass_at_k(
                val_questions, val_answers, k=
            )
            pass_at_4 = .evaluate_pass_at_k(
                val_questions, val_answers, k=
            )

            history[].append(pass_at_1)
            history[].append(pass_at_4)

            ()
            ()

         history

     () -> :
        

        successes = 

         question, answer  (questions, answers):
            
            samples = [
                .model.generate(question, temperature=)
                 _  (k)
            ]

            
             sample  samples:
                 .grpo.check_correctness(sample, answer):
                    successes += 
                    

         successes / (questions)  questions  

Practical Guidance

1. Dual Feedback is Key: Numerical rewards alone plateau. Adding critique enables 36.47% valid self-refinement rates compared to minimal gains from spontaneous reflection.

2. Critique Quality Matters: Use a capable critique model (GPT-4, Claude-3) to generate detailed, specific feedback. Poor critiques won't enable meaningful refinement.

3. Few-Shot Refinement Examples: Include concrete examples of how to address different error types. This dramatically improves refinement success rates.

4. Reward Shaping: Amplify successful refinements (1.5× weight) while penalizing failed ones. This maintains exploration while concentrating learning on improvements.

5. Policy Entropy: The dual-feedback approach maintains higher policy entropy than numerical-only training, enabling better exploration throughout training.

6. Evaluation Metric: Use Pass@1 and Pass@4, not just accuracy. Critique-GRPO improves both—initial generations improve, and refinements provide backup paths.

Reference

• Paper: Critique-GRPO (2506.03106)
• Key Improvement: +15.0-21.6% Pass@1 on Qwen models
• Architecture: Dual feedback (critique + reward) for policy optimization
• Innovation: In-context learning with critique-guided refinement