rl-verifiable-rewards

name	rl-verifiable-rewards
title	Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.14245
keywords	["reinforcement-learning","reasoning","verifiable-rewards","logic-prior","chain-of-thought"]
description	RLVR extends reasoning capabilities by proving answer-only rewards implicitly incentivize correct intermediate reasoning via the Logic Prior principle.

Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning

Core Concept

This work investigates whether reinforcement learning with verifiable rewards (RLVR) genuinely enhances reasoning or merely improves sampling efficiency. The key finding: RLVR can extend the reasoning boundary for both mathematical and coding tasks. The theoretical contribution shows why answer-only rewards work through the "Logic Prior" assumption—that correct reasoning chains more reliably produce correct answers. A novel CoT-Pass@K metric evaluates both answer and reasoning correctness.

Architecture Overview

• RLVR Framework: Apply RL optimization to LLM outputs using verifiable (answer-only) rewards
• Logic Prior Principle: Theoretical explanation that correct reasoning is correlated with correct answers
• CoT-Pass@K Metric: Evaluates both final answer and intermediate reasoning steps for correctness
• Training Dynamics: Track P(CA) [probability correct answer] and P(CC|CA) [probability correct chain given correct answer]
• Generalization Analysis: Demonstrate that RL creates valid reasoning improvements, not just sampling artifacts

Implementation

Step 1: Define Verifiable Reward Function

Create reward signal based on answer correctness alone:

import torch
from typing import List, Dict

class VerifiableRewardModel:
    """
    Verifiable rewards: only correct/incorrect answer classification.
    No intermediate step supervision—testing if RLVR alone improves reasoning.
    """
    def __init__(self, task_type='math'):
        self.task_type = task_type

    def compute_reward(self, answers: List[str],
                      ground_truth: List[str]) -> torch.Tensor:
        """
        Args:
            answers: [batch_size] generated answers
            ground_truth: [batch_size] correct answers

        Returns:
            rewards: [batch_size] binary (1.0 or -1.0)
        """
        rewards = []

        for ans, gt in zip(answers, ground_truth):
            # Simple answer matching (can be more sophisticated)
            is_correct = self._check_correctness(ans, gt)
            reward = 1.0 if is_correct else -1.0
            rewards.append(reward)

        return torch.tensor(rewards, dtype=torch.float32)

    def _check_correctness(self, answer: str, ground_truth: str) -> bool:
        """Check if answer matches ground truth"""
        # Strip whitespace, handle numeric answers
        answer_clean = answer.strip().lower()
        gt_clean = ground_truth.strip().lower()

        
         .task_type == :
             re
            answer_num = re.findall(, answer_clean)
            gt_num = re.findall(, gt_clean)

             answer_num  gt_num:
                 (answer_num[-]) == (gt_num[-])

         answer_clean == gt_clean

Step 2: Extract Chain-of-Thought from Generations

Parse reasoning chains for CoT-Pass@K evaluation:

class ChainOfThoughtExtractor:
    """
    Extracts reasoning chains and final answers from model outputs.
    """
    def __init__(self, reasoning_markers=None):
        if reasoning_markers is None:
            self.reasoning_markers = ['<|thinking|>', '```']
        else:
            self.reasoning_markers = reasoning_markers

    def extract_reasoning_and_answer(self, text: str) -> tuple:
        """
        Args:
            text: full model output

        Returns:
            (reasoning_chain, final_answer)
        """
        lines = text.split('\n')

        reasoning_lines = []
        answer_lines = []
        in_reasoning = False

        for line in lines:
            # Detect reasoning blocks
            if any(marker in line for marker in self.reasoning_markers):
                in_reasoning = not in_reasoning
                continue

            if in_reasoning:
                reasoning_lines.append(line)
            else:
                answer_lines.append(line)

        reasoning_chain = '\n'.join(reasoning_lines).strip()
        final_answer = '\n'.join(answer_lines).strip()

        return reasoning_chain, final_answer

    def extract_step_sequence() -> []:
        
        
         re
        steps = re.split(
            ,
            reasoning_chain
        )
        steps = [s.strip()  s  steps  s.strip()]

         steps

Step 3: Implement CoT-Pass@K Metric

Evaluate both answer and reasoning quality:

class CoTPassAtK:
    """
    Comprehensive metric: Pass@K on both answer and reasoning.
    """
    def __init__(self, judge_model=None):
        # Can use LLM-as-judge or rule-based verification
        self.judge_model = judge_model

    def compute_pass_at_k(self, generations: List[List[str]],
                          ground_truth: List[str],
                          verify_reasoning=True) -> Dict[str, float]:
        """
        Args:
            generations: [batch_size, k_samples] list of generated answers
            ground_truth: [batch_size] correct answers
            verify_reasoning: whether to verify reasoning quality

        Returns:
            metrics: dict with Pass@K values
        """
        batch_size = len(ground_truth)
        k = len(generations[0])

        # Pass@K for final answers
        answer_pass_at_k = 0
        reasoning_pass_at_k = 0

        extractor = ChainOfThoughtExtractor()
        verifier = VerifiableRewardModel()

        for i in range(batch_size):
            sample_gens = generations[i]

            # Check if any sample has correct answer
            answer_correct = False
            reasoning_correct = False

            for gen in sample_gens:
                reasoning, answer = extractor.extract_reasoning_and_answer(gen)

                # Verify answer
                if verifier._check_correctness(answer, ground_truth[i]):
                    answer_correct = 

                    
                     verify_reasoning  .judge_model:
                        reasoning_quality = ._verify_reasoning_quality(
                            reasoning, ground_truth[i]
                        )
                         reasoning_quality > :  
                            reasoning_correct = 

            answer_pass_at_k +=   answer_correct  
            reasoning_pass_at_k +=   reasoning_correct  

        
        metrics = {
            : answer_pass_at_k / batch_size,
            : (reasoning_pass_at_k / batch_size
                                  verify_reasoning  )
        }

         metrics

     () -> :
        
         .judge_model  :
             

        prompt = 

        score = .judge_model.score(prompt)
         score

Step 4: Implement GRPO Training with RLVR

Train using Group Relative Policy Optimization with verifiable rewards:

class GRPOTrainer:
    """
    Group Relative Policy Optimization trainer with verifiable rewards.
    Simpler than full PPO; groups samples for relative advantage estimation.
    """
    def __init__(self, model, reward_model, device='cuda'):
        self.model = model
        self.reward_model = reward_model
        self.device = device

    def compute_advantages_from_rewards(
        self,
        generations: torch.Tensor,  # [batch, num_groups, seq_len]
        log_probs: torch.Tensor,    # [batch, num_groups, seq_len]
        rewards: torch.Tensor       # [batch, num_groups]
    ) -> torch.Tensor:
        """
        Compute group-relative advantages.
        A(g) = r(g) - mean(r) for group g
        """
        batch_size, num_groups = rewards.shape

        # Compute mean reward across group
        group_mean_reward = rewards.mean(dim=1, keepdim=True)

        # Compute relative advantages
        advantages = rewards - group_mean_reward  # [batch, num_groups]

        # Expand for sequence dimension
        advantages_expanded = advantages.unsqueeze(-1).expand(
            batch_size, num_groups, log_probs.shape[-1]
        )  # [batch, num_groups, seq_len]

        return advantages_expanded

    def training_step(self, batch_prompts, batch_generations,
                      batch_ground_truth):
        """
        Single training step with verifiable rewards.
        """
        batch_size = len(batch_prompts)
        num_groups = len(batch_generations[0])

        
        log_probs_list = []

         prompts, gens  (batch_prompts, batch_generations):
             torch.no_grad():
                outputs = .model(prompts)
                lp = torch.log_softmax(outputs.logits, dim=-)
                log_probs_list.append(lp)

        
        all_rewards = []

         i  (batch_size):
            sample_rewards = .reward_model.compute_reward(
                batch_generations[i],
                [batch_ground_truth[i]] * num_groups
            )
            all_rewards.append(sample_rewards)

        rewards = torch.stack(all_rewards)  

        
        log_probs = torch.stack(log_probs_list)
        advantages = .compute_advantages_from_rewards(
            , log_probs, rewards
        )

        
        
        loss = -(advantages * log_probs).() / batch_size

         loss, rewards.mean().item()

Step 5: Analyze Training Dynamics

Track probability metrics throughout training:

def analyze_training_dynamics(model, train_dataloader, num_epochs=10):
    """
    Analyze P(CA) and P(CC|CA) throughout training.
    Shows if RLVR creates genuine reasoning improvements.

    Returns:
        history: dict of metrics over time
    """
    history = {
        'epoch': [],
        'p_correct_answer': [],
        'p_correct_chain_given_answer': [],
        'pass_at_k': []
    }

    extractor = ChainOfThinkingExtractor()
    verifier = VerifiableRewardModel()
    metric_computer = CoTPassAtK()

    for epoch in range(num_epochs):
        p_ca_samples = []
        p_cc_ca_samples = []
        pass_k_samples = []

        for batch in train_dataloader:
            # Generate from model
            generations = model.generate(
                batch['prompts'],
                num_return_sequences=4,
                max_length=512
            )

            # Extract answers and reasoning
            answers = []
            reasonings = []

            for gen in generations:
                reasoning, answer = extractor.extract_reasoning_and_answer(gen)
                answers.append(answer)
                reasonings.append(reasoning)

            # Compute P(CA): probability of correct answer
            correct_answers = [
                verifier._check_correctness(ans, gt)
                for ans, gt in zip(answers, batch['ground_truth'])
            ]
            p_ca = sum(correct_answers) / len(correct_answers)
            p_ca_samples.append(p_ca)

            # Compute P(CC|CA): probability of correct chain given answer
            if sum(correct_answers) > 0:
                correct_chains_given_correct = 
                total_correct_answers = (correct_answers)

                 i, is_correct_ans  (correct_answers):
                     is_correct_ans:
                        
                        is_correct_chain = (reasonings[i]) > 
                         is_correct_chain:
                            correct_chains_given_correct += 

                p_cc_ca = (correct_chains_given_correct /
                          total_correct_answers)
                p_cc_ca_samples.append(p_cc_ca)

            
            pass_k = metric_computer.compute_pass_at_k(
                generations, batch[],
                verify_reasoning=
            )
            pass_k_samples.append(pass_k[])

        
        history[].append(epoch)
        history[].append((p_ca_samples) /
                                          (p_ca_samples))
        history[].append(
            (p_cc_ca_samples) / (p_cc_ca_samples)
             p_cc_ca_samples  
        )
        history[].append((pass_k_samples) /
                                    (pass_k_samples))

     history

Practical Guidance

• Verifiable Tasks: Use domains where correctness is objectively verifiable (math, code) rather than subjective (writing)
• Reward Design: Binary rewards work; can add reward scaling by confidence if needed
• Sampling Strategy: Generate multiple samples (k=4-8) per prompt for robust Pass@K evaluation
• Judge Model: Use larger model as judge for verification (e.g., DeepSeek-R1 for math)
• Training Stability: Use GRPO over PPO for stability with group-relative advantages
• Evaluation Metrics: Always measure both answer correctness AND reasoning quality separately
• Baseline Comparison: Compare against supervised fine-tuning on reasoning to isolate RL contribution

Reference

Paper: arXiv:2506.14245 Key metrics: Extended reasoning boundaries for code/math; CoT-Pass@K confirms reasoning quality Logic Prior: P(correct answer | correct reasoning) > P(correct answer | incorrect reasoning) Related work: RLHF, verifiable rewards, chain-of-thought, policy optimization