confidence-rl-finetuning

name	confidence-rl-finetuning
title	Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.06395
keywords	["reinforcement learning","self-supervision","confidence reward","few-shot"]
description	Improve language model reasoning using only model confidence as reward signals, eliminating need for labels or preference models while achieving substantial gains with minimal data.

Confidence Is All You Need: Few-Shot RL Fine-Tuning

Core Concept

Rather than requiring external reward models, preference annotations, or label engineering, RLSC uses the model's own confidence as training signal. The insight: majority voting implicitly performs mode sharpening—concentrating probability mass on the most likely response. By directly optimizing confidence, you improve agreement and reasoning quality without annotations.

Architecture Overview

• Self-confidence objective: Maximize probability that independent samples agree
• Token-level probabilities: Track confidence across generation sequence
• Minimal data requirements: Only 16 samples per question, 10-20 training steps
• No labels needed: Entirely self-supervised approach
• Strong gains: +13.4% AIME, +21.2% MATH, +21.7% Minerva Math

Implementation

Step 1: Implement Confidence Scoring

Compute confidence from token-level probabilities:

class ConfidenceScorer:
    def __init__(self):
        self.epsilon = 1e-10

    def compute_token_confidence(self, logits: torch.Tensor) -> torch.Tensor:
        """Get confidence (max probability) for each token."""

        # Convert logits to probabilities
        probs = torch.softmax(logits, dim=-1)

        # Confidence is max probability
        confidence, _ = torch.max(probs, dim=-1)

        return confidence

    def compute_sequence_confidence(self,
                                   logits: torch.Tensor,
                                   input_length: int,
                                   mask: torch.Tensor = None
                                   ) -> torch.Tensor:
        """Get confidence for entire generation sequence."""

        # Get token-level confidence
        token_conf = self.compute_token_confidence(logits)

        # Remove input tokens
        generation_conf = token_conf[input_length:]

        # Mask padding if provided
        if mask is not None:
            generation_mask = mask[input_length:]
            generation_conf = generation_conf * generation_mask

        # Sequence confidence: product of token confidence
        # (Use log for numerical stability)
        log_conf = torch.log(generation_conf + self.epsilon)
        log_conf = log_conf.sum(dim=0)
        sequence_conf = torch.exp(log_conf)

         sequence_conf

     () -> torch.Tensor:
        

        
        all_probs = []
         logits  logits_list:
            probs = torch.softmax(logits[input_length:], dim=-)
            all_probs.append(probs)

        
        
        avg_probs = torch.stack(all_probs).mean(dim=)
        agreement = (avg_probs ** ).(dim=-)

         agreement.mean()

Step 2: Create Self-Confidence Reward Signal

Design reward from confidence without external labels:

class ConfidenceReward:
    def __init__(self, alpha: float = 0.1,
                 validity_checker = None):
        self.alpha = alpha
        self.validity_checker = validity_checker
        self.scorer = ConfidenceScorer()

    def compute_reward(self, completions: list,
                      logits_list: list,
                      input_length: int,
                      question: str = None
                      ) -> dict:
        """Compute reward based on confidence.

        R(y) = p(y|x) + α * validity_bonus(y)
        """

        rewards = []

        for completion, logits in zip(completions, logits_list):
            # Base reward: sequence confidence
            seq_conf = self.scorer.compute_sequence_confidence(
                logits,
                input_length
            )
            reward = seq_conf.item()

            # Optional validity bonus (e.g., format checking)
            if self.validity_checker is not None:
                is_valid = self.validity_checker(completion)
                reward += self.alpha * (1.0 if is_valid else -1.0)

            rewards.append(reward)

        return {
            "rewards": torch.tensor(rewards),
            : torch.tensor(rewards).mean(),
            : torch.tensor(rewards).()
        }

     () -> torch.Tensor:
        

        
        majority = ((completions),
                      key=completions.count)

        rewards = []
         completion, logits  (completions, logits_list):
             completion == majority:
                
                seq_conf = .scorer.compute_sequence_confidence(
                    logits,
                    input_length
                )
                rewards.append(seq_conf.item())
            :
                rewards.append()

         torch.tensor(rewards)

Step 3: Implement RLSC Training Loop

Train with confidence rewards and PPO:

class RLSCTrainer:
    def __init__(self, model,
                 num_samples: int = 16,
                 batch_size: int = 4):
        self.model = model
        self.num_samples = num_samples
        self.batch_size = batch_size
        self.optimizer = torch.optim.Adam(
            model.parameters(),
            lr=5e-6
        )
        self.reward_fn = ConfidenceReward(alpha=0.1)

    def generate_completions(self, question: str,
                            num_samples: int = 16
                            ) -> tuple:
        """Generate multiple completions."""

        input_ids = self.model.tokenize(question)
        input_length = input_ids.shape[-1]

        completions = []
        logits_list = []

        for _ in range(num_samples):
            output = self.model.generate(
                input_ids,
                max_new_tokens=256,
                output_hidden_states=False,
                return_dict_in_generate=True,
                output_scores=True
            )

            # Detokenize
            completion = self.model.decode(output.sequences[0])
            completions.append(completion)

            # Collect logits
            logits = torch.stack(output.scores)
            logits_list.append(logits)

         completions, input_ids, input_length, logits_list

     () -> torch.Tensor:
        

        
        rewards = (rewards - rewards.mean()) / (rewards.std() + )

        total_loss = 

         logits, reward  (logits_list, rewards):
            
            log_probs = torch.log_softmax(logits, dim=-)

            
            generation_log_probs = log_probs[input_length:]

            
            loss = -(generation_log_probs.mean() * reward)
            total_loss += loss

         total_loss / (logits_list)

     () -> :
        

        total_loss = 
        total_reward = 
        total_samples = 

         question  questions:
            
            completions, input_ids, input_length, logits_list = (
                .generate_completions(
                    question,
                    num_samples=.num_samples
                )
            )

            
            reward_dict = .reward_fn.compute_reward(
                completions,
                logits_list,
                input_length,
                question
            )
            rewards = reward_dict[]

            
            loss = .compute_policy_gradient(
                logits_list,
                rewards,
                input_length
            )

            
            .optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(
                .model.parameters(),
                max_norm=
            )
            .optimizer.step()

            total_loss += loss.item()
            total_reward += rewards.mean().item()
            total_samples += 

         {
            : total_loss / total_samples,
            : total_reward / total_samples,
            : total_samples
        }

Step 4: Evaluate Improvements

Assess reasoning gains without labels:

def evaluate_reasoning(model,
                      benchmark_questions: list,
                      reference_answers: list,
                      metric: str = "exact_match"
                      ) -> dict:
    """Evaluate on reasoning benchmarks."""

    correct = 0
    avg_confidence = 0.0
    answer_lengths = []

    for question, ref_answer in zip(benchmark_questions,
                                    reference_answers):
        # Generate completion
        completion = model.generate(
            model.tokenize(question),
            max_new_tokens=512
        )

        # Extract answer (assuming last line or similar)
        answer = extract_answer(completion)

        # Check correctness
        is_correct = evaluate_answer(answer, ref_answer, metric)
        if is_correct:
            correct += 1

        avg_confidence += extract_confidence(completion)
        answer_lengths.append(len(completion.split()))

    return {
        "accuracy": correct / len(benchmark_questions),
        "avg_confidence": avg_confidence / len(benchmark_questions),
        "avg_answer_length": sum(answer_lengths) / len(
            answer_lengths
        ),
        "num_eval": len(benchmark_questions)
    }

Practical Guidance

Sample Efficiency: 16 samples per question is sufficient due to confidence being a strong signal. More samples improve stability but require more compute.

Training Duration: 10-20 training steps on 16-sample batches is typical. Longer training may overfit to confidence metric.

Confidence as Prior: The confidence metric works because high-probability outputs tend to be correct. The method implicitly assumes model calibration.

Majority Voting Equivalence: Using agreement-based rewards is similar to majority voting but enables gradient-based optimization via confidence signals.

Emergent Conciseness: Models often produce shorter, more confident answers after RLSC training—an emergent behavior that improves interpretability.

When to Apply: Use RLSC for mathematical reasoning tasks where you want to improve performance without labels, or when external reward models are unavailable.

Reference

RLSC leverages a simple insight: confidence is sufficient training signal without external rewards. By maximizing F(p_θ) = E_y~p_θ[p_θ(y|x)], the method sharpens probability distributions toward high-confidence modes. Substantial gains (+13-21%) demonstrate confidence is a reliable proxy for reasoning quality in few-shot settings.