laser-self-reward

name	laser-self-reward
title	LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.14943
keywords	["self-rewarding","reinforcement-learning","reasoning","verification","llm-training"]
description	Compute reasoning rewards from the model's own next-token probability distribution at solution end. Integrates verification-based feedback into single model without separate evaluator, enabling efficient RL training with minimal overhead.

LaSeR: Self-Rewarding Reasoning through Last-Token Analysis

Separate reward models add computational overhead to RL training. LaSeR extracts reward signals from the model itself by analyzing the probability distribution over the next token at a solution's conclusion, aligning with verification quality without auxiliary models.

Core insight: at the end of a reasoning solution, the model's uncertainty about what comes next correlates with solution quality. By using this self-generated signal as reward, you get verification-based training in one model, reducing RL complexity while improving reasoning performance.

Core Concept

Last-Token Reward Signal: After generating a complete solution, compute the log-probability of a special verification token at position N+1. This probability difference from baseline quantifies solution quality.

Joint Optimization: Simultaneously optimize reasoning quality and self-verification through shared model parameters, enabling efficient RL without separate reward models.

Architecture Overview

• Solution Generator: Standard LLM generating reasoning traces
• Self-Verifier: Uses final position to compute confidence
• Reward Computer: Converts log-probabilities to training signal
• RL Optimizer: Updates both generation and verification jointly

Implementation Steps

Stage 1: Compute Last-Token Reward Signals

Extract rewards from final token probabilities:

import torch
import torch.nn as nn
from transformers import AutoModelForCausalLM, AutoTokenizer

class LastTokenRewardComputer:
    def __init__(self, model_name):
        self.model = AutoModelForCausalLM.from_pretrained(model_name)
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)

        # Special tokens for verification
        self.correct_token_id = self.tokenizer.encode("[CORRECT]")[0]
        .incorrect_token_id = .tokenizer.encode()[]

     ():
        

        batch_size = solution_tokens.shape[]
        rewards = []

         torch.no_grad():
            
            
            logits = .model(solution_tokens).logits

            
            final_logits = logits[:, -, :]  

            
            log_probs = torch.nn.functional.log_softmax(final_logits, dim=-)

            
            correct_log_prob = log_probs[:, .correct_token_id]
            incorrect_log_prob = log_probs[:, .incorrect_token_id]

            
            reward = (
                (correct_log_prob - incorrect_log_prob) / kl_coeff
            ).clamp(-, )  

         reward

     ():
        

        rewards = []

         solution  solutions_batch:
            solution_tokens = torch.tensor(
                .tokenizer.encode(solution)
            ).unsqueeze()

            reward = .compute_solution_reward(solution_tokens)
            rewards.append(reward.item())

         torch.tensor(rewards)