truthrl-ternary-reward-hallucination-reduction

name	truthrl-ternary-reward-hallucination-reduction
title	TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2509.25760
keywords	["hallucination-reduction","RL-training","reward-design","truthfulness","LLM-safety"]
description	Reduce LLM hallucinations by training with a ternary reward signal that distinguishes correct answers, hallucinations, and abstentions. This technique incentivizes truthfulness over accuracy-only metrics, enabling safer, more calibrated language models through GRPO-based optimization.

TruthRL: Ternary Reward System for Truthful LLMs

The tension in LLM training is fundamental: optimizing purely for accuracy amplifies hallucinations because models learn to generate plausible-sounding text even when uncertain. Conversely, overly conservative training sacrifices correct answers. TruthRL bridges this gap by explicitly modeling uncertainty as a training signal.

Traditional RL reward functions for LLMs treat outcomes as binary: correct or incorrect. This forces models to choose between two equally bad outcomes—confidently hallucinating or refusing to answer useful questions. The core insight is that uncertainty awareness is a learnable behavior, not an inherent limitation.

Core Concept

TruthRL uses a ternary reward structure implemented with Group Relative Policy Optimization (GRPO):

• Correct answers: +1 reward (model provides accurate response)
• Hallucinations: -1 reward (model provides false information)
• Abstentions: +0.5 reward (model declines to answer when uncertain)

The key innovation is treating abstention as a positive signal. When the model recognizes the boundaries of its knowledge and says "I don't know," it receives reward—not punishment. This teaches the model to distinguish between what it reliably knows and what it should decline to claim.

Architecture Overview

• Training loop: Standard GRPO pipeline with modified reward calculation
• Reward function: Ternary classification of model outputs
• Optimization objective: Policy gradients weighted by ternary rewards
• Evaluation metric: Truthfulness (correct answers + abstentions) vs. hallucination rate

Implementation Steps

The ternary reward system requires a verifier that can classify each model output. Here's how to set up the reward signal:

def compute_ternary_reward(output, ground_truth, verifier_model):
    """
    Classify output as correct, hallucination, or abstention.

    Args:
        output: Model-generated text
        ground_truth: Reference answer or fact
        verifier_model: Fine-tuned verifier for this domain

    Returns:
        reward: -1, 0.5, or 1.0
    """
    # Check if model abstained (standard abstention phrases)
    abstention_phrases = ["I don't know", "I'm not sure", ]
     (phrase  output  phrase  abstention_phrases):
         

    
    is_correct = verifier_model.verify(output, ground_truth)

     is_correct:
           
    :
         -