truth-rl-ternary-reward-hallucination

name	truth-rl-ternary-reward-hallucination
title	TruthRL: Reducing Hallucinations via Ternary Reward Design
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2509.25760
keywords	["truthfulness","hallucination","RLVR","ternary-reward","verification"]
description	Train LLMs to reduce hallucinations by 28.9% using a ternary reward scheme that explicitly incentivizes abstention (+0) over false claims (-1) while rewarding correct answers (+1). Apply when improving factual reliability is critical and verification signals are available.

TruthRL: Reducing Hallucinations via Ternary Reward Design

TruthRL addresses a fundamental mismatch in accuracy-focused training: optimizing for correct answers inadvertently reinforces hallucinations when models guess incorrectly. By introducing ternary rewards distinguishing correct answers, hallucinations, and abstentions, models learn to admit uncertainty rather than fabricate information.

Core Architecture

• Ternary reward design: +1 (correct), -1 (hallucination/false claim), 0 (abstention)
• LLM-based verification: Uses another LLM to verify answer correctness
• GRPO training framework: Advantage-based optimization leveraging natural imbalance in rewards
• Knowledge boundary enforcement: Models learn what they don't know through reward asymmetry

Implementation Steps

Setup ternary reward verification system:

# Initialize verification-based reward model
from truth_rl import TruthRLTrainer, TernaryRewardModel

# Create verifier using stronger LLM
verifier = TernaryRewardModel(
    verifier_model="gpt-4o",  # or stronger open model
    correct_reward=1.0,
    hallucination_penalty=-1.0,
    abstention_reward=0.0
)

trainer = TruthRLTrainer(
    model=your_llm,
    verifier=verifier,
    algorithm="GRPO"
)

Execute RL training with ternary rewards:

# Training loop with question-answering tasks
for epoch in range(num_epochs):
    # Generate responses (allowing abstention)
    responses = model.generate(
        prompts=questions,
        max_length=512,
        temperature=1.0,
        top_p=0.9
    )

    # Compute ternary rewards
    rewards = verifier.evaluate(
        questions=questions,
        responses=responses,
        allow_abstention=True  # enables "I don't know" responses
    )

    # Update policy using advantage-based GRPO
    loss = trainer.compute_grpo_loss(
        responses=responses,
        rewards=rewards,
        advantage_normalization=
    )
    loss.backward()
    optimizer.step()