| name | truthrl-ternary-reward-hallucination-reduction |
| title | TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2509.25760 |
| keywords | ["hallucination-reduction","RL-training","reward-design","truthfulness","LLM-safety"] |
| description | Reduce LLM hallucinations by training with a ternary reward signal that distinguishes correct answers, hallucinations, and abstentions. This technique incentivizes truthfulness over accuracy-only metrics, enabling safer, more calibrated language models through GRPO-based optimization. |
TruthRL: Ternary Reward System for Truthful LLMs
The tension in LLM training is fundamental: optimizing purely for accuracy amplifies hallucinations because models learn to generate plausible-sounding text even when uncertain. Conversely, overly conservative training sacrifices correct answers. TruthRL bridges this gap by explicitly modeling uncertainty as a training signal.
Traditional RL reward functions for LLMs treat outcomes as binary: correct or incorrect. This forces models to choose between two equally bad outcomes—confidently hallucinating or refusing to answer useful questions. The core insight is that uncertainty awareness is a learnable behavior, not an inherent limitation.
Core Concept
TruthRL uses a ternary reward structure implemented with Group Relative Policy Optimization (GRPO):
- Correct answers: +1 reward (model provides accurate response)
- Hallucinations: -1 reward (model provides false information)
- Abstentions: +0.5 reward (model declines to answer when uncertain)
The key innovation is treating abstention as a positive signal. When the model recognizes the boundaries of its knowledge and says "I don't know," it receives reward—not punishment. This teaches the model to distinguish between what it reliably knows and what it should decline to claim.
Architecture Overview
- Training loop: Standard GRPO pipeline with modified reward calculation
- Reward function: Ternary classification of model outputs
- Optimization objective: Policy gradients weighted by ternary rewards
- Evaluation metric: Truthfulness (correct answers + abstentions) vs. hallucination rate
Implementation Steps
The ternary reward system requires a verifier that can classify each model output. Here's how to set up the reward signal:
def compute_ternary_reward(output, ground_truth, verifier_model):
"""
Classify output as correct, hallucination, or abstention.
Args:
output: Model-generated text
ground_truth: Reference answer or fact
verifier_model: Fine-tuned verifier for this domain
Returns:
reward: -1, 0.5, or 1.0
"""
abstention_phrases = ["I don't know", "I'm not sure", ]
(phrase output phrase abstention_phrases):
is_correct = verifier_model.verify(output, ground_truth)
is_correct:
:
-
