learning-reason-factuality

name	learning-reason-factuality
title	Learning to Reason for Factuality
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.05618
keywords	["factuality","hallucination-reduction","reward-design","reasoning-models"]
description	Multi-dimensional reward function combining factual precision, response detail, and answer relevance for online RL. Reduces hallucinations 23.1% while maintaining helpfulness and detail.

Learning to Reason for Factuality

Core Concept

Reasoning models prone to hallucinations when generating long-form factual content, yet traditional reward functions like FActScore cause reward hacking by reducing output quality. This paper proposes a multi-dimensional reward function that simultaneously optimizes for factual accuracy, response detail level, and answer relevance. Applied through online RL, the approach significantly reduces hallucinations while maintaining helpful, detailed responses.

Architecture Overview

• Multi-dimensional Reward Function: Balances accuracy, detail, and relevance
• Factual Precision Metric: Measures proportion of hallucination-free content
• Detail Level Assessment: Ensures sufficient response completeness
• Relevance Scoring: Verifies information pertains to query
• Online RL Training: Iteratively improves reasoning with nuanced rewards

Implementation Steps

Step 1: Design Multi-dimensional Reward Function

Create reward function balancing multiple objectives.

import torch
from typing import Tuple, Dict

class MultiDimensionalRewardFunction:
    """
    Compute rewards balancing factuality, detail, and relevance.
    """

    def __init__(self, fact_checker, relevance_scorer):
        self.fact_checker = fact_checker
        self.relevance_scorer = relevance_scorer

    def compute_reward(
        self,
        question: str,
        response: str,
        ground_truth: str = None
    ) -> Dict:
        """
        Compute multi-dimensional reward.

        Args:
            question: Original question
            response: Model's response
            ground_truth: Optional ground truth for comparison

        Returns:
            Dict with component rewards and total
        """
        # Component 1: Factual Precision
        factuality_score = self._compute_factuality(response, ground_truth)

        # Component 2: Detail Level
        detail_score = self._compute_detail_level(response)

        # Component 3: Answer Relevance
        relevance_score = self._compute_relevance(question, response)

        # Combine with careful weighting
        # Higher weight on factuality to prevent hallucinations
        total_reward = (
            0.5 * factuality_score +
            0.25 * detail_score +
            0.25 * relevance_score
        )

        return {
            "factuality": factuality_score,
            "detail": detail_score,
            : relevance_score,
            : total_reward
        }

     () -> :
        
        
        claims = ._extract_claims(response)

          claims:
               

        
        correct_claims = 

         claim  claims:
            
            is_factual = .fact_checker.verify(claim, ground_truth)

             is_factual:
                correct_claims += 

        factuality = correct_claims / (claims)

         factuality

     () -> :
        
        
        signals = []

        
        word_count = (response.split())
         word_count < :
            length_score = word_count /   
         word_count < :
            length_score =   
        :
            length_score = (,  - (word_count - ) / )  

        signals.append(length_score * )

        
        sentences = response.split()
        avg_sent_length = word_count / (sentences)  sentences  

         avg_sent_length < :
            complexity_score =   
         avg_sent_length < :
            complexity_score =   
        :
            complexity_score =   

        signals.append(complexity_score * )

        
        detail_keywords = [, , , , ]
        keyword_count = (  kw  detail_keywords  kw  response.lower())

        detail_score = (keyword_count / , )
        signals.append(detail_score * )

        detail = (signals)

         detail

     () -> :
        
        
        relevance = .relevance_scorer.compute_relevance(question, response)

         relevance

     () -> :
        
        
        sentences = text.split()

        claims = []
         sent  sentences:
             (verb  sent.lower()  verb  [, , , ]):
                claims.append(sent.strip())

         claims

Step 2: Implement Fact-Checking System

Create system to verify factuality of claims.

class FactChecker:
    """
    Verify factuality of model-generated claims.
    """

    def __init__(self, knowledge_base=None, external_api=None):
        self.knowledge_base = knowledge_base
        self.external_api = external_api

    def verify(self, claim: str, context: str = None) -> bool:
        """
        Verify whether claim is factual.

        Args:
            claim: Factual claim to verify
            context: Optional context

        Returns:
            True if claim is factually correct
        """
        # Try knowledge base first
        if self.knowledge_base:
            kb_result = self.knowledge_base.check_fact(claim)
            if kb_result is not None:
                return kb_result

        # Try external API (e.g., web search)
        if self.external_api:
            api_result = self.external_api.verify_fact(claim)
            if api_result is not None:
                return api_result

        # If no verification possible, assume correct
        return True

    def extract_and_verify(self, text: str) -> [, ]:
        
        claims = ._extract_claims(text)

        verification = {}
         claim  claims:
            verification[claim] = .verify(claim)

         verification

     () -> :
        
        
         re

        
        sentences = re.split(, text)

        claims = []
         sent  sentences:
            sent = sent.strip()
             ._is_factual_claim(sent):
                claims.append(sent)

         claims

     () -> :
        
        assertion_verbs = [, , , , , , , , ]

         (verb  sentence.lower()  verb  assertion_verbs)

Step 3: Implement Online RL Training with Multi-dimensional Rewards

Train reasoning model using multi-dimensional reward function.

class FactualityRLTrainer:
    """
    Train reasoning model for factuality using online RL.
    """

    def __init__(self, model, reward_fn):
        self.model = model
        self.reward_fn = reward_fn

    def train_step(self, question: str, ground_truth: str = None, use_old_response: str = None):
        """
        Single training step optimizing for factuality.

        Args:
            question: Question to answer
            ground_truth: Ground truth for reward
            use_old_response: Use old response for comparison

        Returns:
            Training metrics
        """
        # Generate response
        response = self.model.generate(question, max_length=500)

        # Compute multi-dimensional rewards
        rewards = self.reward_fn.compute_reward(question, response, ground_truth)

        # Compute log probabilities of generated response
        log_probs = self.model.get_logprobs_for_response(question, response)

        # REINFORCE update: scale gradients by rewards
        # Gradient = log_prob * reward
        policy_loss = -(log_probs * rewards["total"]).mean()

        # Backward pass
        policy_loss.backward()
        torch.nn.utils.clip_grad_norm_(self.model.parameters(), max_norm=1.0)

        self.model.optimizer.step()
        self.model.optimizer.zero_grad()

        return {
            "loss": policy_loss.item(),
            "factuality_reward": rewards[],
            : rewards[],
            : rewards[],
            : rewards[],
            : response
        }

     ():
        
        epoch_metrics = {
            : ,
            : ,
            : ,
            : 
        }

        num_batches = 

         i  (, (questions), batch_size):
            batch_questions = questions[i:i + batch_size]
            batch_truths = ground_truths[i:i + batch_size]

            batch_loss = 
            batch_factuality = 
            batch_detail = 
            batch_relevance = 

             question, ground_truth  (batch_questions, batch_truths):
                metrics = .train_step(question, ground_truth)

                batch_loss += metrics[]
                batch_factuality += metrics[]
                batch_detail += metrics[]
                batch_relevance += metrics[]

            batch_size_actual = (batch_questions)

            epoch_metrics[] += batch_loss / batch_size_actual
            epoch_metrics[] += batch_factuality / batch_size_actual
            epoch_metrics[] += batch_detail / batch_size_actual
            epoch_metrics[] += batch_relevance / batch_size_actual

            num_batches += 

        
         key  epoch_metrics:
            epoch_metrics[key] /= (num_batches, )

         epoch_metrics

Step 4: Evaluate Factuality Improvements

Create evaluation framework for factuality metrics.

def evaluate_factuality(model, test_questions: list, ground_truths: list) -> Dict:
    """
    Evaluate model on factuality metrics.

    Args:
        model: Trained model
        test_questions: Test questions
        ground_truths: Ground truth answers

    Returns:
        Evaluation metrics
    """
    metrics = {
        "hallucination_rate": 0,
        "avg_detail": 0,
        "avg_relevance": 0,
        "overall_quality": 0
    }

    fact_checker = FactChecker()
    reward_fn = MultiDimensionalRewardFunction(fact_checker, None)

    for question, truth in zip(test_questions, ground_truths):
        # Generate response
        response = model.generate(question)

        # Check for hallucinations
        verification = fact_checker.extract_and_verify(response)

        hallucinations = sum(1 for v in verification.values() if not v)
        total_claims = len(verification) if verification else 1

        hallucination_rate = hallucinations / total_claims

        # Compute rewards
        rewards = reward_fn.compute_reward(question, response, truth)

        metrics["hallucination_rate"] += hallucination_rate
        metrics["avg_detail"] += rewards["detail"]
        metrics["avg_relevance"] += rewards["relevance"]

    # Average over test set
    n = len(test_questions)
    metrics[] /= n
    metrics[] /= n
    metrics[] /= n
    metrics[] = (
        ( - metrics[]) *  +
        metrics[] *  +
        metrics[] * 
    )

     metrics

Practical Guidance

When to Use Learning to Reason for Factuality

• Long-form factual generation: Answering complex questions with multiple facts
• Hallucination-prone models: Reasoning models generating plausible but false content
• Balanced optimization: Need both accuracy and response quality
• Online learning scenarios: Continuously improving with feedback

When NOT to Use Learning to Reason for Factuality

• Closed-domain QA: Knowledge base complete and accurate
• Creative tasks: Penalizing hallucinations inappropriate for fiction/poetry
• Latency-sensitive: Fact-checking adds computational overhead
• Toxic content: Fact-checking system itself may have biases

Hyperparameter Recommendations

• Factuality weight: 0.5 (higher priority to prevent hallucinations)
• Detail weight: 0.25 (maintains responsiveness)
• Relevance weight: 0.25 (ensures answer to question)
• Hallucination threshold: Flag claims with <0.5 confidence
• Learning rate: 1e-5 (conservative for stability)

Key Insights

The critical insight is that single-objective reward functions (like FActScore) cause reward hacking—models reduce output length to appear more factual. By combining factuality with detail and relevance, the framework forces genuine accuracy improvements rather than gaming the metric. The multi-dimensional approach learns nuanced trade-offs between competing objectives.

Reference

Learning to Reason for Factuality (arXiv:2508.05618)

Proposes multi-dimensional reward function combining factual precision, response detail, and answer relevance. Reduces hallucinations by 23.1% while maintaining response quality through online RL on reasoning models.