mr-align-meta-reasoning-factuality

name	mr-align-meta-reasoning-factuality
title	MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.24794
keywords	["Reasoning Alignment","Factuality","Meta-Reasoning","Preference Optimization","LLM Reasoning"]
description	Improve factuality in large reasoning models by analyzing reasoning state transitions and reweighting preference optimization signals, suppressing defective reasoning segments while amplifying patterns that lead to factual outputs.

Title: Align Reasoning Trajectories for Factual Correctness

Large reasoning models sometimes identify correct facts during thinking but fail to incorporate them into final answers. MR-Align detects this "reasoning-answer hit gap" by modeling reasoning as transitions between 15 distinct meta-cognitive states (framing, decomposition, verification, etc.). The framework reweights preference optimization based on state transition patterns, amplifying trajectories that consistently lead to factuality.

The key is treating reasoning as a navigable state space, not monolithic text.

Core Concept

State-Transition-Aware Preference Optimization:

• Meta-Reasoning States: 15 cognitive patterns grouped into 4 strategy categories
• Transition Matrices: Track probability of moving between states in factual vs. defective outputs
• Implicit Rewards: Reweight preference signals based on local-to-global transition probabilities
• Segment-Level Supervision: Annotate reasoning with fine-grained cognitive labels
• KTO Adaptation: Enhanced Kahneman-Tversky Optimization with transition-aware weighting

Architecture Overview

• Reasoning Taxonomy: 15 meta-reasoning states (framing, decomposition, chaining, verification, etc.)
• Segment Annotation: LLM-based coarse-to-fine labeling of reasoning text
• Transition Analysis: EM algorithm to estimate state transition matrices from paired examples
• Reward Computation: Token-level signals reweighted by transition probabilities
• Preference Optimization: KTO with segment-aware reward shaping

Implementation Steps

1. Define Meta-Reasoning Taxonomy

Create a taxonomy of cognitive patterns in reasoning.

class MetaReasoningTaxonomy:
    # 4 macro-strategies, 15 total cognitive states
    STRATEGIES = {
        'Meta-Cognitive Regulation': [
            'framing',          # Problem understanding/context setting
            'backtracking',      # Reconsidering previous steps
            'verification'       # Checking intermediate results
        ],
        : [
            ,     
            ,          
                 
        ],
        : [
            ,         
            ,         
                        
        ],
        : [
            ,       
            ,     
                     
        ]
    }


     ():
        
        prompt = 
        state = llm.generate(prompt).strip()
         state