metacognition-as-reward

name

metacognition-as-reward

description

"Metacognition-as-Reward (MaR) — metacognition-inspired RL framework for LLM reasoning. Use when training LLMs to reason better through RL: (1) improving reasoning quality beyond final-answer correctness, (2) providing reward signals for intermediate reasoning behaviors, (3) replacing hand-crafted rubrics with general metacognitive dimensions, (4) training models for process-level reasoning quality.

Metacognition as Reward (MaR)

Core methodology from arXiv:2605.23384 (2026).

Core Concept

MaR extends RL reward signals beyond final-answer correctness to intermediate reasoning behaviors using two general metacognitive dimensions, eliminating the need for hand-crafted instance-specific rubrics:

1. Metacognitive Knowledge — Identifies task-relevant information without hand-crafted instance-specific rubrics
2. Metacognitive Regulation — Plans and adjusts the reasoning process to provide reward guidance beyond final-answer outcomes

Architecture

MaR scaffolds model rollouts into explicit metacognitive components and optimizes them with a trajectory-level reward over three signals:

R = λ₁ * R_knowledge + λ₂ * R_regulation + λ₃ * R_answer

where:
  R_knowledge  = Task knowledge coverage (metacognitive knowledge quality)
  R_regulation = Regulation fidelity (planning/adjustment quality)
  R_answer     = Final-answer correctness (standard RLVR signal)

Metacognitive Knowledge Dimension

• Prompts the model to identify what information from the task is relevant
• Does NOT require hand-crafted, instance-specific rubrics
• Coverage score measures completeness of relevant fact extraction

Metacognitive Regulation Dimension

• Prompts the model to plan and adjust its reasoning trajectory
• Regulation fidelity score measures:
  • How well the plan matches the actual reasoning steps
  • Whether the model adjusts when intermediate reasoning fails
  • Coherence of the reasoning structure

Training Procedure

1. Generate rollouts with explicit metacognitive components
2. Calculate trajectory-level reward: knowledge coverage + regulation fidelity + answer correctness
3. Optimize with policy gradient (e.g., GRPO, DAPO, PPO)

Key Results

• Up to 7.7% gain over base model on 22 benchmarks
• Up to 11.0% gain over vanilla DAPO
• Qwen3.5-9B + MaR narrows gap to frontier models, surpassing GPT-OSS-120B on average
• Process-level analysis shows substantial improvements in reasoning quality
• Generalizes to out-of-domain datasets
• Works with any RL training framework (GRPO, DAPO, PPO)

Implementation Pattern

def metacognitive_reward(trajectory_parts: dict) -> float:
    # 1. Extract metacognitive knowledge from model rollouts
    knowledge_coverage = evaluate_knowledge_coverage(
        extracted_facts=trajectory_parts['knowledge'],
        task_prompt=trajectory_parts['prompt']
    )

    # 2. Evaluate metacognitive regulation
    regulation_fidelity = evaluate_regulation(
        plan=trajectory_parts['plan'],
        execution=trajectory_parts['reasoning_steps']
    )

    # 3. Standard answer correctness
    answer_correct = evaluate_answer(
        trajectory_parts['answer'],
        ground_truth=trajectory_parts['correct_answer']
    )

    # 4. Combined trajectory-level reward
    return (λ_k * knowledge_coverage +
            λ_r * regulation_fidelity +
            λ_a * answer_correct)

Activation Keywords

Metacognition-as-Reward, MaR, metacognitive RL, LLM reasoning reward, process reward model, metacognitive knowledge, metacognitive regulation, trajectory-level reward, reasoning quality, RLVR reasoning, RL for reasoning, beyond final-answer reward, rubrics-as-reward alternative, reasoning process optimization