latent-entropy-aware-decoding

name	latent-entropy-aware-decoding
title	Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.13366
keywords	["Hallucination Mitigation","Entropy-Aware Decoding","Multimodal Reasoning","Uncertainty Quantification","Continuous Embeddings"]
description	Reduce hallucinations in multimodal reasoning by detecting high-entropy (uncertain) states and switching to continuous latent embeddings instead of discrete tokens. Use prior-guided visual anchoring during uncertain phases to maintain grounding.

Latent Entropy-Aware Decoding: Uncertainty-Guided Multimodal Reasoning

Multimodal large reasoning models (MLRMs) frequently hallucinate during high-uncertainty reasoning phases, particularly at transition points where the model is uncertain about causal structure (words like "because," "however," "wait"). LEAD (Latent Entropy-Aware Decoding) mitigates this by monitoring the entropy of the model's latent representations. During high-uncertainty phases, rather than committing to discrete token selections, the model maintains superposed probability-weighted embeddings capturing multiple competing hypotheses. This allows the model to extract contextual information more reliably and reduces unfounded claims during uncertain reasoning.

The technique is plug-and-play and works with existing MLRMs by modifying the decoding strategy, not the underlying architecture.

Core Concept

LEAD operates through entropy-aware mode switching:

1. Entropy Monitoring — Track entropy of model's hidden states during generation
2. Mode Detection — Identify high-uncertainty phases (entropy > threshold)
3. Representation Switching — During uncertainty, use continuous superposed embeddings; during certainty, use discrete tokens
4. Visual Grounding — Apply prior-guided visual anchoring to reinforce factual grounding during uncertain phases

The key insight: at high-entropy states, the model is uncertain about which discrete token to emit, but the continuous probability distribution contains useful information. By leveraging that distribution, you avoid premature commitment to hallucinated tokens.

Architecture Overview

• Entropy Calculator — Computes per-layer entropy over logits during generation
• Uncertainty Detector — Identifies states where entropy exceeds task-specific threshold
• Continuous Embedding Generator — Creates weighted sum of token embeddings proportional to token probabilities
• Visual Anchor Injector — Reinforces visual-semantic alignment during uncertain phases
• Context Aggregator — Maintains latent trajectory across multiple candidate semantics
• Decoding Pipeline — Routes between discrete (low entropy) and continuous (high entropy) paths

Implementation Steps

Begin by computing entropy over model logits and detecting uncertainty phases during generation.

import torch
import torch.nn.functional as F
 numpy  np

 :
    

     ():
        .entropy_threshold = entropy_threshold
        .entropy_history = []

     () -> torch.Tensor:
        
        
        probs = F.softmax(logits, dim=-)

        
        entropy = -(probs * torch.log(probs + )).(dim=-)
         entropy

     () -> :
        
        entropy = .compute_entropy(logits)
        is_high_entropy = entropy > .entropy_threshold

        .entropy_history.append(entropy.item())
         is_high_entropy.item()

     () -> :
        
        uncertainties = [.is_uncertain(logits)  logits  logits_sequence]

        regions = []
        start = 
         i, is_unc  (uncertainties):
             is_unc  start  :
                start = i
              is_unc  start   :
                regions.append((start, i))
                start = 
         start   :
            regions.append((start, (uncertainties)))

         regions