Skip to main content ホーム クリエイター adu2021 skillxiv latent-entropy-aware-decoding
latent-entropy-aware-decoding Reduce hallucinations in multimodal reasoning by detecting high-entropy (uncertain) states and switching to continuous latent embeddings instead of discrete tokens. Use prior-guided visual anchoring during uncertain phases to maintain grounding.
インストールへ移動 Skills Marketplace コミュニティが作成したAIスキルを発見・探索
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/ADu2021/skillXiv --skill latent-entropy-aware-decodingコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Zipをダウンロード ダウンロード中... このリポジトリの他の Skills meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
name latent-entropy-aware-decoding title Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2603.13366 keywords ["Hallucination Mitigation","Entropy-Aware Decoding","Multimodal Reasoning","Uncertainty Quantification","Continuous Embeddings"] description Reduce hallucinations in multimodal reasoning by detecting high-entropy (uncertain) states and switching to continuous latent embeddings instead of discrete tokens. Use prior-guided visual anchoring during uncertain phases to maintain grounding.
Latent Entropy-Aware Decoding: Uncertainty-Guided Multimodal Reasoning
Multimodal large reasoning models (MLRMs) frequently hallucinate during high-uncertainty reasoning phases, particularly at transition points where the model is uncertain about causal structure (words like "because," "however," "wait"). LEAD (Latent Entropy-Aware Decoding) mitigates this by monitoring the entropy of the model's latent representations. During high-uncertainty phases, rather than committing to discrete token selections, the model maintains superposed probability-weighted embeddings capturing multiple competing hypotheses. This allows the model to extract contextual information more reliably and reduces unfounded claims during uncertain reasoning.
The technique is plug-and-play and works with existing MLRMs by modifying the decoding strategy, not the underlying architecture.
Core Concept
LEAD operates through entropy-aware mode switching:
Entropy Monitoring — Track entropy of model's hidden states during generationMode Detection — Identify high-uncertainty phases (entropy > threshold)Representation Switching — During uncertainty, use continuous superposed embeddings; during certainty, use discrete tokensVisual Grounding — Apply prior-guided visual anchoring to reinforce factual grounding during uncertain phases
The key insight: at high-entropy states, the model is uncertain about which discrete token to emit, but the continuous probability distribution contains useful information. By leveraging that distribution, you avoid premature commitment to hallucinated tokens.
Architecture Overview
Entropy Calculator — Computes per-layer entropy over logits during generationUncertainty Detector — Identifies states where entropy exceeds task-specific thresholdContinuous Embedding Generator — Creates weighted sum of token embeddings proportional to token probabilitiesVisual Anchor Injector — Reinforces visual-semantic alignment during uncertain phasesContext Aggregator — Maintains latent trajectory across multiple candidate semanticsDecoding Pipeline — Routes between discrete (low entropy) and continuous (high entropy) paths
Implementation Steps
Begin by computing entropy over model logits and detecting uncertainty phases during generation.
import torch
import torch.nn.functional as F
import
as
class
EntropyMonitor
"""Track entropy in model predictions to detect uncertainty."""
def
__init__
self, entropy_threshold=1.5
self
self
def
compute_entropy
self, logits: torch.Tensor
"""Compute Shannon entropy of logits."""
1
1e-10
sum
1
return
def
is_uncertain
self, logits: torch.Tensor
bool
"""Check if prediction entropy exceeds threshold."""
self
self
self
return
def
get_uncertainty_regions
self, logits_sequence: list
list
"""Identify contiguous uncertainty regions in generation."""
self
for
in
None
for
in
enumerate
if
and
is
None
elif
not
and
is
not
None
None
if
is
not
None
len
return
Next, implement continuous embedding generation during high-entropy states.
class ContinuousEmbeddingDecoder :
"""Generate superposed embeddings instead of discrete tokens during uncertainty."""
def __init__ (self, embedding_matrix: torch.Tensor, temperature=0.7 ):
self .embedding_matrix = embedding_matrix
self .temperature = temperature
def discrete_decode (self, logits: torch.Tensor ) -> int :
"""Standard: select token with highest probability."""
token_id = torch.argmax(logits, dim=-1 )
return token_id.item()
def continuous_decode (self, logits: torch.Tensor ) -> torch.Tensor:
"""Uncertain: create superposed embedding from probability distribution."""
scaled_logits = logits / self .temperature
probs = F.softmax(scaled_logits, dim=-1 )
continuous_embedding = torch.matmul(probs, self .embedding_matrix)
return continuous_embedding
def hybrid_decode (self, logits: torch.Tensor, entropy: float ,
threshold: float = 1.5 ) -> torch.Tensor:
"""Choose decoding based on entropy."""
if entropy > threshold:
return self .continuous_decode(logits)
else :
token_id = self .discrete_decode(logits)
return self .embedding_matrix[token_id]
Now implement visual grounding that reinforces factual anchoring during uncertain phases.
class VisualAnchor :
"""Inject visual information to ground reasoning during uncertainty."""
def __init__ (self, visual_embed_dim=768 , text_embed_dim=768 ):
self .visual_embed_dim = visual_embed_dim
self .text_embed_dim = text_embed_dim
self .text_to_visual = torch.nn.Linear(text_embed_dim, visual_embed_dim)
self .visual_to_text = torch.nn.Linear(visual_embed_dim, text_embed_dim)
def compute_visual_prior (self, visual_embeddings: torch.Tensor,
question_embedding: torch.Tensor ) -> torch.Tensor:
"""Compute prior distribution over visual regions relevant to question."""
question_in_visual = self .text_to_visual(question_embedding)
relevance = torch.matmul(visual_embeddings, question_in_visual)
relevance = F.softmax(relevance, dim=0 )
return relevance
def inject_visual_anchor (self, text_embedding: torch.Tensor,
visual_embeddings: torch.Tensor,
visual_prior: torch.Tensor,
injection_strength: float = 0.3 ) -> torch.Tensor:
"""Blend text embedding with weighted visual information."""
aggregated_visual = torch.matmul(visual_prior, visual_embeddings)
visual_in_text = self .visual_to_text(aggregated_visual)
anchored = (1.0 - injection_strength) * text_embedding + \
injection_strength * visual_in_text
return anchored
Finally, integrate entropy-aware decoding into the generation loop.
class LatentEntropyAwareDecoder :
"""Full decoding pipeline with entropy-aware mode switching."""
def __init__ (self, model, tokenizer, embedding_matrix, visual_model=None ):
self .model = model
self .tokenizer = tokenizer
self .entropy_monitor = EntropyMonitor(entropy_threshold=1.5 )
self .continuous_decoder = ContinuousEmbeddingDecoder(embedding_matrix)
self .visual_anchor = VisualAnchor() if visual_model else None
self .visual_model = visual_model
def generate (self, prompt: str , visual_input=None , max_length=256 ):
"""Generate with entropy-aware mode switching."""
token_ids = self .tokenizer.encode(prompt)
generated = []
embeddings_history = []
for step in range (max_length):
with torch.no_grad():
outputs = self .model(torch.tensor([token_ids]))
logits = outputs.logits[0 , -1 , :]
entropy = self .entropy_monitor.compute_entropy(logits)
if entropy > self .entropy_monitor.entropy_threshold:
embedding = self .continuous_decoder.continuous_decode(logits)
if self .visual_anchor and visual_input is not None :
question_embedding = self .model.encode_text(prompt)
visual_embeddings = self .visual_model(visual_input)
visual_prior = self .visual_anchor.compute_visual_prior(
visual_embeddings, question_embedding)
embedding = self .visual_anchor.inject_visual_anchor(
embedding, visual_embeddings, visual_prior,
injection_strength=0.4 )
embeddings_history.append(embedding)
distances = torch.norm(
self .continuous_decoder.embedding_matrix - embedding,
dim=1 )
next_token = torch.argmin(distances).item()
else :
next_token = torch.argmax(logits, dim=-1 ).item()
embeddings_history.append(
self .continuous_decoder.embedding_matrix[next_token])
generated.append(next_token)
token_ids.append(next_token)
if next_token == self .tokenizer.eos_token_id:
break
return self .tokenizer.decode(generated)
def evaluate_hallucination_mitigation (model, test_cases, visual_inputs=None ):
"""Measure hallucination reduction with LEAD."""
decoder = LatentEntropyAwareDecoder(model, tokenizer, embedding_matrix,
visual_model=vision_model)
hallucination_scores = []
for i, test in enumerate (test_cases):
visual = visual_inputs[i] if visual_inputs else None
output = decoder.generate(test['prompt' ], visual_input=visual)
hallucination_score = compute_factuality(output, test['reference' ])
hallucination_scores.append(hallucination_score)
avg_factuality = np.mean(hallucination_scores)
print (f"Average Factuality Score: {avg_factuality:.3 f} " )
return hallucination_scores
Practical Guidance Hyperparameters and When to Use:
Entropy threshold typically 1.0-2.0; lower values trigger continuous mode more frequently, reducing hallucinations but potentially adding noise
Visual injection strength 0.2-0.5; stronger injection provides better grounding but may reduce model flexibility
Temperature for continuous embeddings 0.5-1.0; lower temperatures sharpen probabilities, higher values smooth them
Apply when visual information is available and can ground reasoning
Most effective for visual QA, multimodal reasoning, and tasks with clear visual-semantic alignment
For text-only models without visual grounding; benefits diminish
When computational budget is tight; continuous embedding generation adds overhead
For tasks where entropy of correct reasoning is naturally high (open-ended generation)
Using fixed entropy threshold across all tasks; calibrate per task or use adaptive thresholds
Visual anchoring without proper alignment learning; train projection matrices on paired visual-semantic data
Switching modes too aggressively, causing incoherent outputs; use gradual mode transitions with blending
Embedding matrix misalignment if using quantized or custom vocabularies; ensure consistency with model tokenizer
Reference 