Reveals that inference-time scaling techniques for LLMs don't transfer to VLMs: majority voting beats verification, self-correction happens in <10% of cases, and models verify better without images. Use insights to design VLM evaluation methods that work rather than assuming LLM techniques apply directly.
Reveals that inference-time scaling techniques for LLMs don't transfer to VLMs: majority voting beats verification, self-correction happens in <10% of cases, and models verify better without images. Use insights to design VLM evaluation methods that work rather than assuming LLM techniques apply directly.
Aha Moment Revisited: Reconsidering VLM Verification at Inference Time
Scaling language models at inference time through self-verification and correction has shown promise for LLMs—models can double-check answers and catch errors. But do these techniques transfer to vision-language models? This paper reveals three surprising findings: (1) simple majority voting substantially outperforms verification-focused strategies, (2) "Aha moments" (successful self-corrections) occur in fewer than 10% of cases and provide minimal improvement, and (3) counterintuitively, models verify answers more accurately when visual information is removed. These findings challenge assumptions that inference-time scaling automatically benefits multimodal systems.
The insight is that VLMs struggle to integrate visual information during self-evaluation—they focus narrowly rather than using image context to verify reasoning. Separate decoding pathways for generation vs. verification may be necessary.
Core Concept
The paper compares three inference strategies on visual reasoning tasks:
Greedy Decoding: Single forward pass, first generated answer
Majority Voting (Generation-Focused): Generate N responses, take majority—"let many models speak"
Best-of-N with Self-Verification (Verification-Focused): Generate N responses, model re-evaluates each and selects best
The critical finding: majority voting (generation) outperforms verification despite verification sounding more sophisticated. This suggests VLMs generate better than they verify—their strength is in seeing and describing, not in abstract evaluation.
Architecture Overview
VLM Base Model: Vision-language model (Qwen-RL-7B variants) capable of chain-of-thought reasoning
Generation Decoder: Samples diverse outputs from same model
Verification Pathway: Same model or independent verifier evaluates outputs
Visual Input Toggle: Tests with/without image context during verification
Majority Voting Baseline: Non-verification aggregation of generations
Evaluation Metrics: Accuracy, success rate of self-correction, confidence calibration
"""
Implements three inference decoding strategies for visual reasoning tasks.
Tests whether verification actually improves VLM accuracy.
"""
def
__init__
self, model, tokenizer, num_samples: int = 5
"""
Args:
model: Pretrained VLM (e.g., Qwen-RL-7B)
tokenizer: Tokenizer for the model
num_samples: Number of outputs to generate for majority voting
"""
self
self
self
def
greedy_decoding
self, image: torch.Tensor, question: str
str
"""
Baseline: single forward pass with greedy decoding.
Most efficient but potentially suboptimal.
"""
# Encode image and question
self
# Single greedy forward pass
with
self
256
0.0
# Greedy
False
self
0
return
def
majority_voting
self,
image: torch.Tensor,
question: str,
num_generations: int = None
Tuple
str
float
"""
Generation-focused scaling: sample diverse outputs, vote.
Does NOT use self-verification—just lets multiple outputs speak.
Args:
image: Input image
question: Visual question
num_generations: Number of samples (default: self.num_samples)
Returns:
voted_answer: Most common answer
confidence: Fraction of votes for winner
"""
if
is
None
self
# Generate multiple diverse responses
for
in
range
self
with
self
256
0.7
# Diverse sampling
True
0.9
self
0
# Count votes (simplified: extract final answer)
for
in
self
0
1
# Select most common
max
return
def
best_of_n_with_verification
self,
image: torch.Tensor,
question: str,
num_candidates: int = None
Tuple
str
float
"""
Verification-focused scaling: generate N answers, verify each, select best.
Uses self-verification to choose winner.
This is the strategy that SHOULD work but doesn't well for VLMs.
"""
if
is
None
self
# Step 1: Generate N candidate answers
for
in
range
self
with
self
256
0.7
True
self
0
# Step 2: Evaluate each candidate through verification
for
in
# Create verification prompt
f"""
Given the visual question: "{question}"
And this proposed answer: "{candidate}"
Is this answer correct? Answer: Yes/No
"""
"""
Test verification accuracy WITHOUT providing the image.
Counterintuitive finding: VLMs verify BETTER without visual info.
This reveals that visual information confuses verification pathways.
"""
# Verification prompt WITHOUT image
f"""
Question: "{question}"
Proposed answer: "{answer}"
Is this answer correct? (Yes/No)
"""
# Encode question and answer, but NO image
self
with
self
# No image!
10
0.0
self
0
1.0
if
'yes'
in
else
0.0
return
def
_extract_final_answer
self, text: str
str
"""Extract final answer from chain-of-thought text."""
# Simplified: look for "Answer:" or take last line
if
"Answer:"
in
return
"Answer:"
1
return
'\n'
1
def
evaluate_all_strategies
self,
test_samples: List[Dict],
num_generations: int = 5
Dict
str
Dict
"""
Evaluate all three strategies on test set.
Returns accuracy, success rate of self-correction, etc.
Args:
test_samples: List of {'image': ..., 'question': ..., 'answer': ...}
Returns:
results: Dict comparing strategies
"""
'greedy'
'correct'
0
'aha_moments'
0
'majority'
'correct'
0
'confidence'
'verification'
'correct'
0
'aha_moments'
0
'verification_no_image'
'correct'
0
for
in
'image'
'question'
'answer'
# Strategy 1: Greedy
self
if
self
'greedy'
'correct'
1
# Strategy 2: Majority voting
self
if
self
'majority'
'correct'
1
'majority'
'confidence'
# Strategy 3: Best-of-N with verification
self
if
self
'verification'
'correct'
1
# Track if this is an "aha moment" (verification corrected wrong answer)
if
not
self
'verification'
'aha_moments'
1
# Strategy 4: Verification WITHOUT visual info
self
if
0.5
# Model said "yes"
'verification_no_image'
'correct'
1
# Normalize by dataset size
len
for
in
if
'correct'
in
'accuracy'
'correct'
return
def
_answers_match
self, predicted: str, ground_truth: str
bool
"""Check if answers match (simplified)."""
# Extract numbers if present
import
r'\d+'
r'\d+'
if
and
return
0
0
return
Analysis of aha moments and when verification works:
defanalyze_aha_moments(
results: Dict,
test_samples: List[Dict],
model: object) -> Dict:
"""
Deep analysis of self-correction "aha moments".
Why do they occur <10% of the time?
"""
aha_moments = []
failed_corrections = []
for sample in test_samples:
image = sample['image']
question = sample['question']
ground_truth = sample['answer']
# Generate greedy baseline
greedy_ans = model.greedy_decoding(image, question)
is_greedy_wrong = not model._answers_match(greedy_ans, ground_truth)
if is_greedy_wrong:
# Generate and verify candidate
candidates, scores = model.best_of_n_with_verification(
image, question, num_candidates=5
)
is_verified_correct = model._answers_match(candidates, ground_truth)
if is_verified_correct:
# Success: aha moment
aha_moments.append({
'question': question,
'greedy_wrong': greedy_ans,
'verified_correct': candidates,
'why': 'Verification selected correct alternative'
})
else:
# Failure: verification didn't help
failed_corrections.append({
'question': question,
'greedy_wrong': greedy_ans,
'verified_still_wrong': candidates,
'reason': 'All candidates wrong, or verification chose wrong one'
})
aha_success_rate = len(aha_moments) / (
len(aha_moments) + len(failed_corrections) + 1e-6
)
return {
'aha_success_rate': aha_success_rate,
'num_aha_moments': len(aha_moments),
'num_failed_corrections': len(failed_corrections),
'finding': f"Aha moments occur in {aha_success_rate*100:.1f}% of cases (<10% expected)",
'insight': 'Verification is unreliable; majority voting more dependable'
}
Practical Guidance
Aspect
Value
Notes
Majority Voting Accuracy
Better than verification
Generation > verification for VLMs
Aha Moment Success Rate
<10%
Self-correction rarely helps
Verification Without Images
Better accuracy
Visual info confuses verification
Best Strategy for VLMs
Majority voting
Scaling through generation, not verification
Confidence Calibration
Poor
Models poorly calibrated on visual tasks
When to use:
Understanding inference-time scaling limitations for VLMs
Designing evaluation methods that work with multimodal models
Deciding between generation vs. verification strategies
Debugging VLM failures on visual reasoning tasks
Comparing VLM capabilities to LLM scaling techniques
When NOT to use:
Applying LLM self-verification directly to VLMs without testing
Assuming "let it think longer" helps VLMs like it helps LLMs
Building systems relying on VLM self-correction for robustness
Tasks where visual context is essential for reasoning (use full images)
Scenarios where single-pass generation already works well
Common pitfalls:
Assuming LLM inference-time scaling directly transfers to VLMs
Over-engineering verification pathways that don't improve accuracy
Not separating generation from verification evaluations
Using same model for generation and verification (no independent check)
Providing images during verification when text-only might work better
Ignoring majority voting as a simpler, often-better baseline
Misinterpreting low aha rates as model inability (generation > verification)
Reference
"Aha Moment Revisited: Are VLMs Truly Capable of Self Verification in Inference-time Scaling?", 2025. arxiv.org/abs/2506.17417
