Aligns reasoning traces with final decisions in preference models using an independent frozen VLM as listener. Achieves 67.4% accuracy on ImageReward by enforcing consistency between explanations and choices. Use when training reward models for image generation quality where both reasoning quality and accuracy matter.
Aligns reasoning traces with final decisions in preference models using an independent frozen VLM as listener. Achieves 67.4% accuracy on ImageReward by enforcing consistency between explanations and choices. Use when training reward models for image generation quality where both reasoning quality and accuracy matter.
Listener-Rewarded Thinking: Enforcing Consistency in Visual Preference Reasoning
Reward models for predicting human visual preferences often produce chain-of-thought explanations that contradict their final decisions—a critical failure mode where the model cannot convincingly justify its preference judgment. This misalignment hurts out-of-distribution generalization and indicates the reasoning isn't actually driving the decisions. Listener-Rewarded Thinking solves this by training with an independent frozen vision-language model as a "listener" that evaluates whether the reasoning explanation would convince it to make the same choice. This creates a soft reward signal penalizing contradictory reasoning while reinforcing coherent judgments.
The insight is that reasoning quality matters not just for human interpretability but for model generalization. Models forced to produce convincing explanations develop more robust preference criteria, transferring better to new visual distributions.
Core Concept
Listener-Rewarded Thinking combines Group Relative Policy Optimization (GRPO) with a novel listener-augmented reward function. Rather than simply optimizing for correct preference predictions, the framework introduces three reward components:
Accuracy Reward: Standard correctness signal (chosen image matches ground truth)
Listener Reward: Independent frozen VLM's confidence that the reasoning supports the conclusion
The combined reward is weighted: r_total = r_fmt + 0.5·r_acc + 0.5·r_list. This architecture ensures both the reasoning traces and final decisions improve together.
Architecture Overview
Base Reasoning Model: Vision-language model generating chain-of-thought preference explanations
# Step 3: Evaluate listener reward (listener agrees with reasoning)
self
return
'r_fmt'
'r_acc'
'r_list'
'total'
0.5
0.5
def
_compute_formatting_reward
self, reasoning_text
"""Reward well-structured chain-of-thought."""
# Check for presence of key reasoning markers
'therefore'
'because'
'this means'
'as a result'
sum
1
for
in
if
in
# Length reward: longer reasoning (200-500 tokens) better than too short
len
self
min
1.0
50
450
# Normalized to [0,1]
0.3
len
0.7
return
def
_compute_accuracy_reward
self, image_pair, preference_idx, reasoning_text
"""Reward correct preference prediction."""
# Extract predicted preference from reasoning
self
# Accuracy: 1.0 if correct, 0.0 if wrong
if
1.0
else
0.0
return
def
_compute_listener_reward
self, image_pair, preference_idx, reasoning_text
"""
Frozen listener evaluates: "Does this explanation justify this preference?"
High confidence = high reward.
"""
# Create prompt for listener
f"""
Given these two images:
[Image 0] and [Image 1]
The reasoning model explains: "{reasoning_text}"
Based on this explanation, which image is preferred?
Respond with only the image number (0 or 1) and confidence.
"""
# Get listener response
self
'pt'
with
self
50
True
True
# Parse listener's prediction
self
0
True
self
# Confidence extraction (simplified)
self
# Listener reward: confidence × correctness
# High reward if listener agrees with reasoning
if
else
1.0
return
def
_extract_preference_from_text
self, text
"""Extract preference index (0 or 1) from text."""
# Heuristic: look for "image 0" or "image 1"
if
'image 0'
in
or
'first'
in
return
0
elif
'image 1'
in
or
'second'
in
return
1
else
return
1
# Ambiguous
def
_extract_confidence
self, text
"""Extract confidence score from listener response."""
# Heuristic: look for percentage or confidence keywords
if
'high confidence'
in
or
'definitely'
in
return
0.8
elif
'medium'
in
or
'likely'
in
return
0.5
elif
'low confidence'
in
return
0.2
else
return
0.5
# Default
class
GRPOTrainer
"""
Group Relative Policy Optimization with listener-augmented rewards.
Trains reasoning model to maximize total reward.
"""
def
__init__
self, model, reward_model, learning_rate=1e-5
self
self
self
def
train_step
self, image_pairs, preferences, num_groups=4
"""
GRPO training: optimize relative to group performance.
Args:
image_pairs: List of (img0, img1) tuples
preferences: List of preference indices
num_groups: Split into groups for relative comparison
"""
len
0
for
in
range
# Get group samples
# Generate reasoning for each sample
for
in
self
# Compute listener-augmented rewards
for
in
zip
self
'total'
# Compute group baseline (for relative comparison)
# Policy gradient: optimize relative advantage
for
in
enumerate
zip
# Log probability of generating this reasoning
self
# Loss: negative advantage-weighted log probability
# Update
self
self
return
Training loop demonstrating listener-augmented learning:
deftrain_with_listener_reward(
reasoning_model, reward_model, train_dataset, num_epochs=10):
"""
Train reasoning model with listener-augmented rewards.
Ensures explanations are convincing and accurate.
"""
trainer = GRPOTrainer(reasoning_model, reward_model)
dataset_size = len(train_dataset)
for epoch inrange(num_epochs):
epoch_loss = 0
num_batches = 0for batch_idx inrange(0, dataset_size, 32):
batch_end = min(batch_idx + 32, dataset_size)
batch = train_dataset[batch_idx:batch_end]
image_pairs = [item['image_pair'] for item in batch]
preferences = [item['preference'] for item in batch]
loss = trainer.train_step(image_pairs, preferences)
epoch_loss += loss
num_batches += 1if num_batches % 10 == 0:
print(f"Batch {num_batches}: Loss = {loss:.4f}")
print(f"Epoch {epoch} avg loss: {epoch_loss / num_batches:.4f}")
return reasoning_model
Practical Guidance
Aspect
Value
Notes
Accuracy on ImageReward
67.4%
State-of-the-art for preference prediction
Listener Disagreement Reduction
8.3% vs 10.1% baseline
Fewer contradictory reasoning events
Data Efficiency
16% of HPSv2
Training on reduced dataset
Generalization Improvement
+6% on 1.2M vote dataset
Better out-of-distribution performance
Reward Components
3 (format + accuracy + listener)
Balanced weighting important
Listener Model Freeze
Yes
Prevents shifting evaluation target
When to use:
Training reward models for image generation quality assessment
Situations where explanations need to justify decisions (interpretability matters)
Improving out-of-distribution generalization for preference models
Reducing contradictory reasoning that hurts user trust
Data-efficient training on limited preference annotations
When NOT to use:
If explanation quality is irrelevant (pure accuracy sufficient)
Scenarios without access to a good frozen listener model
Real-time systems where listener inference adds latency
Tasks where accuracy and explanation quality are decoupled
Binary preference classification without reasoning requirements
Common pitfalls:
Listener model too weak, unable to evaluate reasoning quality properly
