Transfer reasoning behaviors learned in language models to visual domains through two-stage training: cold-start linguistic fine-tuning followed by multimodal RL. Open-Vision-Reasoner achieves 95.3% on MATH500 and 54.6% on MathVerse by learning visual analogs of backtracking, verification, and subgoal decomposition using rule-based rewards.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Transfer reasoning behaviors learned in language models to visual domains through two-stage training: cold-start linguistic fine-tuning followed by multimodal RL. Open-Vision-Reasoner achieves 95.3% on MATH500 and 54.6% on MathVerse by learning visual analogs of backtracking, verification, and subgoal decomposition using rule-based rewards.
Open Vision Reasoner: Transfer Reasoning From Language to Vision
Language models trained with reinforcement learning develop sophisticated reasoning patterns—backtracking when stuck, verifying intermediate steps, decomposing problems into subgoals. These cognitive behaviors are not language-specific; they're general problem-solving strategies. Open Vision Reasoner transfers these patterns to multimodal models by first training extensively on linguistic reasoning, then adapting to visual domains via reinforcement learning with minimal process annotations. The key insight is that linguistic cold-start memorizes diverse reasoning behaviors, while multimodal RL scales up only the patterns effective for vision, achieving 95.3% accuracy on visual math problems with a 7B parameter model.
The approach reveals a fundamental trade-off: linguistic pretraining sometimes initially degrades visual perception (model focuses on text), but multimodal RL recovers this capacity while preserving reasoning sophistication.
Core Concept
Open Vision Reasoner uses a two-stage training pipeline:
Cold-Start Linguistic Fine-tuning: Train on 2M text-only reasoning examples (math, code, logic) to embed backtracking, verification, and subgoal decomposition patterns
Multimodal RL Scaling: Fine-tune on 300K visual reasoning problems using rule-based verifiable rewards (no process annotations needed), scaling up behaviors effective for images
The "Aha Moment" occurs when the model transitions from generic linguistic patterns to visual-specific reasoning (e.g., "visual reflection" = looking at different parts of image, "divide-and-conquer" = solving sub-problems in different image regions).
Architecture Overview
Base Vision-Language Model: Qwen2.5-VL-7B or similar multimodal backbone
Linguistic Reasoning Head: Attention layer fine-tuned on text reasoning
Visual Reasoning Adapter: LoRA or prompt tuning for vision-specific patterns
Rule-Based Reward Model: Deterministic verifier for outcomes (e.g., comparing predicted answer to ground truth)
Process Reward Estimator: Lightweight classifier scoring reasoning quality per step (learned from outcome labels)
Behavior Extraction Module: Identifies which patterns (backtracking, verification, etc.) activate during reasoning
Implementation
The following demonstrates the two-stage pipeline and behavior transfer:
"""
Compute outcome reward (binary) and estimate process rewards per step.
Args:
predicted_answer: Model's final answer
ground_truth: Correct answer
reasoning_steps: List of reasoning text for each step
Returns:
outcome_reward: 1.0 if correct, 0.0 if incorrect
process_rewards: Dict mapping step_idx → reward estimate
"""
# Outcome reward: simple correctness check
1.0
if
self
else
0.0
# Process rewards: heuristic scoring per step
for
in
enumerate
# Simple heuristics: longer steps, steps with justification get higher scores
min
len
100
1.0
# Normalized length
1.0
if
"because"
in
else
0.5
return
@staticmethod
def
_answers_equal
pred: str, truth: str
bool
"""Normalize and compare answers."""
import
# Remove punctuation and normalize whitespace
r'[^\w\s]'
''
r'[^\w\s]'
''
return
class
OpenVisionReasonerModel
"""Complete multimodal reasoning model with visual adapter."""
def
__init__
self, base_model_name: str = "Qwen/Qwen2.5-VL-7B"
super
# Base VLM (frozen backbone, trained in stage 2)
self
None
# Load from base_model_name in practice
# Visual reasoning adapter (trained in stage 2)
self
4096
256
# Reward model (fixed rules, not trained)
self
def
generate_reasoning_trajectory
self, question: str, image: torch.Tensor,
max_steps: int = 10
Tuple
List
str
"""Generate multi-step reasoning for visual question."""
# Process image through VLM
self
self
for
in
range
# Adapt to visual domain using behavior selector
self
1
# Generate next reasoning step
self
1
self
# Update hidden state for next iteration
self
1
return
@staticmethod
def
_decode_token
token: torch.Tensor
str
"""Convert token to text."""
return
"placeholder_text"
def
train_stage1_linguistic
model, dataset: List[Dict], optimizer, num_epochs: int = 3
"""Stage 1: Cold-start fine-tuning on linguistic reasoning."""
for
in
range
0
for
in
enumerate
10000
# Example: sample 10K per epoch
# Forward pass on text
'problem'
'reasoning'
'answer'
# Tokenize and encode
# Generate from problem
# Language modeling loss
1
1
print
f"Stage 1 Epoch {epoch + 1}: Loss {total_loss / len(dataset[:10000]):.4f}"
Extreme visual complexity (medical imaging, remote sensing): Rule-based rewards too simplistic; need learned reward models
Models <1B parameters: Overhead of behavior adaptation exceeds model capacity benefits
Common Pitfalls
Insufficient Stage 1 Data: Using <1M linguistic examples under-trains reasoning patterns. The model won't know sophisticated backtracking/verification. Use diverse datasets (GSM8K, MATH, CodeContests, CommonsenseQA).
Rule-Based Reward Too Strict: If reward=0 for any error, model only learns avoiding mistakes, not reasoning. Use soft rewards: deduct 0.5 for arithmetic errors, 0.2 for format errors.
Behavior Selector Collapse: If model only uses one behavior (e.g., always backtrack), diversity is lost. Add entropy regularization: encourage behavior distribution. Add to loss: 0.1 * entropy(behavior_logits).
Perception-Reasoning Trade-off Ignored: Stage 1 linguistic training can degrade visual perception initially. Monitor visual accuracy during early Stage 2. If dropping >5%, reduce RL learning rate or balance linguistic + visual loss.
Weak Process Reward Signals: Simple heuristics (step length, keyword matching) are noisy. Validate with human annotations on 1-2% of data to ensure rewards correlate with quality.
Reference
Deng, Z., Liu, H., et al. (2025). Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning. arXiv preprint arXiv:2507.05255.
