Improve LMM performance on high-resolution images by training models to ground reasoning on image regions through RL, learning spatial localization without requiring expensive grounding annotations.
Improve LMM performance on high-resolution images by training models to ground reasoning on image regions through RL, learning spatial localization without requiring expensive grounding annotations.
Visual Grounding Reinforcement Learning: Emergent Spatial Reasoning Without Annotations
Large multimodal models (LMMs) struggle with high-resolution images because they process entire images at once, losing detailed visual information critical for accurate reasoning. Humans naturally zoom into relevant regions when analyzing complex images. Yet teaching models to perform spatial grounding typically requires expensive annotations (bounding boxes, coordinates) linking questions to image regions.
This work proposes MGPO (Multi-turn Grounding Policy Optimization), an RL framework enabling LMMs to learn spatial grounding autonomously using only binary correctness signals. The key insight is that grounding can emerge as a side effect of RL training on question-answering tasks if the model architecture allows it. By designing a two-turn dialogue template where the model first predicts coordinates, then answers based on cropped sub-images, grounding emerges naturally during optimization without requiring supervised grounding annotations.
Core Concept
The fundamental insight is that visual grounding—learning to focus on relevant image regions—emerges naturally through RL training when the model architecture supports it, using only binary reward signals (correct/incorrect). Rather than requiring supervised fine-tuning on coordinate annotations, we structure the task so that effective spatial reasoning becomes the optimal policy for improving answer correctness.
The two-turn mechanism works as: (1) Turn 1: Model predicts coordinates of relevant image regions, (2) Turn 2: Model receives cropped sub-image and answers the question. RL training uses only the final answer's correctness to update both the coordination mechanism and the reasoning process, causing the model to learn to focus on helpful regions.
Architecture Overview
Base LMM Encoder: Vision transformer extracting features from high-resolution images
Coordinate Prediction Head: Neural network predicting pixel coordinates [x, y, width, height] for relevant regions
Image Cropping Module: Differentiable or discrete image selection mechanism extracting sub-regions based on predicted coordinates
Question-Answering Head: LLM decoder generating answers from cropped image features and text queries
Multi-turn Dialogue Template: Conversational structure alternating between region prediction (turn 1) and QA reasoning (turn 2)
Reward Model: Binary classifier assessing answer correctness from final output
Document image understanding requiring fine-grained text reading
Any multimodal task where focusing on relevant regions improves accuracy
Scenarios where collecting grounding annotations is expensive or infeasible
Applications where understanding spatial relationships matters (scene understanding, layout analysis)
When NOT to Use
Simple image classification tasks (full image is relevant)
Real-time systems with strict latency requirements (two-turn dialogue adds overhead)
Scenarios where all image regions are equally important
Tasks where pixel-level precision is required (grounding may be coarse)
Systems where interpretability of region selection is less important than accuracy
Common Pitfalls
Expecting supervision to teach grounding better: RL with only binary signals actually outperforms supervised fine-tuning with bounding boxes because it learns task-specific grounding rather than copying annotations.
Ignoring crop size impact: Cropping too aggressively loses context; too conservatively wastes computation. Experiment with output_size parameter.
Insufficient RL training: Grounding emerges slowly. Ensure sufficient epochs (5+) for learning to stabilize.
Not monitoring coordinate consistency: If learned coordinates are too noisy (high std), increase temperature or use coordinate smoothing.
Confusing coordinate scale: Ensure coordinates are normalized to [0, 1] matching image normalization. Mismatches cause poor cropping.
