visual-grounding-reinforcement-learning

name	visual-grounding-reinforcement-learning
title	High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.05920
keywords	["Multimodal LLM","Grounding","Reinforcement Learning","Visual Reasoning","High-Resolution Images"]
description	Improve LMM performance on high-resolution images by training models to ground reasoning on image regions through RL, learning spatial localization without requiring expensive grounding annotations.

Visual Grounding Reinforcement Learning: Emergent Spatial Reasoning Without Annotations

Large multimodal models (LMMs) struggle with high-resolution images because they process entire images at once, losing detailed visual information critical for accurate reasoning. Humans naturally zoom into relevant regions when analyzing complex images. Yet teaching models to perform spatial grounding typically requires expensive annotations (bounding boxes, coordinates) linking questions to image regions.

This work proposes MGPO (Multi-turn Grounding Policy Optimization), an RL framework enabling LMMs to learn spatial grounding autonomously using only binary correctness signals. The key insight is that grounding can emerge as a side effect of RL training on question-answering tasks if the model architecture allows it. By designing a two-turn dialogue template where the model first predicts coordinates, then answers based on cropped sub-images, grounding emerges naturally during optimization without requiring supervised grounding annotations.

Core Concept

The fundamental insight is that visual grounding—learning to focus on relevant image regions—emerges naturally through RL training when the model architecture supports it, using only binary reward signals (correct/incorrect). Rather than requiring supervised fine-tuning on coordinate annotations, we structure the task so that effective spatial reasoning becomes the optimal policy for improving answer correctness.

The two-turn mechanism works as: (1) Turn 1: Model predicts coordinates of relevant image regions, (2) Turn 2: Model receives cropped sub-image and answers the question. RL training uses only the final answer's correctness to update both the coordination mechanism and the reasoning process, causing the model to learn to focus on helpful regions.

Architecture Overview

• Base LMM Encoder: Vision transformer extracting features from high-resolution images
• Coordinate Prediction Head: Neural network predicting pixel coordinates [x, y, width, height] for relevant regions
• Image Cropping Module: Differentiable or discrete image selection mechanism extracting sub-regions based on predicted coordinates
• Question-Answering Head: LLM decoder generating answers from cropped image features and text queries
• Multi-turn Dialogue Template: Conversational structure alternating between region prediction (turn 1) and QA reasoning (turn 2)
• Reward Model: Binary classifier assessing answer correctness from final output
• RL Policy Optimization: GRPO (Group Relative Policy Optimization) extended to multi-turn sequences

Implementation

The following implements visual grounding emergence through RL without requiring coordinate annotations.