rewardmap-sparse-rewards-visual-reasoning

name	rewardmap-sparse-rewards-visual-reasoning
title	RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.02240
keywords	["reward-design","visual-reasoning","sparse-rewards","multi-stage-RL","spatial-reasoning"]
description	Improve multimodal LLMs on fine-grained visual reasoning tasks (e.g., reading transit maps) by decomposing training into stages: basic perception (VQA) -> simple reasoning -> complex spatial reasoning. Incorporates 'detail rewards' for intermediate visual understanding, bootstrapping models from simple to complex tasks while addressing sparse reward challenges.

RewardMap: Bootstrapping Visual Reasoning via Staged Reward Design

Fine-grained visual reasoning tasks—reading complex diagrams, reasoning about spatial relationships, extracting details from images—are notoriously hard for multimodal LLMs. The challenge is that end-to-end RL on these tasks produces sparse rewards: the model either completely solves a transit map puzzle or gets zero reward. This forces the model to learn everything simultaneously, which is inefficient.

RewardMap decomposes this into stages: start with simple perception tasks (answering basic questions about images), advance to spatial reasoning, and finish with complex multi-step tasks. Each stage includes intermediate "detail rewards" that provide richer supervision, turning sparse binary rewards into dense signals that guide learning.

Core Concept

RewardMap's multi-stage curriculum uses three reward types:

1. Perception rewards: VQA-style questions about visual details (e.g., "What color is this box?") Score: 1 if correct.
2. Intermediate rewards: Partial reasoning steps (e.g., "Did you correctly identify the starting point?") Score: 0-1 based on intermediate correctness.
3. Task rewards: Full problem completion (e.g., "Did you find the correct transit route?") Score: 1 if task solved.

By training stage-by-stage with appropriate reward signals, the model learns perception first, then spatial reasoning, then complex multi-step logic—building capabilities progressively rather than learning all at once.

Architecture Overview

• Stage 1 (Perception): Simple VQA on image details, binary rewards
• Stage 2 (Basic Reasoning): Spatial understanding (which object is left of X?), detail rewards
• Stage 3 (Complex Reasoning): Full task completion (find route from A to B), combined rewards
• Reward module: Computes detail, intermediate, and task-level rewards
• Data curator: ReasonMap-Plus dataset with annotated intermediate steps

Implementation Steps

Start by building the perception stage with VQA rewards:

import torch
import torch.nn.functional as F

class VQARewardComputer:
    """
    Compute perception-level rewards from visual question answering.
    """
    def __init__():
        .vqa_model = vqa_model  

     ():
        
        
        ground_truth = .vqa_model.get_answer(image, question)

        
        is_correct = predicted_answer.strip().lower() == ground_truth.lower()

         (is_correct)

     ():
        
        rewards = []
         img, ans, q  (images, answers, questions):
            reward = .compute_perception_reward(img, ans, q)
            rewards.append(reward)

         torch.tensor(rewards)