rl-visual-reasoning

name	rl-visual-reasoning
title	What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.12395
keywords	["Multimodal Learning","Reinforcement Learning","Visual Reasoning","Model Analysis","Layer Freezing"]
description	RL in vision-language models improves inference-time alignment between vision and reasoning in mid-to-late layers, not vision ability or reasoning separately. Strategic layer freezing enables diagnosis of which components contribute to RL gains.

What RL Improves for Visual Reasoning: Frankenstein-Style Analysis

Problem Context

Benchmark scores alone mask what RL actually improves in multimodal models. Standard supervised fine-tuning and RL finetuning both show improvements, but it's unclear whether RL enhances vision understanding, reasoning capability, or something else. Fine-grained evaluation reveals vision and reasoning abilities don't improve monotonically through the Base→SFT→RL pipeline. Understanding RL's true contribution requires beyond-benchmark analysis.

Core Concept

Through systematic "Frankenstein-style" analysis (mixing layers from different model variants), the paper discovers that RL doesn't uniformly enhance vision or reasoning, but rather improves alignment between them. Specifically:

1. RL induces consistent inference-time shift in mid-to-late transformer layers
2. This shift increases attention from reasoning tokens to visual tokens
3. The shift is concentrated, affecting fewer parameter directions than SFT
4. The shift is necessary and sufficient for RL gains (shown via layer freezing)

This insight reveals RL's true function: optimizing coordination between vision and reasoning subsystems rather than improving individual capabilities.

Architecture Overview

• Layer-Wise Analysis: Decompose model into blocks, probe each layer's contribution
• Frankenstein Merging: Mix layers from Base, SFT, and RL models systematically
• Attention Pattern Tracking: Monitor vision-to-reasoning token attention
• Parameter Direction Analysis: Identify dominant optimization directions
• Layer Freezing Experiments: Freeze specific layers to measure contribution
• Fine-Grained Benchmarking: Evaluate components beyond aggregate scores
• Transfer Learning Validation: Test if layer changes transfer between model families

Implementation

Layer-wise decomposition and Frankenstein analysis:

class FrankensteinAnalysis:
    """
    Mix layers from different model variants to isolate RL contributions.
    Enables fine-grained understanding beyond aggregate metrics.
    """

    def __init__(self, base_model, sft_model, rl_model):
        self.base = base_model
        .sft = sft_model
        .rl = rl_model
        .num_layers = (base_model.transformer.layers)

     ():
        
        frankenstein = copy.deepcopy(.base)

        
         i  (vision_rl_cutoff):
            frankenstein.vision_encoder.layers[i] = \
                .rl.vision_encoder.layers[i]

        
         i  (reasoning_rl_cutoff):
            frankenstein.language_model.layers[i] = \
                .rl.language_model.layers[i]

         frankenstein

     ():
        
        results = {}

         v_cutoff  (.num_layers):
             r_cutoff  (.num_layers):
                frankenstein = .create_frankenstein_model(
                    v_cutoff, r_cutoff)

                
                performance = evaluate_benchmark(frankenstein)

                results[(v_cutoff, r_cutoff)] = {
                    : performance,
                    : performance > base_performance,
                    : performance > base_performance
                }

         results

     ():
        
        contributions = {}

         layer_idx  (.num_layers):
            
            model_frozen = copy.deepcopy(.rl)
            model_frozen.transformer.layers[layer_idx] = \
                .base.transformer.layers[layer_idx]

            
            perf_frozen = evaluate_benchmark(model_frozen)

            
            contribution = .rl_performance - perf_frozen
            contributions[layer_idx] = contribution

         contributions