revisual-multimodal-reasoning

name	revisual-multimodal-reasoning
title	ReVisual-R1: Multimodal Reasoning with Cold-Start and Staged RL
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.04207
keywords	["multimodal-reasoning","reinforcement-learning","curriculum-learning","vision-language"]
description	Develop sophisticated multimodal reasoning through text-centric cold-start initialization, prioritized advantage distillation, and staged RL refinement.

ReVisual-R1: Advancing Multimodal Reasoning

Core Concept

ReVisual-R1 demonstrates that unlocking multimodal reasoning capabilities requires a carefully designed three-stage curriculum: text-only cold-start with complex reasoning examples, multimodal RL with prioritized advantage distillation to prevent gradient stagnation, and text-only refinement to consolidate linguistic fluency. This approach outperforms simultaneous mixed-modality training and achieves state-of-the-art performance among 3B/7B open-source models.

Architecture Overview

• Stage 1: Text-Centric Cold Start - 283K curated high-difficulty reasoning examples (language-only)
• Stage 2: Multimodal RL with PAD - GRPO enhanced by Prioritized Advantage Distillation
• Stage 3: Text-Only RL Refinement - Polish linguistic quality and reasoning consistency
• Prioritized Advantage Distillation (PAD): Filters zero-advantage samples and resamples informative trajectories
• Challenge Addressed: Gradient stagnation in multimodal GRPO settings where standard reward signals provide weak gradients

Implementation

Step 1: Prepare Cold-Start Text-Only Dataset

import json
from typing import List, Dict

class ColdStartDatasetBuilder:
    def __init__(self, num_samples=283000):
        self.target_size = num_samples
        self.difficulty_threshold = "high"

    def collect_complex_reasoning_examples(self):
        """Gather high-difficulty text reasoning from multiple sources"""
        sources = {
            'math_olympiad': self.source_olympiad_problems(),
            'theoretical_physics': self.source_physics_proofs(),
            'algorithm_design': self.source_algorithm_challenges(),
            'logical_deduction': self.source_logic_puzzles(),
        }

        dataset = []
        for source_name, examples in sources.items():
            for example in examples:
                if example['complexity'] == 'high':
                    dataset.append({
                        'question': example['prompt'],
                        'reasoning_chain': example['solution'],
                        'source': source_name,
                        'difficulty_score': example['difficulty'],
                        'reasoning_depth': example['step_count']
                    })

         dataset[:.target_size]

     ():
        
        augmented = []

         example  examples:
            
            structured_chain = .reformat_chain_of_thought(
                example[]
            )
            augmented.append({
                **example,
                : structured_chain
            })

         augmented

     ():
        
        steps = reasoning_text.split()

        formatted = []
         i, step  (steps, ):
            formatted.append()

         .join(formatted)

     ():
        
        
        
        
         


builder = ColdStartDatasetBuilder(num_samples=)
cold_start_data = builder.collect_complex_reasoning_examples()
cold_start_data = builder.augment_with_synthetic_reasoning(cold_start_data)

Step 2: Implement Prioritized Advantage Distillation

import torch
import numpy as np

class PrioritizedAdvantageDistillation:
    def __init__(self, temperature=1.0):
        self.temperature = temperature

    def compute_advantages(self, rewards, baseline_values):
        """Calculate advantage for each trajectory"""
        advantages = rewards - baseline_values

        # Normalize for stability
        mean_adv = np.mean(advantages)
        std_adv = np.std(advantages)
        normalized_advantages = (advantages - mean_adv) / (std_adv + 1e-8)

        return normalized_advantages

    def filter_zero_advantage_samples(self, trajectories, advantages,
                                     threshold=0.01):
        """Remove non-informative samples where advantage ≈ 0"""
        filtered = []
        valid_indices = []

        for idx, (traj, adv) in enumerate(zip(trajectories, advantages)):
            if abs(adv) > threshold:  # Keep only informative samples
                filtered.append(traj)
                valid_indices.append(idx)

        print(f"Filtered {len(trajectories) - len(filtered)} zero-advantage samples")
        print(f"Retained {len(filtered)}/{len(trajectories)} informative samples")

        return filtered, valid_indices

     ():
        

        
        
        exp_advantages = np.exp(advantages / temperature)
        probabilities = exp_advantages / np.(exp_advantages)

        
        num_samples = (trajectories)
        resampled_indices = np.random.choice(
            num_samples,
            size=num_samples,
            p=probabilities,
            replace=
        )

        resampled = [trajectories[i]  i  resampled_indices]

         resampled, resampled_indices

 :
     ():
        .model = model
        .pad = pad_module

     ():
        

        
        responses_text = .model.generate(batch_text)
        responses_mm = .model.generate(batch_multimodal)

        
        rewards_text = [reward_fn(r)  r  responses_text]
        rewards_mm = [reward_fn(r)  r  responses_mm]

        
        baseline_text = .estimate_baseline(batch_text)
        baseline_mm = .estimate_baseline(batch_multimodal)

        
        advantages_text = .pad.compute_advantages(
            np.array(rewards_text),
            baseline_text
        )
        advantages_mm = .pad.compute_advantages(
            np.array(rewards_mm),
            baseline_mm
        )

        
        responses_filtered, _ = .pad.filter_zero_advantage_samples(
            responses_mm, advantages_mm
        )
        responses_resampled, _ = .pad.prioritized_resampling(
            advantages_mm[_], responses_mm, temperature=
        )

        
        policy_loss = .compute_policy_loss(
            responses_resampled,
            advantages_mm
        )

        optimizer.zero_grad()
        policy_loss.backward()
        optimizer.step()

         policy_loss.item(), (responses_filtered), (responses_resampled)

Step 3: Implement Three-Stage Training Pipeline

class ThreeStageTrainingPipeline:
    def __init__(self, model, cold_start_data):
        self.model = model
        self.cold_start_data = cold_start_data
        self.pad = PrioritizedAdvantageDistillation()

    def stage_1_text_coldstart(self, num_epochs=3):
        """
        Train exclusively on text-only complex reasoning.
        Duration: Initialize model reasoning capability.
        """
        print("=== Stage 1: Text-Only Cold Start ===")
        print(f"Dataset size: {len(self.cold_start_data)} examples")
        print(f"Focus: Complex reasoning chains, logical coherence")

        optimizer = self.get_optimizer(lr=1e-4)

        for epoch in range(num_epochs):
            total_loss = 0

            for batch in self.create_batches(self.cold_start_data, bs=32):
                # Standard supervised training on reasoning chains
                loss = self.model.compute_language_loss(
                    batch['question'],
                    batch['structured_reasoning']
                )

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

                total_loss += loss.item()

            print(f"Epoch {epoch+1}: Loss = ")

     ():
        
        ()
        ()
        ()

        optimizer = .get_optimizer(lr=)

         iteration  (num_iterations):
            batch = .sample_multimodal_batch(multimodal_data, bs=)

            loss, filtered_count, resampled_count = (
                GRPOWithPAD(.model, .pad).multimodal_grpo_step(
                    batch[],
                    batch[],
                    reward_fn=.compute_multimodal_reward,
                    optimizer=optimizer
                )
            )

             (iteration + ) %  == :
                (
                      )

     ():
        
        ()
        ()
        ()

        optimizer = .get_optimizer(lr=)

         epoch  (num_epochs):
             batch  .create_batches(text_data, bs=):
                
                text_output = .model.generate(batch[])

                
                fluency_reward = .evaluate_fluency(text_output)
                coherence_reward = .evaluate_coherence(text_output)

                combined_reward =  * fluency_reward +  * coherence_reward

                
                log_probs = .model.log_probability(text_output)
                loss = -log_probs * (combined_reward - )

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

Practical Guidance

1. Text-First Initialization: Begin with 283K+ high-difficulty text-only examples before introducing visual data. This establishes reasoning foundations that transfer to multimodal tasks.

2. Detect Gradient Stagnation: Monitor loss plateaus during multimodal GRPO training. If loss stops decreasing despite correct implementation, PAD is likely needed.

3. PAD Implementation Details: Filter out advantages near zero (threshold ≈ 0.01), then resample using temperature-controlled softmax distribution over remaining samples. This concentrates training on informative trajectories.

4. Curriculum Progression: Don't skip stages or overlap them. The three-stage design ensures: (1) reasoning foundation, (2) multimodal alignment, (3) linguistic polish. Simultaneous mixed training fails empirically.

5. Evaluation Protocol: Test performance across text-only, vision-only, and vision-language tasks. Quality improvements should generalize across modalities, indicating robust reasoning rather than overfitting to training distribution.

Reference

• Paper: Advancing Multimodal Reasoning (2506.04207)
• Cold Start: 283K curated complex reasoning examples
• Method: GRPO + Prioritized Advantage Distillation
• Results: State-of-the-art for 3B/7B open-source models on multimodal reasoning