mimo-vl-vision-language

name	mimo-vl-vision-language
title	MiMo-VL: Mixed On-Policy Reinforcement Learning for Vision-Language Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.03569
keywords	["vision-language","multimodal","reinforcement-learning","reasoning","chain-of-thought"]
description	Build state-of-the-art 7B multimodal models by combining four-stage vision-language pretraining with mixed on-policy RL integrating verifiable and human feedback rewards.

MiMo-VL Technical Report

Core Concept

MiMo-VL demonstrates that 7B vision-language models can achieve state-of-the-art results across 50+ benchmarks through strategic data integration and mixed on-policy reinforcement learning. The key insight: incorporating synthetic reasoning data with long chain-of-thought during pretraining—rather than treating it as supplementary fine-tuning—substantially improves performance without saturation effects.

Architecture Overview

• Vision Encoder: Qwen2.5-ViT transformer backbone processing visual input
• Projection Layer: MLP projector aligning visual and linguistic embeddings
• Language Model: MiMo-7B base language model for token generation
• Training Pipeline: Four sequential stages optimizing different modality aspects
  • Stage 1-2: Projector warmup and vision-language alignment
  • Stage 3: General multimodal pretraining across diverse data types
  • Stage 4: Long-context SFT integrating reasoning and extended sequences
  • Post-training: Mixed on-policy RL with multiple reward signals
• Mixed Reward System: Combines verifiable rewards (reasoning correctness, grounding accuracy, counting validity) with human preference alignment

Implementation

Step 1: Prepare Diverse Pretraining Data

# Data composition for 2.4 trillion tokens
data_recipes = {
    "captions": 0.3,           # Image-text pairs
    "interleaved_content": 0.2, # Mixed visual-textual sequences
    "ocr": 0.15,               # Text detection and recognition
    "grounding": 0.15,         # Visual region-text alignment
    "video": 0.1,              # Temporal visual understanding
    "gui": 0.05,               # Interface interaction
    "reasoning": 0.05          # Synthetic CoT with long chains
}

# Quality-aware data loader
class QualityAwareDataLoader:
    def __init__(self, recipes, quality_threshold=0.85):
        self.recipes = recipes
        self.quality_threshold = quality_threshold

    def validate_sample(self, sample):
        # Check image-text alignment scores
        # Verify OCR accuracy
        # Validate reasoning chain coherence
        return sample['quality_score'] >= self.quality_threshold

Step 2: Configure Four-Stage Pretraining

# Stage 1-2: Projector warmup and alignment
warmup_config = {
    "stages": [1, 2],
    "duration_tokens": 50e9,
    "learning_rate": 1e-3,
    "focus": ["vision_encoder_freeze", "projector_train"],
}

# Stage 3: General multimodal pretraining
general_config = {
    "stage": 3,
    "duration_tokens": 1.5e12,
    "learning_rate": 1e-4,
    "data_mix": data_recipes,
    "unfrozen_components": ["all"],
}

# Stage 4: Reasoning and long-context SFT
sft_config = {
    "stage": 4,
    "duration_tokens": 0.8e12,
    "learning_rate": 2e-5,
    "focus": ["chain_of_thought", "long_context"],
    "reasoning_data_weight": 0.3,
}

Step 3: Implement Mixed On-Policy RL

# Multiple reward functions addressing different capabilities
class MultiRewardEvaluator:
    def __init__(self):
        self.reasoning_verifier = ReasoningChecker()
        self.grounding_validator = GroundingValidator()
        self.counting_checker = CountingVerifier()

    def compute_rewards(self, output, ground_truth):
        rewards = {}

        # Verifiable reward: reasoning correctness
        reasoning_score = self.reasoning_verifier.check(
            output['reasoning_chain'],
            ground_truth['correct_answer']
        )
        rewards['reasoning'] = reasoning_score

        # Verifiable reward: grounding accuracy
        grounding_score = self.grounding_validator.evaluate(
            output['bounding_boxes'],
            ground_truth['ground_regions']
        )
        rewards['grounding'] = grounding_score

        # Verifiable reward: counting precision
        counting_score = self.counting_checker.verify(
            output['count'],
            ground_truth['true_count']
        )
        rewards['counting'] = counting_score

        # Human preference alignment (from human feedback data)
        human_score = output['human_preference_signal']
        rewards['human'] = human_score

        # Combined reward with adaptive weighting
        combined = (
            0.4 * reasoning_score +
            0.3 * grounding_score +
            0.2 * counting_score +
            0.1 * human_score
        )

         combined, rewards


 ():
    
    outputs = [model.generate(batch[], temp=t)
                t  [, , ]]

    
    rewards = [reward_fn(out, batch[])
                out  outputs]

    
    advantages = rewards - np.mean(rewards)
    advantages = (advantages - np.mean(advantages)) / (np.std(advantages) + )

    
    log_probs = [model.log_probability(out)  out  outputs]
    pg_loss = (-lp * adv  lp, adv  (log_probs, advantages))

    optimizer.zero_grad()
    pg_loss.backward()
    optimizer.step()

     pg_loss.item()

Step 4: Handle Response Length Disparities

# Reasoning tasks favor longer outputs; grounding tasks prefer shorter
class AdaptiveResponseLengthTraining:
    def __init__(self):
        self.task_length_prefs = {
            "reasoning": {"min": 500, "target": 1200},
            "grounding": {"min": 50, "target": 200},
            "counting": {"min": 20, "target": 80},
        }

    def compute_length_adjusted_loss(self, output, task_type, rewards):
        output_length = len(output.split())
        target_range = self.task_length_prefs[task_type]

        # Soft penalty for length deviation
        ideal_length = target_range['target']
        length_penalty = ((output_length - ideal_length) / 100) ** 2

        # Adjust reward based on task-specific expectations
        adjusted_reward = rewards - 0.05 * length_penalty

        return adjusted_reward

Practical Guidance

1. Data Integration Priority: Incorporate reasoning data directly into pretraining stages rather than treating as post-hoc fine-tuning. The 2.4 trillion token budget should allocate 5-10% to synthetic reasoning chains.

2. Training Curriculum: Follow the four-stage progression strictly. Early stages establish vision-language alignment; later stages build reasoning capacity without degrading perceptual skills.

3. Reward Signal Weighting: Initialize with equal weights across reward types, then adjust based on evaluation performance. Reasoning correctness typically improves fastest initially.

4. Addressing Task Interference: Use separate decoder heads or task-specific LoRA adapters to handle the tension between long-output reasoning tasks and short-output grounding tasks.

5. Evaluation at Scale: Benchmark across 50+ tasks spanning perception, reasoning, and grounding rather than focusing on narrow metrics. MiMo-VL's strength is cross-domain generalization.

Reference

• Paper: MiMo-VL Technical Report (2506.03569)
• Key Architecture: Qwen2.5-VL foundation with enhanced RL pipeline
• Benchmark Suite: 50+ standardized evaluations with reproducible prompts
• Public Release: MiMo-VL-7B-SFT and MiMo-VL-7B-RL models and datasets