modomod-multimodal-rl

name	modomodо-multimodal-rl
title	MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2505.24871
keywords	["Multimodal RL","Data Mixture","Vision-Language","RLVR","Multi-Task"]
description	Optimize data mixtures across diverse vision-language domains when applying RL with verifiable rewards to multimodal LLMs, balancing task-specific performance with generalization.

Optimize Multi-Domain Data Mixtures for Multimodal RL Training

Multimodal large language models (MLLMs) must handle heterogeneous tasks: visual QA, image captioning, spatial reasoning, scene understanding. When applying reinforcement learning with verifiable rewards (RLVR), a critical challenge emerges: how should you balance training data across these diverse domains?

MoDoMoDo addresses this through systematic data mixture optimization: finding the right proportion of training from each domain so the model learns task-specific skills without forgetting others. Unlike single-domain RL, multimodal RL requires careful orchestration of data flows to prevent task interference and maximize transfer.

Core Concept

MoDoMoDo optimizes task mixture ratios for multimodal RL:

• Domain-specific capability: Each domain requires unique visual, logical, and spatial skills
• Interference avoidance: Training hard on one domain shouldn't degrade others
• Curriculum design: Sequence domains in ways that enable positive transfer
• Metric tracking: Monitor per-domain performance during training
• Adaptive mixture: Adjust ratios based on observed performance gaps
• Generalization: Find mixtures that improve broader capability

The key insight is that heterogeneous task distributions demand explicit mixture management—uniform random sampling isn't optimal.

Architecture Overview

• Task taxonomy: Categorize multimodal tasks by required capabilities
• Capacity tracking: Monitor model capacity utilization per task
• Performance monitoring: Track metrics separately for each domain
• Interference detection: Identify when training on one domain hurts others
• Mixture scheduler: Dynamically adjust domain sampling probabilities
• Transfer measurement: Quantify positive/negative transfer between domains
• Consolidation strategy: Ensure all domains reach target performance

Implementation

Build a data mixture optimizer for multimodal RL:

# MoDoMoDo: Multi-Domain Data Mixture Optimization
import torch
from torch.utils.data import DataLoader, WeightedRandomSampler
from typing import Dict, List, Tuple
import numpy as np

class MultiDomainDataMixer:
    """
    Optimize data mixture ratios for multimodal RL training across domains.
    """
    def __init__(self, domain_names: List[str], initial_weights: Dict[str, float] = None):
        self.domains = domain_names
        self.num_domains = len(domain_names)

        # Initialize mixture weights (uniform by default)
        if initial_weights is None:
            initial_weights = {d: 1.0 / self.num_domains for d in domain_names}

        self.mixture_weights = initial_weights
        self.performance_history = {d: [] for d in domain_names}
        self.interference_matrix = np.zeros((self.num_domains, self.num_domains))

    def create_mixed_dataloader(self, domain_datasets: Dict[str, torch.utils.data.Dataset],
                                batch_size:  = , training:  = ):
        
         training:
            
            adjusted_weights = ._adjust_weights_by_performance()

            
            all_indices = []
            all_domain_labels = []

            
            total_samples = 
            dataset_offsets = {}
            concatenated_data = []

            offset = 
             domain  .domains:
                dataset_offsets[domain] = offset
                concatenated_data.extend(domain_datasets[domain])
                offset += (domain_datasets[domain])
                total_samples += (domain_datasets[domain])

            
            sample_weights = [adjusted_weights.get(d,  / .num_domains)
                              d  .domains]

            
            samples_per_domain = {
                d: (adjusted_weights[d] * total_samples)
                 d  .domains
            }

            
            indices = []
             domain  .domains:
                domain_size = (domain_datasets[domain])
                domain_indices = np.random.choice(
                    domain_size,
                    size=(samples_per_domain[domain], domain_size),
                    replace=
                )
                indices.extend(domain_indices + dataset_offsets[domain])

            
            np.random.shuffle(indices)

            
             torch.utils.data  Subset
            subset = Subset(concatenated_data, indices)
            dataloader = DataLoader(subset, batch_size=batch_size, shuffle=)

        :
            
            concatenated_data = []
             domain  .domains:
                concatenated_data.extend(domain_datasets[domain])

            dataloader = DataLoader(concatenated_data, batch_size=batch_size, shuffle=)

         dataloader

     ():
        
        
         domain, metric  per_domain_metrics.items():
            .performance_history[domain].append(metric)

        
        improvement_rates = {}
         domain  .domains:
            hist = .performance_history[domain]
             (hist) >= :
                recent = np.mean(hist[-:])  (hist) >=   hist[-]
                older = np.mean(hist[:])  (hist) >=   hist[]
                improvement = recent - older
                improvement_rates[domain] = improvement
            :
                improvement_rates[domain] = 

        
        mean_improvement = np.mean((improvement_rates.values()))

        new_weights = {}
         domain  .domains:
             improvement_rates[domain] < mean_improvement:
                
                new_weights[domain] = .mixture_weights[domain] * 
            :
                
                new_weights[domain] = .mixture_weights[domain] * 

        
        total_weight = (new_weights.values())
        .mixture_weights = {d: w / total_weight  d, w  new_weights.items()}

        ()

     () -> [, ]:
        
        
        latest_metrics = {}
         domain  .domains:
             .performance_history[domain]:
                latest_metrics[domain] = .performance_history[domain][-]
            :
                latest_metrics[domain] =   

        
        adjusted_weights = {}
         domain  .domains:
            
            target_performance = 
            gap = (, target_performance - latest_metrics[domain])
            adjusted_weights[domain] = ( + gap) * .mixture_weights[domain]

        
        total = (adjusted_weights.values())
        adjusted_weights = {d: w / total  d, w  adjusted_weights.items()}

         adjusted_weights

     ():
        
         d1  .domains:
             d2  .domains:
                 d1 != d2:
                    
                    d2_degradation = before_metrics[d2] - after_metrics[d2]
                     d2_degradation > :  
                        .interference_matrix[.domains.index(d1),
                                              .domains.index(d2)] += 

     () -> []:
        
        
        
        interference_sums = .interference_matrix.(axis=)
        order_indices = np.argsort(interference_sums)
         [.domains[i]  i  order_indices]

Implement RLVR training with mixture management:

def train_mllm_with_mixed_domains(model, domain_datasets: Dict[str, torch.utils.data.Dataset],
                                  num_epochs=10, target_domains: Dict[str, float] = None):
    """
    Train multimodal LLM using RL with verifiable rewards across multiple domains.
    """
    mixer = MultiDomainDataMixer(list(domain_datasets.keys()))

    # Initialize target performance per domain
    if target_domains is None:
        target_domains = {d: 0.8 for d in domain_datasets.keys()}

    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)

    for epoch in range(num_epochs):
        # Create mixed dataloader
        train_loader = mixer.create_mixed_dataloader(domain_datasets, batch_size=32)

        epoch_metrics = {d: [] for d in domain_datasets.keys()}
        before_epoch_metrics = {d: 0.5 for d in domain_datasets.keys()}  # Placeholder

        for batch_idx, batch in enumerate(train_loader):
            images = batch['images']
            questions = batch['questions']
            ground_truths = batch['ground_truths']
            domain_labels = batch['domain']  # Which domain this batch is from

            # Forward pass: get model predictions
            outputs = model(images, questions)

            # Compute per-domain rewards (verifiable)
            per_domain_rewards = {}
             domain  domain_datasets.keys():
                domain_mask = domain_labels == domain
                 domain_mask.() > :
                    domain_outputs = outputs[domain_mask]
                    domain_gts = ground_truths[domain_mask]

                    
                     domain == :
                        reward = verify_vqa_answer(domain_outputs, domain_gts)
                     domain == :
                        reward = verify_caption_quality(domain_outputs, domain_gts)
                     domain == :
                        reward = verify_spatial_reasoning(domain_outputs, domain_gts)
                    :
                        reward = verify_generic_task(domain_outputs, domain_gts)

                    per_domain_rewards[domain] = reward.mean().item()
                    epoch_metrics[domain].extend(reward.cpu().numpy())

            
            mixture_weights = mixer.mixture_weights
            total_loss = 

             domain, reward  per_domain_rewards.items():
                domain_loss = -reward  
                weighted_loss = domain_loss * mixture_weights[domain]
                total_loss += weighted_loss

            optimizer.zero_grad()
            total_loss.backward()
            optimizer.step()

        
        avg_domain_metrics = {d: np.mean(epoch_metrics[d])  epoch_metrics[d]  
                              d  domain_datasets.keys()}
        mixer.update_mixture_from_performance(avg_domain_metrics)

        
        mixer.detect_interference(before_epoch_metrics, avg_domain_metrics)

        
        ()
         domain, metric  avg_domain_metrics.items():
            target = target_domains[domain]
            ()

        before_epoch_metrics = avg_domain_metrics

     mixer

Practical Guidance

Parameter	Typical Range	Notes
Initial mixture ratio	Uniform or inverse performance	Start balanced, then adapt
Weight adjustment speed	1.05 - 1.2 per epoch	Too fast = instability; too slow = slow adaptation
Target performance gap threshold	0.02 - 0.05	When to increase domain weight
Interference detection threshold	0.02 drop	What counts as negative transfer
Num domains	2 - 8	More domains = more complex optimization

When to use MoDoMoDo:

• Training MLLMs on multiple heterogeneous vision-language tasks
• RL with verifiable rewards across diverse domains
• Need to prevent catastrophic forgetting of some tasks
• Want to measure and understand task interference
• Building robust multimodal foundation models

When NOT to use:

• Single task training (mixture optimization not relevant)
• Data is naturally uniform in distribution
• Tasks are already well-balanced in importance
• Computational budget for tracking per-domain metrics is tight
• Using pure supervised learning without RL

Common pitfalls:

• Not tracking per-domain metrics separately (can't detect problems)
• Mixture weights diverge too far (one domain gets almost nothing)
• Ignoring negative transfer signals (interference goes unaddressed)
• Adjusting weights too aggressively (training becomes unstable)
• Using same reward for all domains (domain-specific verification critical)
• Not considering curriculum order (some domains should be learned first)
• Interference detection threshold too loose (false positives)

Reference

MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning https://arxiv.org/abs/2505.24871