internvl3-5-multimodal-cascade-rl

name	internvl3-5-multimodal-cascade-rl
title	InternVL3.5: Cascade RL and Visual Resolution Router for Multimodal Efficiency
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.18265
keywords	["multimodal-models","cascade-reinforcement-learning","visual-efficiency","reasoning","inference-optimization"]
description	Enhance multimodal models through cascade RL for reasoning improvement and visual resolution routing for inference efficiency, achieving 16% reasoning gains and 4.05x speedup.

InternVL3.5: Cascade RL and Visual Resolution Router

Core Concept

InternVL3.5 advances multimodal AI through two complementary techniques: Cascade RL for enhanced reasoning and Visual Resolution Router (ViR) for adaptive efficiency. Cascade RL combines offline RL for stable convergence with online RL for refined alignment, improving performance on visual reasoning benchmarks. ViR dynamically adjusts visual token resolution based on image complexity, maintaining performance while reducing computation. Together, these achieve 16% reasoning improvement and 4.05x inference speedup.

Architecture Overview

• Cascade RL Framework: Two-stage training (offline → online)
• Visual Resolution Router: Dynamic resolution selection per image
• Decoupled Vision-Language Deployment: Distributed inference
• Reasoning Enhancement: Improved visual understanding
• Efficiency Optimization: Token reduction without quality loss

Implementation Steps

1. Implement Cascade RL Framework

Two-stage training for stable and effective policy learning:

import torch
import torch.nn as nn
from typing import Dict, List, Tuple

class CascadeRLTrainer:
    """Two-stage RL: offline stable convergence + online refinement."""

    def __init__(
        self,
        model: "MultimodalLLM",
        offline_steps: int = 1000,
        online_steps: int = 500,
        offline_lr: float = 1e-5,
        online_lr: float = 5e-6
    ):
        self.model = model
        self.offline_steps = offline_steps
        self.online_steps = online_steps

        
        .offline_optimizer = torch.optim.Adam(model.parameters(), lr=offline_lr)
        .online_optimizer = torch.optim.Adam(model.parameters(), lr=online_lr)

        .training_logs = {: [], : []}

     () -> [, ]:
        
        ()

        metrics = {: [], : []}

         step  (.offline_steps):
            
            batch = ._sample_batch(dataset, batch_size=)

            
            outputs = .model(batch[], batch[])
            logits = outputs.logits

            
            rewards = ._compute_correctness_reward(
                outputs.sequences,
                batch[]
            )

            
            
            loss = ._compute_offline_loss(logits, rewards)

            
            .offline_optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(.model.parameters(), )
            .offline_optimizer.step()

            metrics[].append(loss.item())
            metrics[].append(rewards.mean().item())

             (step + ) %  == :
                avg_loss = (metrics[][-:]) / 
                avg_reward = (metrics[][-:]) / 
                ()

        .training_logs[] = metrics
         {: (metrics[]) / (metrics[]),
                : (metrics[]) / (metrics[])}

     () -> [, ]:
        
        ()

        metrics = {: [], : []}

         episode  (num_episodes):
            
            state = env.reset()
            episode_reward = 
            episode_steps = 

             step  (.online_steps):
                
                action = .model.generate(
                    state[],
                    state[],
                    max_tokens=
                )

                
                next_state, reward, done = env.step(action)

                
                ._store_transition(state, action, reward, next_state, done)

                
                loss = ._compute_online_loss(state, action, reward)

                .online_optimizer.zero_grad()
                loss.backward()
                .online_optimizer.step()

                episode_reward += reward
                episode_steps += 
                state = next_state

                 done:
                    

            metrics[].append(episode_reward)
            metrics[].append(episode_steps)

             (episode + ) %  == :
                avg_reward = (metrics[][-:]) / 
                ()

        .training_logs[] = metrics
         {: (metrics[]) / (metrics[]),
                : (metrics[]) / (metrics[])}

     () -> torch.Tensor:
        
        
        log_probs = torch.log_softmax(logits, dim=-)

        
        policy_loss = -(log_probs.mean(dim=) * rewards).mean()

        
        
        
        regularization =   

         policy_loss + regularization

     () -> torch.Tensor:
        
        
        outputs = .model(state[], state[])
        logits = outputs.logits

        
        action_log_prob = ._get_log_prob_of_action(action, logits)

        
        loss = -action_log_prob * reward

         loss

     () -> torch.Tensor:
        
        rewards = []
         pred, target  (predictions, targets):
            
            correct = pred.strip() == target.strip()
            reward =   correct  -
            rewards.append(reward)

         torch.tensor(rewards)

     ():
        
         random
        batch_samples = random.sample(dataset, (batch_size, (dataset)))
         {
            : [s[]  s  batch_samples],
            : [s[]  s  batch_samples],
            : [s[]  s  batch_samples]
        }

     ():
        

     () -> torch.Tensor: