sail-rl-adaptive-reasoning

name	sail-rl-adaptive-reasoning
title	SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.02280
keywords	["Multimodal RL","Thinking Control","Adaptive Reasoning","Reward Shaping","Post-Training"]
description	Teach multimodal models to determine when deep reasoning is necessary and how to reason effectively through dual-reward reinforcement learning, preventing both overthinking on simple tasks and underthinking on complex ones while reducing hallucinations.

Title: Learn Adaptive Reasoning Allocation Through Dual Reward Signals

Reasoning enhances accuracy but costs tokens and latency. SAIL-RL teaches models to reason selectively: answer simple questions directly, think deeply on complex ones. The framework uses two complementary rewards: (1) thinking reward evaluates reasoning quality (factual grounding, logical coherence), and (2) judging reward determines task complexity. Combined through cascading logic, these rewards create a system that learns when thinking helps and when it hurts.

The approach balances accuracy against efficiency at inference time.

Core Concept

Dual-Reward Reasoning Control:

• Thinking Reward: Evaluates reasoning quality (factual, coherent, consistent)
• Judging Reward: Determines when to apply reasoning vs. direct answers
• Cascading Logic: Combined multiplicatively, nullifying bad components
• Discrete Rewards: Binary signals provide sharper gradients than continuous scores
• Multimodal Integration: Works with text and images, maintaining visual grounding

Architecture Overview

• Judge Module: Determines task complexity (simple vs. complex)
• Thinking Module: Generates reasoning (if judge says necessary)
• Answer Module: Produces final output
• Reward Computation: Three-component reward (judge × think × answer)
• DAPO Training: Discrete preference optimization with binary rewards

Implementation Steps

1. Implement Judge-Think-Answer Architecture

Structure model to output decision → reasoning → answer.

class JudgeThinkerAnswerer(nn.Module):
    def __init__(self, vlm_model):
        self.vlm = vlm_model
        self.judge = nn.Linear(vlm_model.hidden_dim, 2)  # simple or complex
        self.thinker = ThinkingModule(vlm_model)
        self.answerer = AnswerModule(vlm_model)

    def ():
        
        features = .vlm.encode(image=image, text=text)

        
        judge_logits = .judge(features)
        judge_probs = F.softmax(judge_logits, dim=-)
        is_complex = judge_probs[:, ] >   

        
        thinking = []
         i, complex_flag  (is_complex):
             complex_flag:
                thought = .thinker(features[i:i+])
                thinking.append(thought)
            :
                thinking.append()

        
        answers = []
         i, thought  (thinking):
             thought:
                
                answer = .answerer(features[i:i+], context=thought)
            :
                
                answer = .answerer(features[i:i+], context=)
            answers.append(answer)

         {
            : is_complex,
            : thinking,
            : answers
        }