thinkmorph-interleaved-multimodal-reasoning

name	thinkmorph-interleaved-multimodal-reasoning
title	ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.27492
keywords	["Multimodal Reasoning","Chain-of-Thought","Vision-Language","Interleaved Tokens","Emergent Behavior"]
description	Train unified models to generate interleaved reasoning steps combining text and image thoughts as complementary modalities, enabling adaptive behavior like autonomous mode-switching and superior test-time scaling for vision-centric tasks without requiring external guidance.

Title: Interleave Text and Visual Reasoning for Adaptive Problem Solving

Multimodal reasoning traditionally treats text and images as separate streams or forces them into a single unified representation. ThinkMorph demonstrates that treating text and images as complementary modalities that jointly advance reasoning toward solutions yields emergent capabilities: models spontaneously switch between text-only and visual reasoning based on task requirements, and interleaved reasoning shows better scaling curves under best-of-N sampling than text-only approaches.

The core insight is that visual reasoning excels at spatial manipulation tasks while textual reasoning maintains logical coherence. By interleaving token generation across both modalities, you create a unified problem-solving process where each modality amplifies the other's strengths.

Core Concept

Interleaved Multimodal Reasoning structures a chain-of-thought as alternating text and image thoughts, where:

• Text tokens maintain logical flow and coherence
• Image tokens represent visual manipulations (edits, annotations, highlights)
• The model learns when to generate each modality based on task requirements
• Training on only ~24K examples enables generalization to out-of-domain tasks

This approach differs from sequential multimodal reasoning (generate all text, then all images) or merged representations by creating tight coupling: each reasoning step may involve both modalities simultaneously.

Architecture Overview

• Base Model: Unified architecture (e.g., Bagel-7B) with shared token vocabulary for text and image patches
• Reasoning Modes: Three training modes: text-only, visual-only, interleaved
• Loss Functions: Dual objectives combining MSE for image reconstruction and NLL for text prediction
• Dataset: ~24K curated traces across four vision-centric task categories with human quality control
• Evaluation: Vision-centric benchmarks (SAT, MMVP, VStar, BLINK, CV-Bench) plus out-of-domain generalization tests

Implementation Steps

1. Curate High-Quality Interleaved Reasoning Traces

Create training data by collecting problem-solving sequences where humans solve vision tasks while generating both text thoughts (reasoning steps) and visual annotations (highlights, edits, overlays). Use GPT-4 to generate candidate traces, then human raters filter for coherence and correctness.

# Example structure of an interleaved trace
trace = {
    "task": ,
    : [
        {: , : },
        {: , : , : },
        {: , : },
        {: , : , : },
    ]
}