thinking-with-video-multimodal-reasoning

name	thinking-with-video-multimodal-reasoning
title	Thinking with Video: Video Generation as Multimodal Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.04570
keywords	["Video Generation","Multimodal Reasoning","Thinking Mechanisms","Problem Solving","Dynamic Representation"]
description	Leverage video generation models as unified multimodal reasoning engines that bridge text and vision by embedding reasoning as dynamic visual processes, enabling models to tackle both spatial puzzles and mathematical problems within a single temporal framework.

Title: Use Video Generation as a Universal Reasoning Modality

Text-centric reasoning (thinking with text) excels at logic but struggles with spatial problems. Image-centric reasoning (thinking with images) handles space but lacks temporal dynamics. Video reasoning bridges both: generate videos showing problem-solving steps, with text embedded in frames. This unifies multimodal understanding within a single continuous temporal process.

The key insight is that video naturally represents processes—drawing, moving, transforming—that neither text nor static images capture well.

Core Concept

Video as Unified Reasoning Substrate:

• Dynamic Representation: Video captures continuous transformations over time
• Text Embedding: Embed reasoning text within video frames (e.g., solving equations step-by-step)
• Spatial Reasoning: Show spatial manipulations explicitly (moving, rotating, drawing)
• Temporal Reasoning: Natural representation of sequences and processes
• Single Model: One video generation model handles diverse reasoning tasks

Architecture Overview

• Video Generation Model: Pre-trained (Sora-2 or similar) for high-fidelity generation
• Task-Specific Prompting: Frame problems as video generation tasks
• Text-in-Video Encoding: Embed reasoning steps as video content
• Evaluation: Both spatial (puzzle accuracy) and mathematical (answer correctness)

Implementation Steps

1. Formulate Problem as Video Generation Task

Convert different problem types into video generation prompts.

class VideoReasoningPromptFormatting:
    @staticmethod
    def spatial_puzzle_to_video(puzzle_description):
        """Convert spatial puzzle to video generation prompt"""
        prompt = f"""Generate a video showing the solution to this puzzle:
        {puzzle_description}

        The video should:
        1. Start with the puzzle state
        2. Show step-by-step manipulations
        3. End with the solution
        4. Include visual indicators (arrows, highlights) showing the transformations
        """
         prompt


     ():
        
        prompt = 
         prompt


     ():
        
        prompt = 
         prompt