deepeyesv2-agentic-multimodal-tool-use

name	deepeyesv2-agentic-multimodal-tool-use
title	DeepEyesV2: Toward Agentic Multimodal Model
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.05271
keywords	["Multimodal AI","Agent Reasoning","Tool Use","Code Execution","Web Search Integration"]
description	Train multimodal agents to dynamically invoke tools (code execution, web search) within reasoning loops through a two-stage pipeline combining cold-start supervised learning with reinforcement learning—enabling task-adaptive tool invocation for perception, reasoning, and retrieval tasks.

Train Agentic Multimodal Models with Dynamic Tool Invocation

Multimodal models excel at perception but struggle with complex reasoning requiring external tools. DeepEyesV2 solves this through a two-stage training approach: first establishing robust tool-use foundations via supervised learning on curated data, then refining tool invocation decisions through reinforcement learning. The result is a model that adaptively chooses when and which tools to use based on task requirements.

The key insight is that direct reinforcement learning fails without foundational tool-use competence—models either abandon code generation or engage in reward hacking. By decoupling cold-start training (establishing reliability) from RL refinement (optimizing invocation), the system achieves both robustness and flexibility.

Core Concept

DeepEyesV2 architecture treats code execution and web search as interchangeable, context-dependent tools. The model generates reasoning plans, decides whether direct reasoning suffices or tool invocation is necessary, and can emit executable Python code or structured search queries. Tool outputs become observations fed back into the reasoning context for continued iteration.

The two-stage training strategy addresses a fundamental challenge: naive RL on tool invocation fails because models lack foundational competence. Cold-start supervised training on high-quality, curated data establishes this foundation before RL refines tool selection.

Architecture Overview

• Multimodal Input Handler: Processes images and text queries simultaneously
• Tool Invocation Selector: Decides whether to reason directly or invoke tools (code/search)
• Code Executor: Runs Python operations on images (crop, numerical analysis, marking, enhancement)
• Web Search Module: Retrieves relevant webpages via SerpAPI for information-seeking tasks
• Context Manager: Converts tool outputs to observations and appends to reasoning context
• Two-Stage Learning Pipeline: Cold-start SFT followed by RL refinement

Implementation Steps

Step 1: Data Curation for Cold-Start Training

Construct a diverse, curated dataset emphasizing cases where tools improve performance. Filter by difficulty (unsolvable by base models) and tool necessity (tools required for correctness).

def curate_tool_training_data(tasks, base_model, tool_types):
    """
    Curate high-quality tool-use training data.

    Args:
        tasks: Pool of training tasks with images, questions, answers
        base_model: Reference model for difficulty filtering
        tool_types: Available tools (code, search, etc.)

    Returns:
        curated_dataset: Filtered tasks with tool-annotated solutions
    """
    curated = []

     task  tasks:
        
        solution = generate_solution_with_tools(
            task, tool_types, model=
        )

        
        base_solution = base_model.solve(task)
         base_solution.correct:
              

        
         solution.correct  solution.uses_tools:
            
             validate_code_execution(solution.code):
                curated.append((task, solution))

     curated