m3-bench-tool-use

name	m3-bench-tool-use
title	M3-Bench: Multi-Modal Multi-Hop Multi-Threaded Tool-Using MLLM Agent Benchmark
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.17729
keywords	["Multimodal Agents","Tool Use","Benchmark Design","Agent Evaluation","MCP Protocol"]
description	Design and evaluate multimodal agents for tool use with M3-Bench: assess three interconnected dimensions (multi-modal grounding, multi-hop causality, multi-threaded parallelism) using similarity-bucketed Hungarian alignment for transparent tool call evaluation without LLM judges.

M3-Bench: Evaluating Multimodal Tool-Using Agents

Existing agent benchmarks typically measure single-capability tasks. This skill demonstrates how to design M3-Bench, a comprehensive evaluation framework for multimodal agents that simultaneously tests three critical capabilities: grounding visual information for tool selection, maintaining causal chains across multiple steps, and recognizing parallelizable operations for concurrent execution.

The core innovation is similarity-bucketed Hungarian alignment—transparent tool evaluation without relying on subjective LLM judges—enabling auditable assessment of agent reasoning.

Core Concept

M3-Bench evaluates agents on three interconnected dimensions:

1. Multi-Modal Capability: Agent grounds decisions in image+text inputs before selecting tools
2. Multi-Hop Reasoning: Agent maintains causal dependency chains across sequential steps
3. Multi-Threaded Execution: Agent recognizes independent operations that can execute in parallel

Architecture Overview

• Task Specification Format: Defines multi-modal inputs, sequential and parallel operations, ground truth tool calls
• Agent Interface: Standard Model Context Protocol (MCP) for tool definition
• Evaluation Metrics: Similarity matching and Hungarian assignment for transparent scoring
• Transparency Layer: Auditable correspondence between predicted and reference tool calls
• Dimension-Specific Tests: Separate scenarios stressing each capability

Implementation Steps

Building and evaluating with M3-Bench requires task design, execution, and assessment.

1. Define Task Specification Format

Create structured format capturing multi-modal, multi-hop, multi-threaded properties.

from dataclasses import dataclass
from typing import List, Dict, Any

@dataclass
class ToolCall:
    """Represents a single tool invocation."""
    tool_name: str
    arguments: Dict[str, ]
    returns:  = 
    execution_time:  = 


 :
    
    step_id: 
    tools_to_call: [ToolCall]
    depends_on: [] =   


 :
    
    step_ids: []
    operations: [SequentialStep]


 :
    
    task_id: 
    description: 

    
    image:   
    text_query: 

    
    sequential_steps: [SequentialStep]
    parallel_groups: [ParallelGroup]
    expected_final_answer: 

    
    difficulty_level:   
    required_capabilities: []  

     ():
        
         {
            : .image,
            : .text_query,
            : .description
        }