Design and evaluate multimodal agents for tool use with M3-Bench: assess three interconnected dimensions (multi-modal grounding, multi-hop causality, multi-threaded parallelism) using similarity-bucketed Hungarian alignment for transparent tool call evaluation without LLM judges.
Design and evaluate multimodal agents for tool use with M3-Bench: assess three interconnected dimensions (multi-modal grounding, multi-hop causality, multi-threaded parallelism) using similarity-bucketed Hungarian alignment for transparent tool call evaluation without LLM judges.
M3-Bench: Evaluating Multimodal Tool-Using Agents
Existing agent benchmarks typically measure single-capability tasks. This skill demonstrates how to design M3-Bench, a comprehensive evaluation framework for multimodal agents that simultaneously tests three critical capabilities: grounding visual information for tool selection, maintaining causal chains across multiple steps, and recognizing parallelizable operations for concurrent execution.
The core innovation is similarity-bucketed Hungarian alignment—transparent tool evaluation without relying on subjective LLM judges—enabling auditable assessment of agent reasoning.
Core Concept
M3-Bench evaluates agents on three interconnected dimensions:
Multi-Modal Capability: Agent grounds decisions in image+text inputs before selecting tools
Multi-Hop Reasoning: Agent maintains causal dependency chains across sequential steps
Multi-Threaded Execution: Agent recognizes independent operations that can execute in parallel
Architecture Overview
Task Specification Format: Defines multi-modal inputs, sequential and parallel operations, ground truth tool calls
Agent Interface: Standard Model Context Protocol (MCP) for tool definition
Evaluation Metrics: Similarity matching and Hungarian assignment for transparent scoring
Transparency Layer: Auditable correspondence between predicted and reference tool calls
Dimension-Specific Tests: Separate scenarios stressing each capability
Implementation Steps
Building and evaluating with M3-Bench requires task design, execution, and assessment.
1. Define Task Specification Format
Create structured format capturing multi-modal, multi-hop, multi-threaded properties.
Test agent's ability to maintain causal dependencies.
defevaluate_multi_hop_reasoning(task: M3BenchTask, agent_execution_trace: List[Dict]) -> Dict:
"""
Evaluate agent's multi-hop reasoning capability.
Checks whether agent maintains correct causal orderings.
Args:
task: M3-Bench task with ground truth sequential steps
agent_execution_trace: Recorded tool calls and results
Returns:
metrics: Multi-hop reasoning evaluation metrics
"""# Extract dependency graph from ground truth
ground_truth_deps = {}
for step in task.sequential_steps:
ground_truth_deps[step.step_id] = step.depends_on or []
# Extract agent's implicit dependencies from execution order
agent_execution_order = [call['tool'] for call in agent_execution_trace]
# Check: did agent execute in valid topological order?defis_valid_execution_order(execution_order, dependencies):
"""Check if execution respects dependencies."""
executed = set()
for operation in execution_order:
# Get dependencies for this operation
deps = dependencies.get(operation, [])
# Check if all dependencies executedifnotall(dep in executed for dep in deps):
returnFalse
executed.add(operation)
returnTrue
valid_order = is_valid_execution_order(agent_execution_order, ground_truth_deps)
# Measure: how many causal relationships did agent respect?
respect_count = 0
total_deps = 0for step_id, deps in ground_truth_deps.items():
for dep in deps:
total_deps += 1# Check if dep executed before step_id in agent's tracetry:
dep_idx = agent_execution_order.index(dep)
step_idx = agent_execution_order.index(step_id)
if dep_idx < step_idx:
respect_count += 1except ValueError:
pass# Tool not in trace
causal_respect_rate = respect_count / max(total_deps, 1)
return {
'valid_execution_order': valid_order,
'causal_respect_rate': causal_respect_rate,
'steps_executed': len(agent_execution_order),
'ground_truth_steps': len(task.sequential_steps)
}
5. Evaluate Multi-Threaded Parallelism
Assess agent's ability to identify parallelizable operations.
defevaluate_parallelism_recognition(
task: M3BenchTask,
agent_execution_trace: List[Dict]
) -> Dict:
"""
Evaluate agent's ability to recognize parallelizable operations.
Args:
task: Contains ground truth parallel groups
agent_execution_trace: Agent's actual execution
Returns:
metrics: Parallelism evaluation metrics
"""# Extract agent's implicit parallelization from timestamps
call_timeline = []
for call in agent_execution_trace:
call_timeline.append({
'tool': call['tool'],
'start': call.get('timestamp', 0),
'end': call.get('timestamp', 0) + call.get('execution_time', 0)
})
# Find concurrent calls (overlapping time ranges)
concurrent_groups = []
for i, call1 inenumerate(call_timeline):
concurrent_group = [i]
for j, call2 inenumerate(call_timeline[i+1:], start=i+1):
# Check overlap: call1.end > call2.start and call1.start < call2.endif call1['end'] > call2['start'] and call1['start'] < call2['end']:
concurrent_group.append(j)
iflen(concurrent_group) > 1:
concurrent_groups.append(concurrent_group)
# Compare to ground truth parallel groups
ground_truth_parallel = []
for group in task.parallel_groups:
ground_truth_parallel.extend(group.step_ids)
# Measure: how many parallelizable operations did agent recognize?
recognized_parallelizable = 0for group_indices in concurrent_groups:
tools_in_group = [call_timeline[idx]['tool'] for idx in group_indices]
# Check if these are actually parallelizable in ground truthfor step in task.sequential_steps:
if step.step_id in ground_truth_parallel:
ifany(tool in [t.tool_name for t in step.tools_to_call] for tool in tools_in_group):
recognized_parallelizable += 1
parallelism_rate = recognized_parallelizable / max(len(ground_truth_parallel), 1)
return {
'concurrent_groups_found': len(concurrent_groups),
'parallelizable_operations_recognized': recognized_parallelizable,
'ground_truth_parallelizable': len(ground_truth_parallel),
'parallelism_recognition_rate': parallelism_rate
}
6. Compute Aggregate M3 Score
Combine the three capabilities into unified assessment.
Transparency: Hungarian matching makes evaluation auditable
Composability: Test each dimension (M, M, M) independently or combined
Scalability: Vectorize similarity computation for large benchmarks
Benchmark Construction Tips:
Create tasks at multiple difficulty levels (easy→hard)
Vary number of sequential steps (3, 5, 10+)
Vary parallelism opportunities (none, some, many)
Include both visual-heavy and text-heavy tasks for balanced evaluation
Integration with LLMs:
M3-Bench works with any agent implementing MCP protocol. Supports Claude, open-source models, or custom agents through standard tool interface.
