mcp-mark-comprehensive-agent-benchmark

name	mcp-mark-comprehensive-agent-benchmark
title	MCPMark: Comprehensive Benchmarking of MCP-based LLM Agents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2509.24002
keywords	["MCP","agent-evaluation","benchmarking","tool-use","LLM-agents"]
description	Evaluate LLM agents through realistic multi-turn tool-use workflows across 127 complex MCP tasks spanning CRUD operations, state management, and error handling. Use when assessing agent capabilities on real-world tool orchestration beyond shallow read-only interactions.

MCPMark: Comprehensive Benchmarking of MCP-based LLM Agents

MCPMark addresses a critical evaluation gap in LLM agent benchmarks by introducing 127 complex, real-world MCP (Model Context Protocol) tasks that require agents to execute multi-step workflows with persistent state changes, error recovery, and verification logic—moving beyond shallow read-heavy interactions.

Core Architecture

• 127 diverse tasks across 5 MCP environments (GitHub, Slack, Linear, Notion, Google Drive)
• 5 MCP servers with 55 integrated endpoints enabling realistic tool interactions
• Metrics system with pass@1 (single attempt), pass@4 (best-of-4), pass^4 (all-4-succeed) scoring
• Verification framework with programmatic state checking and rollback on test failure
• Task diversity: CRUD-balanced with ~16 agent turns per task vs. 3-7 in prior benchmarks

Implementation Steps

Create evaluation infrastructure by instantiating the benchmark:

# Initialize MCPMark benchmark with environment configuration
from mcpmark import MCPMark

benchmark = MCPMark(
    environments=["github", "slack", "linear", "notion", "googledrive"],
    num_tasks=127,
    task_timeout=300,  # seconds per task
    max_turns=20,
    verification_enabled=True
)

# Execute evaluation on target agent
results = benchmark.evaluate(
    agent=your_agent_instance,
    metrics=["pass@1", "pass@4", "pass^4"],
    parallel=False  # sequential execution preserves state integrity
)

For multi-turn workflow evaluation, construct task-specific assertions:

# Define task verification logic with state inspection
from mcpmark.tasks import Task

def verify_task_success():
    
     task_state.current_state == expected_outcome.final_state