livemcpbench-agent-tool-navigation

name	livemcpbench-agent-tool-navigation
title	LiveMCPBench - Evaluating Agents in Large-Scale Tool Ecosystems
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.01780
keywords	["benchmarking","agent-evaluation","tool-selection","mcp-tools"]
description	Benchmark framework for evaluating LLM agents navigating large-scale Model Context Protocol ecosystems with multi-tool composition across 95 daily tasks.

LiveMCPBench: Agent Navigation in Tool Ecosystems

LiveMCPBench evaluates how well LLM-based agents can discover and compose tools from large, realistic Model Context Protocol (MCP) environments. The core challenge: with thousands of available tools, agents must develop sophisticated retrieval and composition strategies. The benchmark addresses this by providing 70 MCP servers with 527 tools, real-world tasks, and an LLM-as-Judge evaluation framework that handles dynamic data and multiple valid solutions.

Core Concept

Standard agent benchmarks test tool use in toy environments with 10-50 tools. Real MCP ecosystems contain thousands of servers and tools—requiring agents to solve a dual problem: (1) retrieve relevant tools from massive search spaces, and (2) compose multiple tools coherently to solve tasks. LiveMCPBench bridges this gap by benchmarking both retrieval effectiveness and multi-tool reasoning with reproducible, task-agnostic evaluation.

Architecture Overview

• Task Design: 95 daily tasks across 6 domains (Office, Lifestyle, Leisure, Finance, Travel, Shopping) emphasizing temporal dynamics and real-time information retrieval
• LiveMCPTool Collection: 70 Docker-packaged MCP servers providing 527 tools without external API dependencies, enabling reproducible evaluation
• MCP Copilot Agent: ReACT-based agent operating as a POMDP with Route, Execute, Response operations; tool selection via joint server-tool description alignment
• LiveMCPEval: LLM-as-Judge framework using "key points" (critical subtasks) for human-aligned evaluation supporting dynamic solutions
• Tool Taxonomy: Hierarchical organization enabling efficient retrieval and composition analysis

Implementation Steps

Step 1: Build the Task Definition Framework

Design tasks that require multi-step tool composition with temporal reasoning:

from enum import Enum
from dataclasses import dataclass
from datetime import datetime
from typing import List, Dict

class TaskDomain(Enum):
    OFFICE = "office"
    LIFESTYLE = "lifestyle"
    LEISURE = "leisure"
    FINANCE = "finance"
    TRAVEL = "travel"
    SHOPPING = "shopping"

@dataclass
class Task:
    """Represents a single evaluation task."""
    id: str
    domain: TaskDomain
    description: str
    key_points: List[str]  # Critical subtasks for success
    required_tools: List[str]  # Tools needed (for analysis)
    temporal_constraints: Dict[str, str]  # Time-dependent requirements
    success_criteria: str

    def to_agent_prompt(self):
        """Convert task to prompt for agent."""
        prompt = f"""Task: {self.description}

Success Criteria:
{self.success_criteria}

Key Milestones to Achieve:
"""
        for kp in self.key_points:
            prompt += 

         .temporal_constraints:
            prompt += 
             constraint, details  .temporal_constraints.items():
                prompt += 

         prompt


tasks = [
    Task(
        =,
        domain=TaskDomain.OFFICE,
        description=,
        key_points=[
            ,
            ,
            ,
            
        ],
        required_tools=[, , ],
        temporal_constraints={: , : },
        success_criteria=
    ),
    Task(
        =,
        domain=TaskDomain.SHOPPING,
        description=,
        key_points=[
            ,
            ,
            ,
            ,
            
        ],
        required_tools=[, , ],
        temporal_constraints={: },
        success_criteria=
    ),
]

 ():
    
    all_tasks = []
     domain  TaskDomain:
        domain_tasks = [t  t  tasks  t.domain == domain]
        
        all_tasks.extend(domain_tasks[:num_tasks_per_domain])
     all_tasks

Step 2: Implement the Tool Retrieval System

Build effective tool discovery from large tool spaces:

from typing import Tuple
import numpy as np
from sentence_transformers import SentenceTransformer

class ToolRetriever:
    """
    Retrieve relevant tools from massive tool catalog using semantic search.
    Addresses the critical bottleneck: retrieval accounts for ~50% of agent failures.
    """
    def __init__(self, tool_catalog, embedding_model='all-MiniLM-L6-v2'):
        self.tool_catalog = tool_catalog
        self.embedder = SentenceTransformer(embedding_model)

        # Pre-compute embeddings for all tools
        self.tool_descriptions = [
            f"{tool['name']}: {tool['description']}"
            for tool in tool_catalog
        ]
        self.tool_embeddings = self.embedder.encode(self.tool_descriptions)

    def retrieve_tools(self, query: str, k: int = 5) -> List[Dict]:
        """
        Retrieve top-k tools using semantic similarity on combined server-tool description.
        Joint alignment is more effective than individual descriptions.
        """
        query_embedding = self.embedder.encode(query)

        # Compute similarities
        similarities = np.dot(self.tool_embeddings, query_embedding)
        top_indices = np.argsort(similarities)[::-][:k]

        retrieved_tools = [.tool_catalog[i]  i  top_indices]
         retrieved_tools

     () -> [[, ]]:
        
        task_embedding = .embedder.encode(task_query)

        scored_tools = []
         tool  .tool_catalog:
            
            joint_desc = 
            joint_embedding = .embedder.encode(joint_desc)

            
            alignment_score = np.dot(joint_embedding, task_embedding)
            scored_tools.append((tool, alignment_score))

        
        scored_tools.sort(key= x: x[], reverse=)
         scored_tools[:k]


retriever = ToolRetriever(tool_catalog)
retrieved = retriever.retrieve_with_joint_alignment(
    ,
    k=
)

Step 3: Build the ReACT Agent for Tool Composition

Implement the agent loop with Route, Execute, Response operations:

from enum import Enum
from typing import Callable, Any

class AgentOperation(Enum):
    ROUTE = "route"
    EXECUTE = "execute"
    RESPONSE = "response"
    REVISE = "revise"

class MCPCopilotAgent:
    """
    ReACT-based agent formulated as a POMDP.
    - State: current task progress, available tools, previous actions
    - Actions: Route (select tools), Execute (call tools), Respond (generate answer), Revise (recover from errors)
    """
    def __init__(self, model_name: str, retriever: ToolRetriever):
        self.model_name = model_name
        self.retriever = retriever
        self.action_history = []
        self.tool_execution_results = {}

    def route(self, state: Dict, task: Task, k: int = 5) -> List[Dict]:
        """
        Route operation: Select k candidate tools for current task state.
        Uses retriever with joint alignment.
        """
        # Construct query from task and current state
        query = task.description
        if state.get('recent_failures'):
            query += f" (Note: Previously tried and failed: {state['recent_failures']})"

        # Retrieve candidate tools
        candidates = self.retriever.retrieve_with_joint_alignment(query, k=k)
        selected_tools = [tool  tool, score  candidates]

         selected_tools

     () -> :
        
        tool_name = tool[]
        server_name = tool.get()

        
        result = {
            : tool_name,
            : server_name,
            : ,
            : input_params
        }

        :
            
            output = .call_mcp_tool(server_name, tool_name, input_params)
            result[] = 
            result[] = output
         Exception  e:
            result[] = 
            result[] = (e)

        
        .action_history.append(result)
        .tool_execution_results[] = result

         result

     () -> :
        
        
        evidence = 
         action  .action_history:
             action[] == :
                evidence += 

        
        prompt = 

        response = .model.generate(prompt)
         response

     () -> [, ]:
        
        state = {
            : task,
            : ,
            : [],
            : []
        }

         step  (max_steps):
            state[] = step

            
            candidate_tools = .route(state, task, k=)

              candidate_tools:
                

            
            
             tool  candidate_tools[:]:  
                params = .generate_tool_params(tool, state, task)
                result = .execute(tool, params)

                 result[] == :
                    state[].append(tool[])
                :
                    
                     kp  task.key_points:
                         kp   state[]:
                            state[].append(kp)

            
             (state[]) >= (task.key_points):
                

        
        final_response = .response(state, task)

        stats = {
            : state[] + ,
            : (.action_history),
            : (  a  .action_history  a[] == ),
            : (  a  .action_history  a[] == ),
            : (state[]),
            : (task.key_points),
        }

         final_response, stats

     () -> :
        
        
        

     () -> :
        
        
        prompt = 
        params = .model.generate_json(prompt)
         params

Step 4: Implement LLM-as-Judge Evaluation

Evaluate agent responses using key points instead of fixed ground truth:

from dataclasses import dataclass

@dataclass
class EvaluationResult:
    task_id: str
    success: bool
    key_points_achieved: int
    key_points_total: int
    response_quality: float  # 0-1
    confidence: float  # Evaluator confidence
    reasoning: str

class LiveMCPEval:
    """
    LLM-as-Judge evaluation framework.
    Uses key points (critical subtasks) for human-aligned evaluation.
    Handles dynamic data and multiple valid solutions.
    """
    def __init__(self, judge_model_name: str):
        self.judge_model = judge_model_name

    def evaluate_response(self, response: str, task: Task, execution_stats: Dict) -> EvaluationResult:
        """
        Evaluate agent response against key points.
        Key points are subtasks that must be satisfied for success.
        """
        prompt = f"""Evaluate this agent response for the following task:

Task: {task.description}

Agent Response:
{response}

Execution Statistics:
- Steps taken: {execution_stats.get('steps_taken')}
- Tools used: {execution_stats.get('tools_used')}
- Successful tool calls: {execution_stats.get('successful_calls')}

Key Points that MUST be addressed:
"""
        for i, kp in enumerate(task.key_points, 1):
            prompt += 

        prompt += 

        
        judgment = .judge_model.generate(prompt)

        
        result = ._parse_judgment(judgment, task)
         result

     () -> EvaluationResult:
        
        
        lines = judgment.split()

        keypoints_achieved = 
         line  lines:
               line.lower()  ((i)  line  i  ((task.key_points))):
                keypoints_achieved += 

        quality =   
        confidence =   

        success = keypoints_achieved >= (task.key_points) *   

         EvaluationResult(
            task_id=task.,
            success=success,
            key_points_achieved=keypoints_achieved,
            key_points_total=(task.key_points),
            response_quality=quality / ,
            confidence=confidence / ,
            reasoning=judgment
        )

     () -> [EvaluationResult]:
        
        results = []
         response, task, stats  (responses, tasks, execution_stats):
            result = .evaluate_response(response, task, stats)
            results.append(result)
         results

Step 5: Analyze Bottlenecks

Identify failure sources to improve agent design:

def analyze_failure_modes(results: List[EvaluationResult], execution_stats: List[Dict]) -> Dict:
    """
    Analyze agent performance to identify bottlenecks.
    LiveMCPBench shows retrieval accounts for ~50% of failures.
    """
    failures = [r for r in results if not r.success]

    analysis = {
        'total_tasks': len(results),
        'successful_tasks': sum(1 for r in results if r.success),
        'success_rate': sum(1 for r in results if r.success) / len(results),
        'failures': {
            'retrieval_failures': 0,
            'execution_failures': 0,
            'composition_failures': 0,
            'reasoning_failures': 0
        }
    }

    for failure, stats in zip(failures, [execution_stats[results.index(r)] for r in failures]):
        # Retrieval failure: too few successful tool calls
        if stats['successful_calls'] == 0:
            analysis['failures']['retrieval_failures'] += 
        
         stats[] >   failure.key_points_achieved < (failure.key_points_total) * :
            analysis[][] += 
        
         stats[] >   failure.key_points_achieved < (failure.key_points_total) * :
            analysis[][] += 
        
        :
            analysis[][] += 

     analysis

Practical Guidance

When to Use:

• Evaluating agents that must discover tools from large ecosystems (>500 tools)
• Scenarios emphasizing multi-tool composition and reasoning
• Applications requiring reproducible benchmarking without external API dependencies
• Cases where evaluation must support multiple valid solution paths

When NOT to Use:

• Simple, single-tool tasks with clear ground truth answers
• Real-time evaluation (LLM-as-Judge adds latency)
• Domains with strict, deterministic output requirements (medical)
• Low-resource settings where maintaining 70 servers is infeasible

Hyperparameters:

Parameter	Default	Impact
retrieval_k	5	Number of candidate tools retrieved per decision; higher = better coverage, higher latency
max_agent_steps	10	Episode length limit; balance exploration vs. computational cost
keypoint_success_threshold	0.8	Fraction of key points needed for task success; lower = more lenient
evaluator_confidence_threshold	0.7	Minimum confidence required for evaluation; higher = stricter

Common Challenges:

• Tool explosion: 527 tools causes retrieval challenges; improve via better descriptions and taxonomy
• Task ambiguity: Multiple solution paths; key points framework handles this better than ground truth
• Real-time data: Temporal constraints require live data; use mock data for reproducibility

Reference

Paper: LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? (2508.01780)

• 95 realistic daily tasks across 6 domains with temporal dynamics
• 70 reproducible MCP servers with 527 tools (no external API dependencies)
• Identifies retrieval as dominant bottleneck (~50% of failures)
• LLM-as-Judge evaluation with 81% human agreement
• Performance ceiling at 78.95% (Claude-Sonnet-4), most models at 30-50%