mcp-test

name	mcp-test
description	Universal MCP smoke test. Auto-discovers project MCP configuration, loads and tests all tools, generates structured report. Triggers on: 'test mcp', 'mcp test', '测试 mcp', '/mcp-test'. Optional argument: server name to test only that specific server.

Universal MCP Smoke Test

Automatically discover, analyze, and test MCP servers configured in the current project. Works with any project that has MCP configuration — zero project-specific knowledge required.

Input

• $ARGUMENTS (optional): A specific server name to test. If empty, test all configured servers.

---

Phase 1: Discover Configuration

Search for MCP configuration in the project root, in this order:

1. .mcp.json
2. mcp.json
3. .cursor/mcp.json
4. claude_desktop_config.json

Parse the mcpServers object. For each server, extract:

• Server name (the key)
• Command and args
• Environment variables
• Transport type (stdio by default)

If $ARGUMENTS is provided and non-empty, filter to only the named server.

If no MCP configuration is found anywhere, report the error and stop.

---

Phase 2: Load Project Test Config

Check if mcp-test.yaml exists in the project root. If found, parse it. The schema:

settings:
  unsafe_strategy: ask    # ask | skip | try  (default: ask)
  timeout: 300            # seconds per tool call (default: 300)

servers:
  <server_name>:
    safe: [tool1, tool2]     # explicitly mark as safe to test
    skip: [tool3]            # never test these
    tests:                   # explicit test cases
      - tool: tool_name
        input: { key: value }
        expect: "natural language description of expected result"

2.1 If no mcp-test.yaml exists

When no mcp-test.yaml is found, immediately ask the user before proceeding:

"未找到 mcp-test.yaml 配置文件。你可以： (a) 直接开始自动测试（auto mode） (b) 先提供测试数据（如文件 URL、jobId 等），我会用它们测试并自动生成 mcp-test.yaml"

If the user chooses (b) or provides any custom inputs (URLs, parameters, etc.):

• Record these inputs as user-provided test data
• Immediately match them to the most likely tool parameters based on the data type (e.g., a URL → files parameter of create_extraction; a jobId string → jobId parameter of check_status/get_result/export_document)
• These matched inputs carry forward automatically to Phase 4 and Phase 5 — never ask the user for the same data again
• Persist them into the generated mcp-test.yaml in Phase 7

If the user chooses (a) or just wants to proceed, continue in auto mode.

Key principle — ask once, remember forever: Any custom input the user provides at ANY point in the test session is recorded once and automatically reused wherever applicable. The user should NEVER need to provide the same URL, jobId, or parameter value twice. In Phase 4 batch confirmation, show the user which inputs are already assigned from Phase 2 data.

---

Phase 3: Load and Probe Tools

For each server:

1. Use ToolSearch with keyword search +<server_name> to discover all available tools
2. If nothing found, also try keyword mcp <server_name> as fallback
3. For each discovered tool, record:
   • Full tool name (e.g., mcp__myserver__list_items)
   • Description text
   • Input schema (parameters, required fields, types)
4. If a server yields zero tools, mark it as CONNECTION_FAILED and include diagnostic hints:
   • Is the command path valid?
   • Are dependencies installed?
   • Are required env vars set?

---

Phase 4: Classify Tools

For each loaded tool, determine its safety level.

4.1 Classification Rules

SAFE — read-only, no side effects:

• Name patterns: list_*, get_*, search_*, query_*, check_*, describe_*, read_*, fetch_*, count_*, show_*, find_*, browse_*, lookup_*
• Description signals: "retrieve", "list", "get", "query", "read", "search", "check", "status", "info", "display"
• Explicitly returns data without modifying state

UNSAFE — likely has side effects:

• Name patterns: create_*, delete_*, update_*, send_*, post_*, write_*, remove_*, execute_*, run_*, modify_*, set_*, put_*, push_*, deploy_*, publish_*, drop_*, kill_*, stop_*, start_*
• Description signals: "create", "delete", "modify", "send", "execute", "write", "remove", "deploy", "update", "mutate"

AMBIGUOUS — can't confidently classify:

• Doesn't clearly fit either category
• Name or description is vague (e.g., process, handle, transform, do)

If uncertain, always classify as UNSAFE.

4.2 Apply Config Overrides

If mcp-test.yaml exists and defines overrides for this server:

• Tools in safe list → force to SAFE (overrides LLM judgment)
• Tools in skip list → force to SKIP (absolute, never test)
• All other tools → use LLM classification

4.3 Determine Strategy for Unsafe Tools

Check unsafe_strategy from config (default: ask):

• ask → proceed to batch confirmation (4.4)
• skip → move all UNSAFE and AMBIGUOUS tools to SKIP
• try → promote all UNSAFE to SAFE (advanced users only)

4.4 Batch Confirmation

Present the full classification to the user in one message. If user-provided test data from Phase 2 exists, show the auto-matched inputs inline with the unsafe tools. Format:

## Tool Classification: [server_name]

✅ Safe（自动测试）:
  - list_items — List all items in the store
  - get_status — Check current processing status

⚠️  需要确认（已自动关联用户数据）:
  - create_extraction — 创建提取任务
    → 已关联: files=["https://example.com/test.pdf"] (来自 Phase 2 用户输入)
  - export_document — 导出文档
    → 将使用 create_extraction 返回的 jobId + list_exporters 的导出器名称

❌ Will skip:
  - delete_all — Removes all records

确认执行？
  (a) 按当前计划执行
  (b) 调整工具分类
  (c) 跳过所有 unsafe 工具

Important: Do NOT offer an option to "provide test inputs" if user data from Phase 2 has already been matched to tools. The whole point is to avoid asking twice. Only offer input provision if there are unsafe tools with NO matched data.

Wait for user response. Adjust classifications accordingly, then proceed.

---

Phase 5: Execute Tests

5.1 Run Explicit Test Cases First

If mcp-test.yaml defines tests for a server, run them in declared order.

For chained dependencies (e.g., test 2 needs the jobId from test 1):

• Analyze the full test sequence before executing
• After each test, capture the response
• When a subsequent test's input contains a value that obviously refers to a previous result (same field name, placeholder-like value), substitute the actual value from the previous response
• Use LLM reasoning to identify which output fields map to which input parameters

For expect strings:

• Interpret as natural language assertions
• Evaluate whether the actual response satisfies the described expectation
• Report PASS or FAIL with explanation

5.2 Auto-test Remaining Safe Tools

For tools classified as SAFE that don't already have explicit test cases:

Parameter generation rules:

• Satisfy only required fields; omit all optional fields
• Strings: use "mcp-test-probe", "test", "example" — never real data
• Numbers: use small typical values (1, 10, 100); avoid 0 and negatives unless the schema explicitly allows them
• Booleans: prefer false (less likely to trigger actions)
• Enums: use the first defined value
• Arrays: single-element arrays with a minimal item
• Objects: only required properties with minimal values
• Never generate values that look like real credentials, emails, IPs, file paths, or PII

Call the tool and record:

• Success or error
• Response structure (top-level keys and their types)
• Response time
• Error message if failed

5.3 Test Confirmed Unsafe Tools

For unsafe tools the user approved:

• If user provided specific inputs, use those
• Otherwise, generate conservative inputs using the same rules as 5.2
• Record all results the same way

5.4 Handle Async Operations

If a tool's response looks like an async job (contains fields like jobId, taskId, requestId, id paired with a status like pending, queued, processing):

1. Look for a corresponding status-checking tool on the same server (e.g., check_status, get_status, get_job)
2. If found and it's classified as SAFE (or already tested), poll it:
   • Interval: 5-10 seconds
   • Max total wait: respect timeout (default 300s)
   • Stop when status indicates completion or failure
3. Record the final state and total elapsed time
4. If the async result has a "get result" companion tool, call that too

If no status tool exists or the pattern isn't clear, just record the initial response and move on.

---

Phase 6: Generate Report

Output the report directly in the conversation. Use the user's system language (auto-detect from conversation context).

Report Template

# MCP Test Report

**Project**: {directory name}
**Date**: {current date and time}
**Config**: mcp-test.yaml {found | not found — using auto mode}

## Summary

| Server | Status | Tools | Tested | Passed | Failed | Skipped |
|--------|--------|-------|--------|--------|--------|---------|
| ...    | ...    | ...   | ...    | ...    | ...    | ...     |

---

## Server: {server_name}

### Connection
- Status: ✅ / ❌
- Command: `{command} {args}`
- Tools discovered: {count}

### Tool Schema Quality

Evaluate each tool's definition quality:
- Is the description specific enough for an LLM to understand and use correctly?
- Is the input schema complete (types, required markers, descriptions)?
- Are there any obvious issues?

| Tool | Description Quality | Schema Quality | Notes |
|------|-------------------|----------------|-------|

### Test Results

| Tool | Input (summary) | Status | Response Time | Notes |
|------|----------------|--------|---------------|-------|

For failures, include the actual error message.

### Skipped Tools

| Tool | Reason |
|------|--------|

### Issues Found

Numbered list of concrete, actionable issues. For each:
- What: the specific problem
- Impact: how it affects MCP usability
- Suggestion: how to fix it

### Recommendations

Prioritized list of improvements for the MCP server developer.

Repeat the "Server" section for each server tested.

---

Phase 7: Generate and Maintain mcp-test.yaml

Only if no mcp-test.yaml existed at the start of testing.

7.1 Early Creation (after Phase 4)

Immediately after Phase 4 batch confirmation is complete, generate and write a preliminary mcp-test.yaml to disk. This ensures user-provided data is persisted even if the test session is interrupted.

The preliminary file should contain:

• All servers discovered
• safe / skip classifications from Phase 4
• tests section with user-provided inputs (marked with # user-provided comments)
• expect fields left as "pending — will be filled after test execution" for tests not yet run

Write this file immediately — do NOT wait for user confirmation at this stage. The file is a working draft.

7.2 Progressive Updates (during Phase 5)

As each test completes, update the mcp-test.yaml in place:

• Fill in expect fields with actual results (e.g., "Job created successfully with jobId and pending status")
• Add auto-tested safe tools as additional test cases if they succeeded

7.3 Final Version (after Phase 6)

After the report is generated, present the final mcp-test.yaml content in the conversation:

"已生成 mcp-test.yaml 并保存到项目根目录，包含本次测试的所有数据。下次运行 /mcp-test 将自动复用。"

7.4 Preserve User-Provided Inputs

Critical: User-provided inputs are the most valuable part of the config. They MUST always be preserved with a # user-provided comment so future edits don't accidentally remove them.

Example generated file:

settings:
  unsafe_strategy: ask
  timeout: 300

servers:
  insureforge:
    safe: [list_exporters, check_status, get_result]
    skip: []
    tests:
      # user-provided test URL for document extraction
      - tool: create_extraction
        input:
          files: ["https://example.com/test.pdf"]
        expect: "Job created successfully with jobId and pending status"

      - tool: check_status
        input:
          jobId: "$create_extraction.jobId"  # chained from create_extraction
        expect: "Returns job status with progress info"

      - tool: get_result
        input:
          jobId: "$create_extraction.jobId"  # chained from create_extraction
        expect: "Returns extracted insurance application data with confidence scores"

      - tool: list_exporters
        input: {}
        expect: "Returns list of available exporters including raw_json, product_liability_quotation, application_report"

      - tool: 
        
             
           
         

Note the $tool_name.field syntax for chained dependencies — this tells the test runner to substitute values from previous test results.

---

Rules

1. Always load tools via ToolSearch before calling them. Never assume a tool is available.
2. Never skip the batch confirmation for unsafe tools (unless unsafe_strategy is skip or try in config).
3. Never generate test inputs that resemble real data — no real emails, credentials, IP addresses, file paths containing user info.
4. Always respect skip in mcp-test.yaml — this is absolute, no override, no asking.
5. If classification is uncertain, default to UNSAFE.
6. Don't retry failed tool calls. Record the failure and move on to the next tool.
7. One report, one conversation. Don't split the report across multiple messages. If it's long, that's fine.
8. If all servers fail to connect, still produce a diagnostic report with troubleshooting guidance.