mcp-benchmark

description

Best practices for running mcpbr evaluations including cost optimization, sample sizing, MCP server configuration, and result interpretation.

faq

[{"q":"What are the best practices for running mcpbr evaluations?","a":"Start with small sample sizes (n=5-10) to validate your setup, use templates for common scenarios, monitor costs by tracking token usage, and save results to compare different configurations. Always test your MCP server standalone before benchmarking."},{"q":"How can I optimize mcpbr evaluation costs?","a":"Use faster/cheaper models like Haiku for development, start with small samples, reduce max_iterations for initial testing, use --mcp-only to skip baseline runs during development, and enable verbose logging to catch issues early."},{"q":"What's the recommended workflow for testing a new MCP server?","a":"1) Test server standalone, 2) Run quick-test template with n=1, 3) Scale to n=5-10, 4) Analyze tool usage in results, 5) Optimize based on findings, 6) Run full evaluation with n=25-50+."},{"q":"How do I avoid common mcpbr pitfalls?","a":"Test Docker setup first, verify API keys are set, use pre-built images when available, set appropriate timeouts for your tasks, start with small samples, and always save results to enable comparisons and regression detection."},{"q":"How should I secure API keys when running mcpbr?","a":"Never hardcode API keys in config files. Use environment variables, .env files (gitignored), or secret managers like 1Password CLI, AWS Secrets Manager, or HashiCorp Vault. In CI/CD, store keys as repository secrets."},{"q":"How do I optimize mcpbr performance for large benchmarks?","a":"Tune max_concurrent based on available RAM (roughly 1-3 GB per container), use pre-built Docker images, enable result caching with cache_enabled, and use setup_command for MCP server pre-computation. Monitor with docker stats."},{"q":"How do I set up mcpbr in a CI/CD pipeline?","a":"Install mcpbr in your CI runner, set ANTHROPIC_API_KEY from secrets, use small sample sizes (n=10) for PRs, enable regression detection with --baseline-results and --regression-threshold, and output JUnit XML for test reporting."},{"q":"How do I set a budget limit for mcpbr evaluations?","a":"Set budget: 50.00 in your config YAML to cap spending at $50. Combine with sample_size, max_iterations, and timeout_seconds for layered cost control. Monitor actual costs via token usage in JSON output."},{"q":"How do I make statistically valid comparisons between MCP servers?","a":"Use the same task set (--task flags), same model, same benchmark, and same parameters. Sample sizes below 25 have high variance. Check if confidence intervals overlap before concluding one config is better."}]

Best Practices Guide

This guide helps you get the most value from mcpbr while avoiding common pitfalls. Whether you're testing a new MCP server, optimizing costs, or setting up CI/CD pipelines, these practices will help you work effectively.

Quick Reference

Scenario	Recommended Approach
First-time setup	Use quick-test template, verify with n=1
MCP server testing	Standalone test → quick-test → scale gradually
Cost optimization	Use Haiku model, small samples, --mcp-only flag
Production evaluation	Use production template, save all outputs
CI/CD integration	Use regression detection, JUnit XML, notifications
Security testing	Start with cybergym-basic, progress to advanced
Debugging failures	Enable -vv, use --log-dir, analyze tool usage

Benchmark Selection Guidelines

Choosing the Right Benchmark

Use SWE-bench when:

• Testing code exploration and bug-fixing capabilities
• Evaluating Python-focused MCP servers
• Need proven, standardized benchmarks
• Want fast evaluation with pre-built images

Use CyberGym when:

• Testing security analysis capabilities
• Evaluating C/C++ code understanding
• Need vulnerability detection benchmarks
• Want to test different difficulty levels

SWE-bench Best Practices

Start Small, Scale Gradually

# Step 1: Single task smoke test
mcpbr run -c config.yaml -n 1 -v

# Step 2: Small sample
mcpbr run -c config.yaml -n 5 -o results-5.json

# Step 3: Medium sample
mcpbr run -c config.yaml -n 25 -o results-25.json

# Step 4: Full evaluation (if needed)
mcpbr run -c config.yaml -o results-full.json

Use Pre-built Images

use_prebuilt_images: true  # Much faster and more reliable

Pre-built images provide:

• Validated dependency installation
• Consistent evaluation environment
• Faster startup (no package installation)
• Working Python imports inside containers

Anti-pattern: Disabling pre-built images without good reason