evalscope

# EvalScope Read only the relevant reference file for the matched workflow — don't preload all of them. | Workflow | When | Reference | |----------|------|-----------| | Eval (accuracy) | evaluate / benchmark / score | [eval-reference.md](eval-reference.md) | | Perf (stress test) | throughput / latency / QPS / perf | [perf-reference.md](perf-reference.md) | | RAG Evaluation | RAG / embedding / retrieval quality | [rag-reference.md](rag-reference.md) | | Visualization | view results / compare / dashboard | (below) | | Benchmark Discovery | list / find / what benchmarks | (below) | | Troubleshooting | errors / failures / debug | [troubleshooting.md](troubleshooting.md) | ## Install EvalScope and the Project Skill Install the EvalScope CLI in the environment that will run evaluations: ```bash evalscope --version # verify installation pip install evalscope # basic pip install 'evalscope[perf]' # perf only pip install 'evalscope[rag]' # RAG only (RAGAS, MTEB, CLIP) pip install 'evalscope[service]' # Web dashboard ``` For a coding agent, install or copy this repository's [`skills/evalscope`](https://github.com/modelscope/evalscope/tree/main/skills/evalscope) directory into the agent's configured skill-discovery path. Use that agent's official skill installer or setup guide rather than assuming a universal local path. Before handing off an evaluation task, confirm that the agent can read this `SKILL.md` and the relevant reference file. ## Decision Tree - User wants **accuracy evaluation** (evaluate / benchmark / score / 评测) - Local checkpoint path or HuggingFace/ModelScope ID → `--model PATH` (auto `llm_ckpt`) - API endpoint → `--model NAME --api-url URL` (auto `openai_api`) - Anthropic → `--eval-type anthropic_api` - LiteLLM multi-provider → `--eval-type litellm --model provider/name` - OpenAI Responses API → `--eval-type openai_responses_api` - Pipeline test → `--model mock --eval-type mock_llm` - Image generation → `--eval-type text2image` - TTS → `--eval-type text2speech` - Image editing → `--eval-type image_editing` - User wants **performance test** (throughput / latency / QPS / 压测) - → `evalscope perf` workflow - API types: `openai` (default), `openai_responses` / `responses`, `openai_embedding` / `embedding`, `openai_rerank` / `rerank`, `local`, or `local_vllm` - User wants **RAG evaluation** (RAG / embedding quality / retrieval) - → `evalscope eval --eval-backend RAGEval` with tool config - User wants **visualization** (view / compare / dashboard) - → `evalscope service` - User wants **benchmark info** (list / find / what benchmarks / 有哪些评测集) - → `evalscope benchmark-info` ## Workflow 1: Eval (Accuracy) Core command pattern: ```bash # Local checkpoint evalscope eval --model Qwen/Qwen2.5-0.5B-Instruct --datasets gsm8k --limit 10 # API endpoint (auto-detects openai_api when --api-url is set) evalscope eval --model qwen-plus --datasets gsm8k arc \ --api-url http://localhost:8000/v1/chat/completions --api-key sk-xxx --limit 10 # Anthropic evalscope eval --model claude-3-5-sonnet --eval-type anthropic_api --datasets mmlu --api-key sk-ant-xxx ``` Key parameters: `--datasets`, `--limit`, `--generation-config`, `--dataset-args`, `--eval-backend`, `--judge`. For full parameter list → [eval-reference.md](eval-reference.md). Output: `outputs/<timestamp>/reports/*.json` (scores), `report.html` (summary). ## Workflow 2: Perf (Stress Test) Core command pattern: ```bash # Basic throughput test evalscope perf --model qwen-plus \ --url http://localhost:8000/v1/chat/completions --api openai \ --dataset openqa --parallel 5 --number 200 --stream # Concurrency gradient (--parallel and --number must pair) evalscope perf --model qwen-plus --url http://localhost:8000/v1/chat/completions \ --api openai --parallel 1 5 10 20 --number 50 250 500 1000 --stream # Embedding model evalscope perf --model text-embedding-v3 --url http://localhost:8000/v1/embeddings \ --api embedding --parallel 10 --number 500 # Rerank model evalscope perf --model bge-reranker --url http://localhost:8000/v1/rerank \ --api rerank --parallel 5 --number 200 ``` Key parameters: `--parallel`, `--number`, `--dataset`, `--max-tokens`, `--sla-auto-tune`. For full parameter list → [perf-reference.md](perf-reference.md). Output: console table (TTFT/TPOT/throughput p50-p99) + HTML report. ## Workflow 3: RAG Evaluation Uses `--eval-backend RAGEval` with a Python dict/YAML config. Three tools: RAGAS, MTEB, clip_benchmark. ```python from evalscope import run_task run_task({ 'eval_backend': 'RAGEval', 'eval_config': { 'tool': 'MTEB', # or 'RAGAS' or 'clip_benchmark' ... } }) ``` For config schemas and examples → [rag-reference.md](rag-reference.md). ## Workflow 4: Visualization ```bash evalscope service --host 0.0.0.0 --port 9000 --outputs ./outputs ``` Options: `--host` (default 0.0.0.0), `--port` (default 9000), `--outputs PATH` (scan dir), `--debug`. Requires: `pip install 'evalscope[service]'`. ## Workflow 5: Benchmark Discovery ```bash evalscope benchmark-info --list # all benchmarks evalscope benchmark-info --list --tag Math Coding # filter by tags (OR, case-insensitive) evalscope benchmark-info gsm8k # text summary evalscope benchmark-info gsm8k --format json # structured JSON evalscope benchmark-info gsm8k --format markdown # full docs ``` ## Workflow 6: Sandbox Evaluation For code-execution benchmarks (HumanEval, MBPP, etc.) with Docker isolation: ```bash evalscope eval --model qwen-plus --datasets humaneval \ --api-url http://localhost:8000/v1/chat/completions \ --sandbox '{"enabled": true, "engine": "docker"}' ``` Requires Docker daemon running. See `evalscope eval --help` for `--sandbox` schema. ## Quick Lookup Table For up-to-date results: `evalscope benchmark-info --list --tag <TAG>` | User Need | Tags | Typical Benchmarks | |-----------|------|--------------------| | Math / reasoning | Math, Reasoning | gsm8k, math_500, aime24, competition_math | | Coding | Coding | humaneval, mbpp, live_code_bench | | General knowledge | Knowledge, MCQ | mmlu, ceval, cmmlu, mmlu_pro | | Chinese | Chinese | ceval, cmmlu, chinese_simpleqa | | Multimodal / vision | MultiModal | mmmu, mm_bench, math_vista | | Instruction following | InstructionFollowing | ifeval, multi_if | | Function calling | FunctionCalling | bfcl_v3, bfcl_v4 | | Long context | LongContext | needle_haystack, longbench_v2 | | Agent | Agent | tau_bench | Common suites: - **General LLM**: `mmlu gsm8k bbh humaneval ifeval` - **Chinese**: `ceval cmmlu chinese_simpleqa` - **Multimodal**: `mmmu mm_bench math_vista mm_star` ## General Notes - Always `--limit 5` for first-run validation - Default output: `./outputs/<timestamp>/` - Long runs: background + `tail -f outputs/<timestamp>/logs/eval_log.log` - Resume interrupted runs: `--use-cache outputs/<previous_timestamp>` - Full parameter help: `evalscope eval --help` / `evalscope perf --help` - On errors → [troubleshooting.md](troubleshooting.md)