evalscope
LLM evaluation & inference performance testing via the evalscope CLI. Translates natural language requests into evalscope commands for: (1) Model accuracy evaluation — runs registered benchmarks against local checkpoints or API endpoints (OpenAI-compatible, Anthropic, LiteLLM); (2) Performance stress testing — TTFT, TPOT, throughput, latency under configurable concurrency; (3) RAG evaluation — RAGAS quality metrics, MTEB embedding benchmarks, CLIP retrieval; (4) Benchmark discovery — list/filter/inspect benchmarks by tag. Trigger on: evaluate / benchmark / score a model, throughput / latency / QPS / stress test, find benchmarks, view results, 评测模型, 压测, 跑 benchmark, 性能测试, 查看评测结果, 有哪些评测集, RAG 评测, embedding 评测. Do NOT trigger for: model training / finetuning / deployment / serving requests.
소스 정보
- 저장소
- modelscope/evalscope
- 최근 소스 활동
- 2026년 9월 9일 16:59
- 감지된 SKILL.md 언어
- 영어
- 스타
- 3,475
- 포크
- 499
설치 방법
기본적으로 소스를 먼저 확인하는 Prompt가 선택됩니다. 직접 명령으로 전환하거나 로컬 사본을 다운로드할 수도 있습니다.
소스 파일 검토
설치 여부를 결정하기 전에 SKILL.md와 SkillsMP에 표시된 보조 파일을 읽어 보세요.
파일 탐색기
6 개 파일SKILL.md 표시 중
SKILL.md
소스 지침 · 읽기 전용 미리보기- name
- evalscope
- description
- LLM evaluation & inference performance testing via the evalscope CLI. Translates natural language requests into evalscope commands for: (1) Model accuracy evaluation — runs registered benchmarks against local checkpoints or API endpoints (OpenAI-compatible, Anthropic, LiteLLM); (2) Performance stress testing — TTFT, TPOT, throughput, latency under configurable concurrency; (3) RAG evaluation — RAGAS quality metrics, MTEB embedding benchmarks, CLIP retrieval; (4) Benchmark discovery — list/filter/inspect benchmarks by tag. Trigger on: evaluate / benchmark / score a model, throughput / latency / QPS / stress test, find benchmarks, view results, 评测模型, 压测, 跑 benchmark, 性能测试, 查看评测结果, 有哪些评测集, RAG 评测, embedding 评测. Do NOT trigger for: model training / finetuning / deployment / serving requests.
# EvalScope
Read only the relevant reference file for the matched workflow — don't preload all of them.
| Workflow | When | Reference |
|----------|------|-----------|
| Eval (accuracy) | evaluate / benchmark / score | [eval-reference.md](eval-reference.md) |
| Perf (stress test) | throughput / latency / QPS / perf | [perf-reference.md](perf-reference.md) |
| RAG Evaluation | RAG / embedding / retrieval quality | [rag-reference.md](rag-reference.md) |
| Visualization | view results / compare / dashboard | (below) |
| Benchmark Discovery | list / find / what benchmarks | (below) |
| Troubleshooting | errors / failures / debug | [troubleshooting.md](troubleshooting.md) |
## Install EvalScope and the Project Skill
Install the EvalScope CLI in the environment that will run evaluations:
```bash
evalscope --version # verify installation
pip install evalscope # basic
pip install 'evalscope[perf]' # perf only
pip install 'evalscope[rag]' # RAG only (RAGAS, MTEB, CLIP)
pip install 'evalscope[service]' # Web dashboard
```
For a coding agent, install or copy this repository's
[`skills/evalscope`](https://github.com/modelscope/evalscope/tree/main/skills/evalscope)
directory into the agent's configured skill-discovery path. Use that agent's
official skill installer or setup guide rather than assuming a universal local
path. Before handing off an evaluation task, confirm that the agent can read
this `SKILL.md` and the relevant reference file.
## Decision Tree
- User wants **accuracy evaluation** (evaluate / benchmark / score / 评测)
- Local checkpoint path or HuggingFace/ModelScope ID → `--model PATH` (auto `llm_ckpt`)
- API endpoint → `--model NAME --api-url URL` (auto `openai_api`)
- Anthropic → `--eval-type anthropic_api`
- LiteLLM multi-provider → `--eval-type litellm --model provider/name`
- OpenAI Responses API → `--eval-type openai_responses_api`
- Pipeline test → `--model mock --eval-type mock_llm`
- Image generation → `--eval-type text2image`
- TTS → `--eval-type text2speech`
- Image editing → `--eval-type image_editing`
- User wants **performance test** (throughput / latency / QPS / 压测)
- → `evalscope perf` workflow
- API types: `openai` (default), `openai_responses` / `responses`,
`openai_embedding` / `embedding`, `openai_rerank` / `rerank`, `local`,
or `local_vllm`
- User wants **RAG evaluation** (RAG / embedding quality / retrieval)
- → `evalscope eval --eval-backend RAGEval` with tool config
- User wants **visualization** (view / compare / dashboard)
- → `evalscope service`
- User wants **benchmark info** (list / find / what benchmarks / 有哪些评测集)
- → `evalscope benchmark-info`
## Workflow 1: Eval (Accuracy)
Core command pattern:
```bash
# Local checkpoint
evalscope eval --model Qwen/Qwen2.5-0.5B-Instruct --datasets gsm8k --limit 10
# API endpoint (auto-detects openai_api when --api-url is set)
evalscope eval --model qwen-plus --datasets gsm8k arc \
--api-url http://localhost:8000/v1/chat/completions --api-key sk-xxx --limit 10
# Anthropic
evalscope eval --model claude-3-5-sonnet --eval-type anthropic_api --datasets mmlu --api-key sk-ant-xxx
```
Key parameters: `--datasets`, `--limit`, `--generation-config`, `--dataset-args`, `--eval-backend`, `--judge`. For full parameter list → [eval-reference.md](eval-reference.md).
Output: `outputs/<timestamp>/reports/*.json` (scores), `report.html` (summary).
## Workflow 2: Perf (Stress Test)
Core command pattern:
```bash
# Basic throughput test
evalscope perf --model qwen-plus \
--url http://localhost:8000/v1/chat/completions --api openai \
--dataset openqa --parallel 5 --number 200 --stream
# Concurrency gradient (--parallel and --number must pair)
evalscope perf --model qwen-plus --url http://localhost:8000/v1/chat/completions \
--api openai --parallel 1 5 10 20 --number 50 250 500 1000 --stream
# Embedding model
evalscope perf --model text-embedding-v3 --url http://localhost:8000/v1/embeddings \
--api embedding --parallel 10 --number 500
# Rerank model
evalscope perf --model bge-reranker --url http://localhost:8000/v1/rerank \
--api rerank --parallel 5 --number 200
```
Key parameters: `--parallel`, `--number`, `--dataset`, `--max-tokens`, `--sla-auto-tune`. For full parameter list → [perf-reference.md](perf-reference.md).
Output: console table (TTFT/TPOT/throughput p50-p99) + HTML report.
## Workflow 3: RAG Evaluation
Uses `--eval-backend RAGEval` with a Python dict/YAML config. Three tools: RAGAS, MTEB, clip_benchmark.
```python
from evalscope import run_task
run_task({
'eval_backend': 'RAGEval',
'eval_config': {
'tool': 'MTEB', # or 'RAGAS' or 'clip_benchmark'
...
}
})
```
For config schemas and examples → [rag-reference.md](rag-reference.md).
## Workflow 4: Visualization
```bash
evalscope service --host 0.0.0.0 --port 9000 --outputs ./outputs
```
Options: `--host` (default 0.0.0.0), `--port` (default 9000), `--outputs PATH` (scan dir), `--debug`.
Requires: `pip install 'evalscope[service]'`.
## Workflow 5: Benchmark Discovery
```bash
evalscope benchmark-info --list # all benchmarks
evalscope benchmark-info --list --tag Math Coding # filter by tags (OR, case-insensitive)
evalscope benchmark-info gsm8k # text summary
evalscope benchmark-info gsm8k --format json # structured JSON
evalscope benchmark-info gsm8k --format markdown # full docs
```
## Workflow 6: Sandbox Evaluation
For code-execution benchmarks (HumanEval, MBPP, etc.) with Docker isolation:
```bash
evalscope eval --model qwen-plus --datasets humaneval \
--api-url http://localhost:8000/v1/chat/completions \
--sandbox '{"enabled": true, "engine": "docker"}'
```
Requires Docker daemon running. See `evalscope eval --help` for `--sandbox` schema.
## Quick Lookup Table
For up-to-date results: `evalscope benchmark-info --list --tag <TAG>`
| User Need | Tags | Typical Benchmarks |
|-----------|------|--------------------|
| Math / reasoning | Math, Reasoning | gsm8k, math_500, aime24, competition_math |
| Coding | Coding | humaneval, mbpp, live_code_bench |
| General knowledge | Knowledge, MCQ | mmlu, ceval, cmmlu, mmlu_pro |
| Chinese | Chinese | ceval, cmmlu, chinese_simpleqa |
| Multimodal / vision | MultiModal | mmmu, mm_bench, math_vista |
| Instruction following | InstructionFollowing | ifeval, multi_if |
| Function calling | FunctionCalling | bfcl_v3, bfcl_v4 |
| Long context | LongContext | needle_haystack, longbench_v2 |
| Agent | Agent | tau_bench |
Common suites:
- **General LLM**: `mmlu gsm8k bbh humaneval ifeval`
- **Chinese**: `ceval cmmlu chinese_simpleqa`
- **Multimodal**: `mmmu mm_bench math_vista mm_star`
## General Notes
- Always `--limit 5` for first-run validation
- Default output: `./outputs/<timestamp>/`
- Long runs: background + `tail -f outputs/<timestamp>/logs/eval_log.log`
- Resume interrupted runs: `--use-cache outputs/<previous_timestamp>`
- Full parameter help: `evalscope eval --help` / `evalscope perf --help`
- On errors → [troubleshooting.md](troubleshooting.md)
GitHub에서 보기