Skip to main content Accueil Créateurs ascend-ai-coding awesome-ascend-skills ais-bench
ais-bench AISBench Benchmark - AI model evaluation tool for Ascend NPU. Supports accuracy evaluation (service/local models on text, multimodal datasets), performance evaluation (latency, throughput, stress testing, steady-state, real traffic simulation), vLLM/Triton inference services, 15+ benchmarks (MMLU, GSM8K, MMMU, docvqa, ocrbench_v2, etc.), multi-turn dialogue, Function Call (BFCL), and custom datasets.
Aller à l'installation Skills Marketplace Découvrez et explorez les compétences IA créées par la communauté.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Afficher les détails du prompt Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
npx skills add https://github.com/ascend-ai-coding/awesome-ascend-skills --skill ais-benchLa commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.
Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.
Télécharger Zip Téléchargement... Plus depuis ce dépôt 评估 CANN 算子仓的「进阶教程 / 开发指南」类文档质量——通读文档 + 对照算子代码**静态**查证(默认不跑),按五轴(找得到/信得过/学得会/可操作/读得懂)找出 漏讲/讲不清/过时/对不上代码/概念讲错,产出带证据与改进建议的体检报告(MD + HTML)。涉及「评进阶教程 / 开发指南文档质量 / 文档对不对得上代码 / 教程审稿 / tutorial 体检 / 文档信不信得过」等意图时使用。只评不改不跑,只对着文档与代码出诊断。
Run NPU inference/training tests on a remote SSH server with vllm-ascend Docker container. Use when the user asks to test models on NPU, run inference on Ascend devices, or deploy models to an SSH server.
vllm-daily-pr-issue-tracker Track daily PRs and Issues from vllm-project/vllm and vllm-project/vllm-ascend, filter by model (DeepSeek/Qwen/GLM/MiniMax/Kimi) and tech topics (PD disaggregation, MTP, quantization, graph mode, performance), analyze with LLM, and generate a Markdown report. Use when user wants vllm daily tracker, PR/Issue digest, or Ascend inference ecosystem monitoring.
Métiers associés SOC
Basé sur la classification professionnelle SOC
Explorateur de fichiers
12 fichiers name ais-bench description AISBench Benchmark - AI model evaluation tool for Ascend NPU. Supports accuracy evaluation (service/local models on text, multimodal datasets), performance evaluation (latency, throughput, stress testing, steady-state, real traffic simulation), vLLM/Triton inference services, 15+ benchmarks (MMLU, GSM8K, MMMU, docvqa, ocrbench_v2, etc.), multi-turn dialogue, Function Call (BFCL), and custom datasets. keywords ["ais-bench","aisbench","model evaluation","benchmark","accuracy evaluation","performance evaluation","vllm","multimodal","llm evaluation"] github_url https://github.com/AISBench/benchmark github_hash 03f0f43383c3efdb6ecdcc845eb24bc7a1537575 version 1.0.0 created_at 2026-02-25T00:00:00.000Z entry_point ais_bench
AISBench Benchmark Tool
AISBench Benchmark is a model evaluation tool built based on OpenCompass. It supports evaluation scenarios for both accuracy and performance testing of AI models on Ascend NPU.
Overview
Accuracy Evaluation : Accuracy verification of service-deployed models and local models on various QA and reasoning benchmark datasets, covering text, multimodal, and other scenarios.Performance Evaluation : Latency and throughput evaluation of service-deployed models, extreme performance testing under stress test scenarios, steady-state performance evaluation, and real business traffic simulation.
Supported Scenarios
Scenario Description Accuracy Evaluation Model accuracy on text/multimodal datasets Performance Evaluation Latency, throughput, stress testing Steady-State Performance Obtain true optimal system performance Real Traffic Simulation Simulate real business traffic patterns Multi-turn Dialogue Evaluate multi-turn conversation models Function Call (BFCL) Function calling capability evaluation
Supported Benchmarks
Text : GSM8K, MMLU, Ceval, FewCLUE series, dapo_math, levalMultimodal : docvqa, infovqa, ocrbench_v2, omnidocbench, mmmu, mmmu_pro, mmstar, videomme, textvqa, videobench, vocalsoundMulti-turn Dialogue : sharegpt, mtbenchFunction Call : BFCL (Berkeley Function Calling Leaderboard)
Installation
Environment Requirements
Python Version : Only Python 3.10 , 3.11 , or 3.12 is supported.
conda create --name ais_bench python=3.10 -y
conda activate ais_bench
Install from Source
git clone https://github.com/AISBench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517
Optional Dependencies
pip3 install -r requirements/api.txt
pip3 install -r requirements/extra.txt
pip3 install -r requirements/hf_vl_dependency.txt
pip3 install -r requirements/datasets/bfcl_dependencies.txt --no-deps
Quick Start
Basic Command Structure ais_bench --models <model_task> --datasets <dataset_task> [--summarizer example]
--models: Specifies the model task configuration--datasets: Specifies the dataset task configuration--summarizer: Result presentation task (default: example)
Find Configuration Files
ais_bench --models vllm_api_general_chat --datasets demo_gsm8k_gen_4_shot_cot_chat_prompt --search
Example: Service Model Accuracy Evaluation
Start vLLM inference service (follow vLLM documentation)
Prepare dataset :
Download GSM8K from opencompass
Extract to ais_bench/datasets/gsm8k/
Modify model configuration (vllm_api_general_chat.py):
from ais_bench.benchmark.models import VLLMCustomAPIChat
models = [
dict (
attr="service" ,
type =VLLMCustomAPIChat,
abbr='vllm-api-general-chat' ,
path="" ,
model="" ,
stream=False ,
request_rate=0 ,
retry=2 ,
api_key="" ,
host_ip="localhost" ,
host_port=8080 ,
url="" ,
max_out_len=512 ,
batch_size=1 ,
trust_remote_code=False ,
generation_kwargs=dict (
temperature=0.01 ,
ignore_eos=False ,
)
)
]
Run evaluation :
ais_bench --models vllm_api_general_chat --datasets demo_gsm8k_gen_4_shot_cot_chat_prompt
Output Results dataset version metric mode vllm_api_general_chat
----------------------- -------- -------- ----- ----------------------
demo_gsm8k 401e4c accuracy gen 62.50
Model Task Types
Service-Deployed Models Model Type Description vllm_api_general_chatGeneral vLLM API chat model vllm_api_function_call_chatFunction calling model (BFCL) triton_api_*Triton inference service
Local Models Model Type Description hf_*HuggingFace models vllm_offline_*vLLM offline inference
Performance Evaluation
Key Metrics Metric Description TTFT Time to First Token TPOT Time Per Output Token Throughput Tokens per second Latency Request latency (P50, P90, P99)
Performance Test Example ais_bench --models vllm_api_general_chat --datasets custom_performance \
--mode performance --concurrency 100
Steady-State Performance For obtaining true optimal system performance:
ais_bench --models vllm_api_general_chat --datasets sharegpt \
--stable-stage --duration 300
Real Traffic Simulation ais_bench --models vllm_api_general_chat --datasets custom \
--rps-distribution rps_config.json
Multi-task Evaluation
Multiple Models ais_bench --models model1 model2 model3 --datasets dataset1
Multiple Datasets ais_bench --models model1 --datasets dataset1 dataset2 dataset3
Parallel Execution ais_bench --models model1 model2 --datasets dataset1 dataset2 --parallel 4
Custom Datasets
Performance Custom Dataset Create a JSONL file with custom requests:
{ "input" : "Your prompt here" , "max_output_length" : 512 }
Accuracy Custom Dataset
Output Structure outputs/default/20250628_151326/
├── configs/ # Combined configuration
├── logs/ # Execution logs
│ ├── eval/ # Evaluation logs
│ └── infer/ # Inference logs
├── predictions/ # Raw inference results
├── results/ # Calculated scores
└── summary/ # Final summaries
├── summary_*.csv
├── summary_*.md
└── summary_*.txt
Task Management Interface During execution, a real-time task management interface displays:
Task name and progress
Time cost and status
Log path
Extended parameters
P key: Pause/Resume screen refreshCtrl+C: Exit
Common CLI Options Option Description --modelsModel task name(s) --datasetsDataset task name(s) --summarizerResult summarizer --searchList config file paths --debugPrint detailed logs --modeEvaluation mode (accuracy/performance) --parallelNumber of parallel tasks --resumeResume from breakpoint --failed-onlyRe-run failed cases only
Advanced Features
Breakpoint Resume ais_bench --models model1 --datasets dataset1 --resume outputs/default/20250628_151326
Failed Case Re-run ais_bench --models model1 --datasets dataset1 --failed-only --resume outputs/default/20250628_151326
Multi-file Dataset Merge For datasets like MMLU with multiple files:
ais_bench --models model1 --datasets mmlu_merged
Repeated Inference for pass@k ais_bench --models model1 --datasets dataset1 --repeat-n 5
Troubleshooting
Installation Issues
Python version mismatch : Use Python 3.10/3.11/3.12Dependency conflicts : Use conda environmentbfcl_eval pathlib issue : Use --no-deps flag
Runtime Issues
Model connection failed : Check host_ip, host_port, and service statusDataset not found : Download dataset to ais_bench/datasets/Memory issues : Reduce batch_size or use smaller dataset
Helper Scripts Quick utility scripts for common operations:
bash scripts/check_env.sh
bash scripts/run_accuracy_test.sh vllm_api_general_chat demo_gsm8k --host-port 8080
bash scripts/run_performance_test.sh vllm_api_general_chat sharegpt --concurrency 100
python scripts/parse_results.py outputs/default/20250628_151326
References Detailed documentation for specific use cases:
Templates Ready-to-use templates for custom evaluation:
