ais-bench

name	ais-bench
description	AISBench Benchmark - AI model evaluation tool for Ascend NPU. Supports accuracy evaluation (service/local models on text, multimodal datasets), performance evaluation (latency, throughput, stress testing, steady-state, real traffic simulation), vLLM/Triton inference services, 15+ benchmarks (MMLU, GSM8K, MMMU, docvqa, ocrbench_v2, etc.), multi-turn dialogue, Function Call (BFCL), and custom datasets.
keywords	["ais-bench","aisbench","model evaluation","benchmark","accuracy evaluation","performance evaluation","vllm","multimodal","llm evaluation"]
github_url	https://github.com/AISBench/benchmark
github_hash	03f0f43383c3efdb6ecdcc845eb24bc7a1537575
version	1.0.0
created_at	2026-02-25T00:00:00.000Z
entry_point	ais_bench

AISBench Benchmark Tool

AISBench Benchmark is a model evaluation tool built based on OpenCompass. It supports evaluation scenarios for both accuracy and performance testing of AI models on Ascend NPU.

Overview

• Accuracy Evaluation: Accuracy verification of service-deployed models and local models on various QA and reasoning benchmark datasets, covering text, multimodal, and other scenarios.
• Performance Evaluation: Latency and throughput evaluation of service-deployed models, extreme performance testing under stress test scenarios, steady-state performance evaluation, and real business traffic simulation.

Supported Scenarios

Scenario	Description
Accuracy Evaluation	Model accuracy on text/multimodal datasets
Performance Evaluation	Latency, throughput, stress testing
Steady-State Performance	Obtain true optimal system performance
Real Traffic Simulation	Simulate real business traffic patterns
Multi-turn Dialogue	Evaluate multi-turn conversation models
Function Call (BFCL)	Function calling capability evaluation

Supported Benchmarks

• Text: GSM8K, MMLU, Ceval, FewCLUE series, dapo_math, leval
• Multimodal: docvqa, infovqa, ocrbench_v2, omnidocbench, mmmu, mmmu_pro, mmstar, videomme, textvqa, videobench, vocalsound
• Multi-turn Dialogue: sharegpt, mtbench
• Function Call: BFCL (Berkeley Function Calling Leaderboard)

---

Installation

Environment Requirements

Python Version: Only Python 3.10, 3.11, or 3.12 is supported.

# Create conda environment
conda create --name ais_bench python=3.10 -y
conda activate ais_bench

Install from Source

git clone https://github.com/AISBench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517