Menu
Run ONNX Runtime tests. Use this skill when asked to run tests, debug test failures, or find and execute specific test cases in ONNX Runtime.
| name | ort-test |
| description | Run ONNX Runtime tests. Use this skill when asked to run tests, debug test failures, or find and execute specific test cases in ONNX Runtime. |
ONNX Runtime uses Google Test for C++ and unittest (preferred) / pytest for Python.
| Executable | What it tests |
|---|---|
onnxruntime_test_all | Core framework, graph, optimizer, session tests |
onnxruntime_provider_test | Operator/kernel tests (Conv, MatMul, etc.) across execution providers |
Use --gtest_filter to select specific tests:
./onnxruntime_provider_test --gtest_filter="*Conv3D*"
Always run from the build output directory — tests may fail to find dependencies otherwise.
# Linux
cd build/Linux/Release
./onnxruntime_provider_test --gtest_filter="*TestName*"
# macOS
cd build/MacOS/Release
./onnxruntime_provider_test --gtest_filter="*TestName*"
# Windows
cd build\Windows\Release
.\onnxruntime_provider_test.exe --gtest_filter="*TestName*"
You can also run all tests via the build script (assumes a prior successful build):
./build.sh --config Release --test
.\build.bat --config Release --test # Windows
The default path follows the pattern build/<Platform>/<Config>/ where Platform is Linux, MacOS, or Windows. With Visual Studio multi-config generators on Windows, the config may appear twice (e.g., build/Windows/Release/Release/). The path can also be customized via --build_dir.
If you can't find a test binary, search for it:
# Windows
Get-ChildItem -Path build -Recurse -Filter "onnxruntime_provider_test.exe" | Select-Object -ExpandProperty FullName
# Linux/macOS
find build -name "onnxruntime_provider_test" -type f
Use pytest as the test runner:
pytest onnxruntime/test/python/test_specific.py # entire file
pytest onnxruntime/test/python/test_specific.py::TestClass::test_method # specific test
pytest -k "test_keyword" onnxruntime/test/python/ # by keyword
Python test naming convention: test_<method>_<expected_behavior>_[when_<condition>]
AGENTS.md.> test_output.txt 2>&1) — output can be large.--gtest_filter to run a targeted subset when the full suite takes too long.Patterns and pitfalls for the ONNX domain Attention operator (opset 23/24) CUDA implementation. Use when modifying the dispatch cascade in core/providers/cuda/llm/attention.cc, writing mask/bias CUDA kernels, debugging attention test routing, or adding features to the ONNX Attention op. NOT for contrib domain MultiHeadAttention/GroupQueryAttention.
When reviewing or fixing Python code that uses setattr() with user-controlled kwargs to configure C++ extension objects (SessionOptions, RunOptions, etc.) in ONNX Runtime. Use this to apply the allowlist pattern that prevents arbitrary file writes and other attacks via reflected property access.
Build ONNX Runtime from source. Use this skill when asked to build, compile, or generate CMake files for ONNX Runtime.
Lint and format ONNX Runtime code. Use this skill when asked to lint, format, or check code style for C++ or Python files in ONNX Runtime.