ort-test

# Running ONNX Runtime Tests ONNX Runtime uses **Google Test** for C++ and **unittest** (preferred) / **pytest** for Python. ## C++ tests ### Test executables | Executable | What it tests | |---|---| | `onnxruntime_test_all` | Core framework, graph, optimizer, session tests | | `onnxruntime_provider_test` | Operator/kernel tests (Conv, MatMul, etc.) across execution providers | ### Two `attention_op_test.cc` files — don't confuse them There are two same-named files testing **different operators**. Both build into `onnxruntime_provider_test`: | Path | Operator | gtest suite | |---|---|---| | `test/providers/cpu/llm/attention_op_test.cc` | **ONNX-domain** `Attention` (opset 23/24) | `AttentionTest.*` | | `test/contrib_ops/attention_op_test.cc` | **contrib** MultiHeadAttention / GroupQueryAttention | `ContribOpAttentionTest.*` | The MEA negative-offset regression tests (`Attention_Causal_NonPadKVSeqLen_MEA_*`, e.g. `..._MEA_NegOffset_ForceFlashDisabled_FP16_CUDA`) live in the **providers/cpu/llm** file — the ONNX-domain op. Use `--gtest_filter` to select specific tests: ```bash ./onnxruntime_provider_test --gtest_filter="*Conv3D*" ``` ### Running tests **Always run from the build output directory** — tests may fail to find dependencies otherwise. ```bash # Linux cd build/Linux/Release ./onnxruntime_provider_test --gtest_filter="*TestName*" # macOS cd build/MacOS/Release ./onnxruntime_provider_test --gtest_filter="*TestName*" # Windows cd build\Windows\Release .\onnxruntime_provider_test.exe --gtest_filter="*TestName*" ``` You can also run all tests via the build script (assumes a prior successful build): ```bash ./build.sh --config Release --test .\build.bat --config Release --test # Windows ``` ### Locating the build output directory The default path follows the pattern `build/<Platform>/<Config>/` where Platform is `Linux`, `MacOS`, or `Windows`. With Visual Studio multi-config generators on Windows, the config may appear twice (e.g., `build/Windows/Release/Release/`). The path can also be customized via `--build_dir`. If you can't find a test binary, search for it: ```powershell # Windows Get-ChildItem -Path build -Recurse -Filter "onnxruntime_provider_test.exe" | Select-Object -ExpandProperty FullName # Linux/macOS find build -name "onnxruntime_provider_test" -type f ``` ## Python tests Use `pytest` as the test runner: ```bash pytest onnxruntime/test/python/test_specific.py # entire file pytest onnxruntime/test/python/test_specific.py::TestClass::test_method # specific test pytest -k "test_keyword" onnxruntime/test/python/ # by keyword ``` Python test naming convention: `test_<method>_<expected_behavior>_[when_<condition>]` ## Agent tips - **Activate a Python virtual environment** before running tests. See "Python > Virtual environment" in `AGENTS.md`. - **Beware false-green results** — a green run does not always prove anything. See the "False-green taxonomy" section below for the four ways a test can pass without testing your change. - **Redirect test output to a file** (e.g., `> test_output.txt 2>&1`) — output can be large. - For C++ tests, verify the build directory exists and a prior build completed before running. - Use `--gtest_filter` to run a targeted subset when the full suite takes too long. - **Running WebGPU tests locally on Linux without a GPU** — WebGPU op tests build into `onnxruntime_provider_test` and can run against a software Vulkan adapter (Mesa lavapipe). See the `webgpu-local-testing` skill. ## False-green taxonomy — ways a test can "pass" without proving anything A green result is not always a real pass. Watch for all five modes: 1. **Zero-match filter.** A `--gtest_filter` that matches no tests still exits 0 (green). Confirm the `[==========] N tests ran` line is non-zero — a zero-match run prints `0 tests from 0 test suites`. Many operator/kernel gtests run only in **`onnxruntime_provider_test`** (CI runs this), NOT `onnxruntime_test_all`; the wrong binary matches nothing and looks green. 2. **Stale binary from an incremental build.** If the build did not actually recompile your change (e.g. a header not tracked by the compiler's depfile), the "passing" run executes the OLD code. A test that was failing cannot truly flip to passing without a real rebuild — treat an unexpected FAIL→PASS with suspicion and confirm the linked artifact's mtime advanced. CUDA/CUTLASS instance (nvcc depfiles don't track `cutlass_fmha/*.h`): see the `cuda-cutlass-fmha-incremental-rebuild` skill. 3. **Checking the wrong artifact's freshness.** With a dlopen'd shared provider (e.g. `libonnxruntime_providers_cuda.so`), the test executable is NOT relinked when the provider recompiles — its mtime stays old while the `.so` advances. Verify the artifact that actually links your change, not the test exe. Detail: `cuda-cutlass-fmha-incremental-rebuild` skill. 4. **A correct fallback path masks the intended path.** A value-only assertion can pass via a *different, correct* code path without ever exercising the one you meant to test (e.g. a test meant for MEA silently handled by the unfused fallback). Assert/verify **which path ran**, not just the output value — see "Verify which path/kernel actually executed" below. 5. **Arch-portability false-green (verified on only one GPU arch).** A CUDA kernel that launches on a large-dynamic-smem arch (e.g. sm90/H100, ~227KB) can **fail to launch** on a smaller opt-in cap (sm86/89 ~99KB, sm80 ~163KB) with `CUDA failure 1: invalid argument` — and a path with no fallback (e.g. ORT's MEA) turns that into a hard error, not a silent degrade. So a green run on your local GPU can mask a launch failure on CI's arch. Verify arch-portability, or pick a config whose shared-memory footprint fits **every** target arch (e.g. a small `head_size`). Concrete instance: CUTLASS MEA `head_size=512` FP16 exceeds sm86's smem opt-in cap and dies at launch — live bug #28388 (the `cuda-attention-kernel-patterns` skill §1 has the dispatch detail). ## Verify which path/kernel actually executed Value equality alone does not prove the intended code path ran — a correct fallback can produce the right answer (false-green mode 4 above). When a test targets a specific kernel/path, confirm it actually dispatched there instead of trusting the output: - Enable verbose logging and check the dispatch log line. ORT attention logs one of these exact strings (`core/providers/cuda/llm/attention.cc`): - `ONNX Attention: using Flash Attention` (:1400) - `ONNX Attention: using Memory Efficient Attention` (:1451) - `Attention: using unified unfused path` (:1482) — note: **no `ONNX ` prefix** and it reads "unified unfused path", not "Unfused". - Or force the path via the relevant env var / build config AND add a compile-time guard so the test **SKIPs** (not silently passes) when the target path is unavailable — e.g. `SKIP_IF_MEA_NOT_COMPILED`. Operator-specific routing/forcing details: `cuda-attention-kernel-patterns` skill §1/§7.