| name | testing |
| description | Testing guide and pre-commit testing strategy for PTO Runtime. Use when running tests, adding tests, or deciding what to test before committing. |
Testing
Test Types
- Python unit tests (ut-py) (
tests/ut/): Standard pytest tests for the Python compilation pipeline and nanobind bindings. Run with pytest tests/ut. Tests declaring @pytest.mark.requires_hardware[("<platform>")] auto-skip unless --platform points to a matching device.
- C++ unit tests (ut-cpp) (
tests/ut/cpp/): GoogleTest-based tests for pure C++ modules. Run with cmake -B tests/ut/cpp/build -S tests/ut/cpp && cmake --build tests/ut/cpp/build && ctest --test-dir tests/ut/cpp/build -LE requires_hardware --output-on-failure. Hardware-required tests carry a requires_hardware or requires_hardware_<platform> ctest label and are filtered via -LE.
- Scene tests (
examples/{arch}/*/, tests/st/{arch}/*/): End-to-end @scene_test classes declared inside test_*.py. Sim variants run cross-platform (Linux/macOS); hardware variants require the CANN toolkit and an Ascend device. Discovery is by pytest (batch) or python test_*.py (standalone); #591's parallel orchestrator handles device bin-packing and ChipWorker reuse automatically.
Running Tests
Important: Always read .github/workflows/ci.yml first to extract the current --pto-isa-commit and --pto-session-timeout values. These ensure reproducible builds by pinning the PTO-ISA dependency to a known-good commit.
Runtime rebuild decision
Before running tests, determine whether runtime binaries need recompilation:
| What changed | Rebuild needed? | How |
|---|
Runtime/platform C++ (src/{arch}/runtime/, src/{arch}/platform/) | Yes | Re-run pip install --no-build-isolation -e . (incremental via build/cache/) |
Nanobind bindings (python/bindings/) | Yes | Re-run pip install -e . |
| Python-only code, examples, kernels | No | Just re-run the test |
In CI, pip install . pre-builds all runtimes before tests run.
pytest tests/ut
pytest tests/ut --platform a2a3
cmake -B tests/ut/cpp/build -S tests/ut/cpp && cmake --build tests/ut/cpp/build
ctest --test-dir tests/ut/cpp/build -LE requires_hardware --output-on-failure
ctest --test-dir tests/ut/cpp/build -L "^requires_hardware(_a2a3)?$" --output-on-failure
pytest examples tests/st --platform a2a3sim \
--clone-protocol https --pto-isa-commit <commit> --pto-session-timeout <timeout>
pytest examples tests/st --platform a2a3 --device <range> \
--clone-protocol https --pto-isa-commit <commit> --pto-session-timeout <timeout>
pytest examples tests/st --platform a2a3sim --runtime host_build_graph \
--clone-protocol https --pto-isa-commit <commit>
pytest examples/a2a3/host_build_graph/vector_example --platform a2a3sim \
--clone-protocol https --pto-isa-commit <commit>
python examples/a2a3/host_build_graph/vector_example/test_vector_example.py \
-p a2a3sim --clone-protocol https --pto-isa-commit <commit>
Pre-Commit Testing Strategy
When changed files require testing (C++, Python, or CMake), follow these steps to decide what to test and how.
Step 1 — Platform Availability and Detection
command -v npu-smi &>/dev/null
| Result | Platforms to test |
|---|
| Found | <arch>sim (simulation) and <arch> (hardware) |
| Not found | Simulation only (default a2a3sim) |
When npu-smi is found, detect the platform by parsing chip name from npu-smi info output:
| Chip name contains | Platform |
|---|
910B or 910C | a2a3 (sim: a2a3sim) |
950 | a5 (sim: a5sim) |
Use the detected platform for all subsequent --platform flags. If the chip name is unrecognized, warn and default to a2a3.
Step 2 — Test Scope
Run git diff --name-only (or git diff --cached --name-only for staged changes) and match the first applicable rule:
| Changed paths | Scope | Command pattern |
|---|
src/{arch}/platform/* | Full (all runtimes) | pytest examples tests/st --platform <platform> |
src/{arch}/runtime/<rt>/* | Single runtime | pytest examples tests/st --platform <platform> --runtime <rt> |
examples/{arch}/<rt>/<ex>/* | Single example | python <ex>/test_*.py -p <platform> (or pytest <ex> --platform <platform>) |
tests/ut/* (Python) | Python UT only | pytest tests/ut (add --platform <platform> on a device runner) |
tests/ut/cpp/* | C++ UT only | cmake -B tests/ut/cpp/build -S tests/ut/cpp && cmake --build tests/ut/cpp/build && ctest --test-dir tests/ut/cpp/build -LE requires_hardware |
| Mixed (spans multiple categories) | Escalate to the widest matching scope | — |
Note on runtime C++ changes: When changed paths include src/{arch}/runtime/ or src/{arch}/platform/, re-run pip install --no-build-isolation -e . before testing to rebuild the runtime binaries in build/lib/ (incremental via build/cache/). There is no rebuild-on-import — editable.rebuild = false.
Step 3 — Parallel Strategy
Parallelism is handled by the #591 scheduler (simpler_setup/parallel_scheduler.py) based on --device and --max-parallel:
Simulation (a2a3sim): --max-parallel auto = min(nproc, len(--device)). Pass --device 0-15 for a big virtual pool; auto caps in-flight at the CPU count. Override with --max-parallel N on CPU-constrained runners.
Hardware (a2a3): --max-parallel auto = len(--device). One in-flight subprocess per physical device — each device runs a dedicated ChipWorker (see docs/ci.md).
Step 4 — Device Detection (hardware only)
When testing on a2a3, detect idle devices:
npu-smi info
Pick devices whose HBM-Usage is 0 and find the longest consecutive sub-range (at most 4). Pass as --device <start>-<end> (or --device <id> if only one idle device). If no idle device is found, skip hardware testing and warn.
Decision Tree
git diff --name-only
│
├─ Only docs/config? ──→ SKIP tests
│
└─ Code changed?
│
├─ Determine SCOPE (Step 2)
│ ├─ platform → full (pytest --platform ...)
│ ├─ runtime → single runtime (--runtime ...)
│ └─ example → single example (standalone test_*.py or pytest <ex>)
│
├─ Runtime C++ changed (src/{arch}/)? ──→ pip install --no-build-isolation -e . first
│
└─ npu-smi found?
├─ Yes → sim + hardware (idle devs, max 4)
└─ No → sim only
Adding a New Scene Test
- Create a directory under the appropriate arch and runtime:
- Examples:
examples/{arch}/<runtime>/<name>/
- Device-only scene tests:
tests/st/{arch}/<runtime>/<name>/
- Add
test_<name>.py with a @scene_test-decorated class (see docs/testing.md for the full template: CALLABLE, CASES, generate_args, compute_golden). End with if __name__ == "__main__": SceneTestCase.run_module(__name__) so the file runs standalone.
- Add kernel source files under
kernels/aic/, kernels/aiv/, and/or kernels/orchestration/ — referenced by CALLABLE["orchestration"]["source"] / CALLABLE["incores"][*]["source"] as paths relative to the test file.
- Pytest auto-discovers any
test_*.py under examples/ and tests/st/; no registration needed.
Related Skills
git-commit — Complete commit workflow (runs testing as a prerequisite)