onnx-opset-bump-checklist

# ONNX Opset / Version Bump Checklist for ONNX Runtime Repeatable process for upgrading the ONNX dependency in ONNX Runtime. Recurs on every ONNX release. Canonical reference PRs: **#27601** (ONNX 1.21, incremental rc1→formal — best example), #26579 (1.20.1), #25678 (1.19). See also `docs/How_To_Update_ONNX_Dev_Notes.md`. ## 0. Strategy: RC → formal (incremental) ONNX partner-validates each release candidate against ORT **before** publishing the formal release — that is the whole point of the integration issue. When the issue points at a `rel-X.Y.0` branch: 1. **Integrate the RC first.** Pin by the release-branch **HEAD commit** (the `vX.Y.0rcN` git *tag* usually does not exist yet). Run the full build + tests; file ONNX bugs upstream (tag the ONNX release manager) for any ONNX-side defects found. 2. **Re-pin per RC** (rc2, rc3, …). Usually only the Group-A version-plumbing files change. **Drop any `onnx.patch` hunks that ONNX has merged upstream** in the new RC. 3. **Re-pin to the formal tag** `vX.Y.0` once the GitHub Release is published. Note the release can sit as a **draft** (no git tag, `git/ref/tags/vX.Y.0` → 404) for a while. ```bash gh api repos/onnx/onnx/git/ref/heads/rel-X.Y.0 --jq '.object.sha' # RC branch HEAD curl -sL https://raw.githubusercontent.com/onnx/onnx/rel-X.Y.0/VERSION_NUMBER # e.g. X.Y.0rcN gh api 'repos/onnx/onnx/releases?per_page=5' --jq '.[].tag_name' # '' => draft ``` ## 1. File taxonomy — what to change Grouped so parallel work is safe. **Group A must land first** (the tree must build before B/C/D can be validated). Throughout this section, bold letters in parentheses (e.g. gotcha **a**, **g**) refer to the lettered gotchas **a**–**i** defined in §4. ### Group A — version plumbing (always required, mechanical) | File | Note | |---|---| | `cmake/deps.txt` (`onnx;` line) | archive URL + **SHA1 of the `.zip`** (see §2). ⚠️ Before advancing this pin, verify the target commit still ships `onnx/backend/test/data/node/` — see gotcha **p** (#7959 deletes the on-disk node-test corpus → silent-green CI). | | `cmake/external/onnx` (submodule) | `git -C cmake/external/onnx fetch && git -C cmake/external/onnx reset --hard <sha> && git add cmake/external/onnx` | | `cmake/vcpkg-ports/onnx/vcpkg.json` | `version-semver`; reset `port-version` to 0 on a real version bump | | `cmake/vcpkg-ports/onnx/portfile.cmake` | `REF` + **SHA512 of the `.tar.gz`** (see §2). RC = bare commit as `REF`; formal = `REF "v${VERSION}"` | | `cmake/vcpkg-ports/onnx/binskim.patch` | keep **byte-identical** to `onnx.patch` (see §3) | | `cmake/patches/onnx/onnx.patch` | rebase to the new source (see §3) | | `cmake/vcpkg-ports/onnx/fix-dependency-protobuf.patch`, `fix-cmakelists.patch` | re-`diff` only if they fail to apply | | **The 7 `requirements.txt` files** with `onnx==X.Y.Z` | `onnxruntime/test/python/requirements.txt`; `tools/ci_build/github/linux/python/requirements.txt`; `tools/ci_build/github/windows/python/requirements.txt`; `tools/ci_build/github/linux/docker/scripts/{requirements.txt,manylinux/requirements.txt,lort/requirements.txt}`; `tools/ci_build/github/linux/docker/inference/aarch64/python/cpu/scripts/requirements.txt`. Confirm with `git grep -n "onnx==<OLD>"` (e.g. `onnx==1.21`) — grep the **old** pin you are replacing, not bare `onnx==1`. ⚠️ **Do NOT bump all matches.** `git grep -n "onnx==1"` also lists 3 transformers-model files — `onnxruntime/python/tools/transformers/models/{llama,phi2,stable_diffusion}/requirements.txt` — that are **intentionally frozen at `onnx==1.18.0`**. Leave those alone; only the 7 CI files above get the new pin. | | **The 4 QNN/android CI yaml + the minimal `requirements-materialize-onnx-node-tests.txt` they now use** (node-test materialization legs — NOT covered by the `requirements.txt` grep above) | `tools/ci_build/github/azure-pipelines/linux-qnn-ci-pipeline.yml`; `tools/ci_build/github/azure-pipelines/win-qnn-arm64-ci-pipeline.yml`; `tools/ci_build/github/azure-pipelines/android-arm64-v8a-QNN-crosscompile-ci-pipeline.yml`; `.github/workflows/windows_qnn_x64.yml`. Each does `pip install -r requirements-materialize-onnx-node-tests.txt` (repo-root file; the linux+android legs keep `--user`, the two Windows legs omit it) feeding the `onnxruntime_MATERIALIZE_ONNX_NODE_TESTS` gate (see gotcha **p**). **Bump the pin IN the requirements file, in lockstep with `cmake/deps.txt` + the numpy MIN pins** — do NOT re-inline. Confirm with `git grep -n "onnx==<OLD>"`. (The pins now live in a **dedicated minimal onnx+numpy `requirements*.txt` at the repo root** — the sole pip directory Dependabot scans — so Dependabot can track these two upgrades while the QNN/Android legs still avoid dragging in the heavy docker `requirements.txt`. This REVERSES the earlier "inline by design" rationale: the pins must now stay pin-locked with the CMake gate, which HARD-FATALs on installed!=pinned.) | ### Group B — opset enablement | File | Change | |---|---| | `onnxruntime/core/optimizer/transpose_optimization/optimizer_api.h` | `kMaxSupportedOpset` → new max opset (e.g. 26 → 27) | | `onnxruntime/core/providers/cpu/cpu_execution_provider.cc` | add `// Opset N` forward-declares + `BuildKernelCreateInfo<...>` entries for new/updated CPU kernels; mirror the previous opset block exactly | | `onnxruntime/core/graph/contrib_ops/contrib_defs.h`, `dml_ops/dml_defs.h` | apply ONNX-header-driven `OpSchemaRegisterOnce` macro fixes **only if the build emits those errors** | > **🆕 Tradition: bump EVERY EP that registers the op, in the SAME PR.** When an op's kernel set > changes for the new opset (e.g. `Range` gaining fp16/bf16 at opset 27), version-split / bump > that op's registration in **every** EP that registers it — **CPU and CUDA at minimum** — so no > EP silently lags behind CPU and the advertised opset boundaries stay consistent. **Even an > open-ended kernel that already binds the new opset** (e.g. CUDA `Range` at `SinceVersion(11)`, > which already matches opset-27 nodes) should still be **version-split** for convention/clarity > and to keep the kernel's advertised boundary matching the schema. Worked example — **PR #28754** > split `Range` into `[11,26]` + `27` in **both** CPU and CUDA (verified), keeping the same > numeric type set and deferring fp16/bf16 to ONNX function-expansion. **EP checklist when an op's kernel set changes for the new opset:** - [ ] For **each** EP, `grep -rn "<Op>)" onnxruntime/core/providers/<ep>/` (and its `*_execution_provider.cc`) to find every registration of the changed op. - [ ] EPs that register ONNX kernels via the `ONNX_OPERATOR_[VERSIONED_]KERNEL[_CLASS_NAME]` macros — **cpu**, **cuda**, **js**, and **rocm** if it registers the op (rocm is often hipified from cuda): version-split each into `[prev_start, N-1]` + a new `N` registration (class forward-declare **and** `BuildKernelCreateInfo` entry). - [ ] EPs with their **own** registration systems assess per their conventions, not the macro split — **dml** (`REG_INFO(ver, Op, …)` in `OperatorRegistration.cpp`), **webgpu**, **coreml/nnapi/qnn/openvino/migraphx**. A partition/capability check (e.g. MIGraphX's `optype == "Range"`) is **not** a kernel registration and needs no split. - [ ] Bump the EP `GetMaxSupportedOpSet` ceilings (**coreml/nnapi/vsinpu/webnn**) in lockstep — see §4 gotcha **b**. > **IR version is NOT bumped manually.** ORT reads `ONNX_NAMESPACE::Version::IR_VERSION` from > the ONNX headers (`onnxruntime/core/graph/model.cc`); it follows the submodule automatically. ### Group C — docs & test data (mostly auto-generated) | File | Change | |---|---| | `docs/OperatorKernels.md` | regenerate (see gotcha **e** — needs a built ORT module): `python tools/python/gen_opkernel_doc.py --output_path docs/OperatorKernels.md` | | `js/web/docs/webgl-operators.md` | `cd js/web && npm install && npm run build:doc` (the WebAssembly CI "Check out of dated documents" stage fails otherwise) | | `onnxruntime/test/testdata/onnx_backend_test_series_filters.jsonc` | exclude backend tests for ops whose kernels are deferred (gotcha **g**) | | `onnxruntime/test/testdata/onnx_backend_test_series_overrides.jsonc` | tolerance overrides for new tests | | `onnxruntime/test/onnx/TestCase.cc` | `broken_tests` entries for genuinely-unsupported cases | ### Group D — conditional (touch only if build/tests flag it — but see §4 gotchas) `onnxruntime/core/framework/kernel_type_str_resolver_utils.cc`, `onnxruntime/core/optimizer/layout_transformation/layout_transformation_potentially_added_ops.h`, `onnxruntime/core/optimizer/qdq_transformer/qdq_util.cc`, EP `base_op_builder.h` max-opset guards (gotcha **b**), optimizer fusion path-matchers (gotcha **a**, **c**), and any CPU op file hard-coding a `SinceVersion` ceiling for a changed op. ## 2. Archive-hash procedures `cmake/deps.txt` 3rd field = **SHA1 of the downloaded `.zip`** (verified: `sha1sum` of `v1.21.0.zip` equals the pinned `321d4acc...`): ```bash URL="https://github.com/onnx/onnx/archive/<commit-or-refs/tags/vX.Y.0>.zip" curl -sL -o onnx.zip "$URL" && sha1sum onnx.zip # paste: onnx;$URL;<sha1> ``` vcpkg `portfile.cmake` uses **SHA512 of the `.tar.gz`**: ```bash curl -sL -o onnx.tgz "https://github.com/onnx/onnx/archive/<commit-or-tag>.tar.gz" sha512sum onnx.tgz ``` Shortcut: pin a wrong hash and build — ORT/FetchContent prints the expected one. > The submodule field (a git commit SHA) and the deps.txt field (a SHA1 of the archive > *contents*) are **different by design** — never interchange them. > **vcpkg artifact must be mirrored** to the Microsoft vcpkg store before `--use_vcpkg` > builds can download it (Terrapin upload — see below). Not self-service; coordinate with > infra. This is gotcha **f**. ### Mirroring the ONNX archive to the MS vcpkg store (required before `--use_vcpkg` builds pass) The SHA512 in `portfile.cmake` references a `.tar.gz` that vcpkg downloads from the **Microsoft vcpkg artifact mirror**, not from GitHub. Until that exact archive is uploaded to the mirror, `--use_vcpkg` builds fail with a download/hash error. The upload is done with the **Terrapin Retrieval Tool** and is a **Windows + `az` auth + internal-credential** step — it **cannot** run from a generic Linux CI host. **Step 1 — auth (PowerShell):** ```powershell $authScope = 'https://mspmecloud.onmicrosoft.com/RebuildManager.Web/.default' $env:TRT_UPLOAD_AUTH_TOKEN = $(az account get-access-token --scope $authScope --query 'accessToken' --output tsv) ``` **Step 2a — commit-version (RC phase, archive keyed by commit SHA):** ```powershell C:\local\Terrapin\TerrapinRetrievalTool.exe -b https://vcpkg.storage.devpackages.microsoft.io/artifacts/ ` -a true -u Environment ` -p https://github.com/onnx/onnx/archive/<commit-sha>.tar.gz ` -s <sha512-of-tar.gz> ` -d "<build>\Windows\vcpkg\downloads\onnx-onnx-<commit-sha>.tar.gz.part" ``` **Step 2b — tag-version (formal phase, archive keyed by release tag):** ```powershell C:\local\Terrapin\TerrapinRetrievalTool.exe -b https://vcpkg.storage.devpackages.microsoft.io/artifacts/ ` -a true -u Environment ` -p https://github.com/onnx/onnx/archive/refs/tags/v<version>.tar.gz ` -s <sha512> ` -d "<build>\Windows\vcpkg\downloads\onnx-onnx-v<version>.tar.gz" ``` Key notes: - **(a)** `-s <sha512>` **MUST equal** the `portfile.cmake` SHA512 (the `sha512sum` of the same `.tar.gz` — see §2). A mismatch re-uploads the wrong blob and the hash check still fails. - **(b)** This is *why* `--use_vcpkg` builds fail with a download error until the upload lands — the mirror has no copy of the new archive yet. - **(c)** Microsoft-internal infra step: coordinate with the infra owner (e.g. **@snnn**). External / Linux CI **cannot** perform it (needs Windows, the Terrapin tool, and `az` creds). - **(d)** The **`cmake/deps.txt` path does NOT need this** — that build fetches the `.zip` straight from GitHub. Only the vcpkg (`--use_vcpkg`) path depends on the mirror. - **(e)** The `.part` suffix on the 2a `-d` path is vcpkg's **in-progress-download** temp name (vcpkg renames it to the final `.tar.gz` once the hash verifies). Match whatever the failing `--use_vcpkg` build actually requests in its `downloads/` dir — the RC run wrote a `.part`; the formal-tag run wrote the bare `.tar.gz`. When unsure, copy the exact path from the build's download error. - **(f) Why there is no GitHub fallback — `x-block-origin` (the actual failure mode).** `tools/ci_build/build.py` (`add_default_vcpkg_options`, the `--use_vcpkg_ms_internal_asset_cache`