trt-strong-typing-migration

# Migrate Weakly-Typed to Strongly-Typed TensorRT Build Strong typing is the recommended build mode for TensorRT 11.x; weak typing was deprecated in 10.12 and is scheduled for removal. This skill migrates a weakly-typed build flow to a strongly-typed equivalent, one path per entry point (Python network, `trtexec`, C++ builder), plus the ModelOpt AutoCast pre-step needed when the source is an ONNX file that must run in mixed precision. **Version behavior.** On **TensorRT 11.x strong typing is unconditional** — every network is strongly typed, the `kSTRONGLY_TYPED` creation flag is accepted but ignored, and `trtexec --stronglyTyped` is a no-op (default-on). On **10.12–10.x** you opt in explicitly via that flag / `--stronglyTyped`. Setting it is therefore always safe: required on 10.x, harmless on 11.x. Either way the migration is the same — remove the precision *hints* (gone in 11.x) and move precision into the graph. ## When to Use | Scenario | Use this skill? | |----------|-----------------| | Existing Python build calls `config.set_flag(trt.BuilderFlag.FP16 / INT8 / TF32)` | Yes | | Existing C++ build calls `config->setFlag(BuilderFlag::kFP16 / kINT8 / kTF32)` | Yes | | Existing `trtexec` line uses `--fp16`, `--int8`, `--best`, `--bf16` and the developer wants strong typing | Yes | | Build emits "weak typing is deprecated, use STRONGLY_TYPED" warning at runtime | Yes | | Build fails with "flag X is incompatible with strongly typed networks" | Yes | | Developer has a FP32 ONNX file and wants a strongly-typed mixed FP16 engine | Yes — start at Step 1 (AutoCast) | | Developer is importing a brand-new ONNX file with no existing build flow | No — use `trt-onnx-quickstart` first; come back here to switch the flag | | Developer's engine produces wrong outputs *after* migration | No — use `trt-quantization-accuracy` | | Developer is on TensorRT 10.11 or earlier | No — strong typing is fully available from 10.12 onward; upgrade before migrating | ## Prerequisites 1. **TensorRT 10.12 or later (11.x recommended).** Check: ```bash python3 -c "import tensorrt; print(tensorrt.__version__)" trtexec --help 2>&1 | grep -i stronglyTyped ``` The version check is the determinant — strong typing is available from 10.12 on. The `trtexec` probe just confirms the CLI flag for the trtexec path; a missing `--stronglyTyped` means a pre-10.12 install. 2. **ModelOpt installed when the source is an ONNX file requiring mixed precision.** ```bash pip install nvidia-modelopt onnx onnxruntime ``` If the ONNX file is already strongly typed (contains explicit `Cast` nodes and FP16 initializers), AutoCast is not needed — skip directly to Step 2. 3. **A working baseline.** Capture the weakly-typed engine's outputs on a known input set before migrating. Strong typing is stricter, and any silent precision substitutions weak typing was making will become observable. 4. **The verifier script.** `scripts/verify.sh` runs `migrate.py` against a committed sample and asserts the rewrite is well-formed. Run it before editing anything to confirm the helper works. ## Migration paths There are three migration paths — Python builder, `trtexec`, C++ builder. **Infer the path from the query when signals are clear; ask only if genuinely uncertain.** Match the strongest signal: | Signal in the request | Path | Entry point | |------------------------|------|-------------| | Python tokens: `trt.BuilderFlag`, `set_flag(`, `create_network(`, `.py` | **A** | Python `INetworkDefinition` builder | | Standalone `trtexec --fp16` / `--int8` / `--best` / `--bf16` | **B** | `trtexec` CLI | | C++ tokens: `BuilderFlag::`, `createNetworkV2`, `->setFlag`, `.cpp` / `.h` | **C** | C++ builder | | Multiple signals | — | do each, most-specific first | | No usable signal (generic "how do I migrate") | — | ask which entry point, or proceed with Python and say you assumed it | The migration preserves existing precisions (kFP16 → FP16 cast, kINT8 → Q/DQ) and is identical whether the network came from a parser or was hand-built — only Step 1 (AutoCast) is ONNX-specific, so source and precision goal rarely need a question. Inferring the path is a low-stakes default — it does **not** extend to destructive operations, which always need confirmation. `migrate.py` defaults to dry-run (prints a diff); the `--write` flag is the human-in-the-loop confirmation. Treat any other destructive step (overwriting a `.plan`, replacing an ONNX) the same way: show, confirm, then act. For a strongly-typed INT8 model straight from an ONNX file (not a migration of existing builder code), jump to Step 1's ModelOpt quantization recipe. ## Step 1 — (ONNX source only) AutoCast the model to mixed precision Skip if the source is not ONNX, or if the ONNX file is already strongly typed. A FP32 ONNX file built for a weakly-typed `--fp16` flow carries no precision information — TensorRT was free to pick FP16 or FP32 per layer. Strong typing requires the graph itself to declare precision via `Cast` operators and typed initializers. ModelOpt's AutoCast inserts these. Use the ModelOpt CLI — it's the supported entry point and takes `.npz` calibration directly. Use `--calibration_data` for both tools (autocast's canonical flag; quantization's abbreviation of `--calibration_data_path`): ```bash # FP16 / BF16 mixed-precision AutoCast python -m modelopt.onnx.autocast \ --onnx_path model.fp32.onnx \ --output_path model.mixed.onnx \ --low_precision_type fp16 \ --calibration_data calib.npz # INT8 / FP8 strongly-typed quantization (produces ONNX with explicit Q/DQ) python -m modelopt.onnx.quantization \ --onnx_path model.fp32.onnx \ --output_path model.int8.onnx \ --quantize_mode int8 \ --high_precision_dtype fp16 \ --calibration_data calib.npz ``` Keep accuracy-sensitive ops in FP32 via AutoCast's node/op-type exclusion lists (e.g. exclude `MatMul`/`LayerNormalization` on transformer-heavy networks), and keep FP32 graph I/O when downstream stages expect it. See the [Model Optimizer docs](https://nvidia.github.io/Model-Optimizer) for the exact module and flags for your ModelOpt version. Verify the converted model in ONNX Runtime before passing it to TRT: ```python import onnxruntime, numpy as np sess = onnxruntime.InferenceSession("model.mixed.onnx", providers=["CPUExecutionProvider"]) # Compare a few outputs against the FP32 model with np.allclose(..., rtol=5e-3, atol=5e-3). ``` If parity fails here, **stop** — the conversion is wrong and no amount of TRT engine work will recover it. Re-run with tighter exclusion lists. **Next step — build the strongly-typed plan from the converted ONNX:** ```bash # For the INT8 case (modelopt.onnx.quantization output) trtexec --onnx=model.int8.onnx --stronglyTyped --saveEngine=model.int8.plan # For the mixed-precision case (modelopt.onnx.autocast output) trtexec --onnx=model.mixed.onnx --stronglyTyped --saveEngine=model.fp16.plan ``` **Critical**: do NOT pass `--int8` or `--fp16` alongside `--stronglyTyped`. Those flags are removed in TRT 11, and on TRT 10.12+ they conflict with strong typing. The precision is already encoded in the ONNX (via explicit Cast / Q / DQ nodes); the builder honors it. The reference recipe for the end-to-end INT8 flow is `samples/python/strongly_type_autocast/sample.py` in `tensorrt-oss`. ## Step 2A — Migrate a Python `INetworkDefinition` builder Mechanical replacements: | Weakly typed | Strongly typed | |--------------|----------------| | `builder.create_network(0)` or `builder.create_network(1 << int(NetworkDefinitionCreationFlag.EXPLICIT_BATCH))` | `builder.create_network(1 << int(NetworkDefinitionCreationFlag.STRONGLY_TYPED))` | | `config.set_flag(trt.BuilderFlag.FP16)` | **Remove.** Types come from the network. | | `config.set_flag(trt.BuilderFlag.BF16)` | **Remove.** | | `config.set_flag(trt.BuilderFlag.INT8)` | **Remove.** Quantization expressed via Q/DQ nodes in the graph. | | `config.set_flag(trt.BuilderFlag.TF32)` | **Keep.** `kTF32` is retained in 11.x — it allows (not requires) TF32 for FP32 tensors and is orthogonal to typing. | | `layer.precision = trt.float16` | **Remove.** Cast the input tensor instead (`network.add_cast`). | | `layer.set_output_type(0, trt.float16)` | **Remove.** Output type follows the network; for cast/quantize/dequantize/fill layers use `set_to_type` instead. | | `builder.platform_has_fast_int8 / has_fast_fp16` checks gating the above | **Remove the gating logic** along with the flag. | Worked rewrite (the MNIST builder from `samples/python/network_api_pytorch_mnist/sample.py` already uses the strongly-typed form — note `NetworkDefinitionCreationFlag.STRONGLY_TYPED` and the absence of any `BuilderFlag.FP16` call): ```python builder = trt.Builder(TRT_LOGGER) network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)) config = builder.create_builder_config() config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, common.GiB(1)) # ... populate network ... plan = builder.build_serialized_network(network, config) ``` If the network is built via the ONNX parser, the parser respects strong typing automatically — no code change is needed beyond the flag. Where a removed `precision` / `set_output_type` hint actually mattered, encode it in the graph: cast a tensor's dtype with `add_cast`, or insert an explicit Quantize/Dequantize pair for INT8 (the scale is a build-time constant; ModelOpt derives it from calibration, or supply it by hand): ```python # kFP16 hint -> explicit cast on the tensor: cast = network.add_cast(some_tensor, trt.float16) # kINT8 hint -> explicit Q/DQ pair (per-tensor scalar scale). NOTE: trt.Weights does # not copy the numpy buffer, so q_scale must stay alive until the engine is built. q_scale = np.array(1.0 / 127.0, dtype=np.float32).reshape(()) # rank-0 scalar scale = network.add_constant(shape=(), weights=trt.Weights(q_scale)).get_output(0) q = network.add_quantize(some_tensor, scale, trt.int8).get_output(0) dq = network.add_dequantize(q, scale, trt.float32).get_output(0) # Feed dq into the consumer; TensorRT fuses the Q/DQ into it so it runs in INT8. ``` Drive the rewrite from `scripts/migrate.py`: ```bash python3 scripts/migrate.py path/to/build.py # dry-run diff python3 scripts/migrate.py path/to/build.py --write # rewrite in place ``` The script edits AST nodes (not regexes) — it understands `create_network` and `set_flag` call shapes, so it is safe on files that mix conventions. ## Step 2B — Migrate a `trtexec` command line Replacements: | Weakly typed flag | Strongly typed equivalent | |-------------------|---------------------------| | `--fp16` | **Remove.** Precision comes from the ONNX. See the `--stronglyTyped` note below. | | `--bf16` | **Remove.** Same as `--fp16`. | | `--int8` | **Remove.** Q/DQ must be present in the ONNX. | | `--best` | **Remove.** Not allowed with `--stronglyTyped` on 10.12–10.x (see Common Errors). | | `--stronglyTyped` | **Version-gated.** Required on 10.12–10.x to opt into strong typing; a no-op (default-on) on 11.x — drop it or leave it, your choice. | | `--noTF32` | **Keep.** `kTF32` is retained in 11.x; `--noTF32` still disables TF32 and is orthogonal to typing. | | `--precisionConstraints=obey` / `prefer` | **Remove.** Constraints are expressed in the graph. | | `--layerPrecisions=...` | **Remove.** Use ONNX `Cast` nodes / AutoCast. | | `--layerOutputTypes=...` | **Remove.** | Worked rewrite: ```bash # Before (weakly typed) trtexec --onnx=model.onnx --fp16 --saveEngine=model.fp16.plan \ --minShapes=input:1x3x224x224 --optShapes=input:8x3x224x224 \ --maxShapes=input:16x3x224x224 # After (strongly typed; ONNX has been AutoCast'd to mixed FP16) # On 11.x --stronglyTyped is the default and a no-op — the line below is equivalent # with or without it. On 10.12–10.x it is REQUIRED, or the build stays weakly typed. trtexec --onnx=model.mixed.onnx --stronglyTyped --saveEngine=model.fp16.plan \ --minShapes=input:1x3x224x224 --optShapes=input:8x3x224x224 \ --maxShapes=input:16x3x224x224 ``` The `trtexec` README's [Example 6](https://github.com/NVIDIA/TensorRT/blob/main/samples/trtexec/README.md#example-6-create-a-strongly-typed-plan-file) is the canonical reference. ## Step 2C — Migrate a C++ builder Replacements mirror the Python path: | Weakly typed (C++) | Strongly typed (C++) | |---|---| | `builder->createNetworkV2(0)` | `builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED))` | | `config->setFlag(BuilderFlag::kFP16)` | **Remove.** | | `config->setFlag(BuilderFlag::kBF16)` | **Remove.** | | `config->setFlag(BuilderFlag::kINT8)` | **Remove.** | | `config->setFlag(BuilderFlag::kTF32)` | **Keep.** `kTF32` is retained in 11.x — orthogonal to typing. | | `config->setFlag(BuilderFlag::kOBEY_PRECISION_CONSTRAINTS / kPREFER_PRECISION_CONSTRAINTS)` | **Remove.** No per-layer precision exists to constrain. | | `layer->setPrecision(DataType::kHALF)` | **Remove.** Cast the input tensor instead (`addCast`). | | `layer->setComputePrecision(...)` | **Remove.** Deprecated 10.16, removed; compute precision follows input dtypes. |