sentis-model-converter

name

sentis-model-converter

description

Convert ANY model (PyTorch / HF transformers / prebuilt ONNX) into ONNX that Unity Sentis 2.6 actually imports AND runs correctly. Owns the whole conversion layer: choosing the right tool (torch.onnx.export vs optimum-cli vs Olive vs onnx-graphsurgeon), structural rewrites (KV-cache as plain I/O, unmerged seq2seq decoders, hoisting RNN state to tensors, splitting pre/post-processing into their own graphs), graph surgery on unsupported or attribute-dropping ops, weight inlining, f16 downcast for the 2 GB cap, and the ORT-parity verification ladder. Use whenever the task mentions: exporting or converting a model for Sentis/Unity, ONNX export flags, KV-cache export, onnx-graphsurgeon / graph surgery / editing an ONNX graph, optimum-cli or Olive, a model that imports into Unity but outputs garbage, or unsupported-operator import errors — even if the user doesn't say "convert". (Model *selection* = sentis-model-scout; fine-tuning = sentis-training-pipeline; C# runtime = sentis-inference.)

Sentis Model Converter — any model → Sentis-importable, parity-verified ONNX

POSITION in the 4-skill split: sentis-model-scout (find & judge) → THIS SKILL (convert: export / structural rewrite / surgery / verify) → sentis-inference (C# runtime). sentis-training-pipeline (recipe/data/fine-tune) hands its checkpoint here for export — the export path is identical for base and fine-tuned weights, so convert with base weights FIRST, then swap in the checkpoint.

The hard truth this skill encodes: Sentis does NOT import most ONNX cleanly, and "imports with 0 errors" ≠ "works". A general model answers "just export to ONNX" — that answer wastes days. Every rule below is verified against Sentis 2.6.1 in this project or the official manual (reference/sentis_supported_ops.md). The only real acceptance gate is the triple-parity gate — original vs ORT vs Sentis (§7).

0. Inputs — establish these before converting anything

1. What you have: PyTorch source (best) / HF repo / only a prebuilt ONNX (worst).
2. The I/O contract Unity needs: fixed input shapes, KV-cache tensors, state tensors, what pre/post-processing lives in C# vs in a graph. Write it down; it drives every choice.
3. Size budget: single inline .onnx hard cap = 2 GB (protobuf). fp32 > 2 GB ⇒ f16 first.
4. If the model wasn't scanned yet, run scripts/scan_onnx.py <model.onnx|HF-repo> — it flags contrib ops, control flow, external data, opset, >8-dim/string/sparse tensors, size.

1. Choose the path (decision table)

Situation	Path
HF model WITH an optimum exporter config (whisper/BERT/most mainstream)	optimum-cli / main_export — unfused + unmerged out of the box (§2b)
HF model WITHOUT an optimum config (gemma3-class), or custom I/O needed (KV wrapper, state hoisting, baked pre/post)	torch.onnx.export eager wrapper (§2a)
Only a prebuilt ORT-optimized ONNX, no source access	graph surgery (§6) or the custom Sentis importer-converter (reference/contrib_op_converter_cases.cs); prefer finding the original PyTorch repo instead
Graph is Sentis-clean but > 2 GB or fp32-fat	Olive size/dtype pass or onnxconverter-common f16, or torch .half() before export (§2c)
Good graph, a few bad nodes (Gelu-tanh, accidental If, baked NMS, sidecar weights)	targeted surgery scripts (§6) — smallest possible diff
Vendor CV model (detector family)	use the vendor's own exporter with an explicit opset, dynamic=False, and nms=False (NMS/decode in C#, §4e)

2. Tool cheat sheet (verified gotchas — these override priors)

a. torch.onnx.export — maximum control; required when the export must CHANGE model I/O

• Wrap the model in an nn.Module that defines the exact Unity-facing I/O: KV tensors as *past_flat varargs, explicit numeric 4D masks fed as inputs (deletes the fragile in-graph mask subgraph), recurrent state as plain tensors, pre/post baked in or out deliberately.
• Always attn_implementation="eager" for transformers — SDPA/fused paths emit ops Sentis lacks; eager decomposes attention/RoPE/RMSNorm into supported primitives.
• transformers 4.5x + legacy exporter (dynamo=False) crashes on vmap mask creation (functorch ... unordered_map::at) → use dynamo=True + dynamic_shapes (NOT dynamic_axes; from torch.export import Dim; DYN = Dim.DYNAMIC). With a *past_flat wrapper torch.export sees 3 logical inputs → shapes = ({0:DYN,1:DYN},{0:DYN,1:DYN}, tuple-of-2N {0:DYN,2:DYN}).
• BERT-class encoders: the legacy exporter (dynamo=False) works and is simpler.
• dynamo export writes model.onnx + model.onnx.data → MUST inline afterwards (§5).
• 🔴 NEVER build/run the reference forward under torch.inference_mode() — tensors it produces throw "Inference tensors cannot be saved for backward" the instant the ONNX tracer (autograd) runs on them. Use torch.no_grad() + model.requires_grad_(False) instead.
• Export only the minimal sub-graph the target task needs — don't export model(). For a composite research model (detector + text encoder + memory + task heads), read the actual predict path and wrap ONLY the submodules it calls, composed from the model's own attributes (verified: EfficientSAM3 point-prompt = backbone.forward_image + sam_prompt_encoder + sam_mask_decoder, skipping the text/memory towers entirely). Smaller graph, smaller file, fewer unsupported-op risks.

b. optimum — first choice for HF models it supports

• optimum 2.x split the exporter into a separate package: install BOTH optimum AND optimum-onnx (the optimum[exporters] extra alone leaves optimum.exporters.onnx missing).
• main_export(repo, out, task="<task>-with-past", opset=18, no_post_process=True, monolith=False) → task-aware, unfused graphs, unmerged decoders (encoder_model / decoder_model / decoder_with_past_model, no If-carrying decoder_model_merged), weights inline per file.
• If the HF repo only ships an ORT-optimized ONNX, run optimum against the original repo.

c. Olive — size/dtype passes ONLY

• Use for float16 conversion when re-exporting from torch is painful.
• ⚠ NEVER its ORT transformer-optimization (fusion) passes — they ADD com.microsoft ops, the exact thing Sentis rejects. NEVER INT4/INT8 passes — the quantized-op family (QLinear*, MatMulInteger…) cannot import, ever. Quantize at Sentis runtime (Float16/Uint8) instead.

d. onnx-graphsurgeon — topology edits on an existing graph

• Install: pip install onnx_graphsurgeon --extra-index-url https://pypi.ngc.nvidia.com (also mirrored on PyPI). API notes + recipes: reference/graphsurgeon.md.
• Use when the fix is rewiring topology: removing/replacing nodes, extracting subgraphs, re-routing tensors, injecting nodes (Concat for KV, Slice for onesided-STFT workaround).
• For linear in-place decompositions (one op → primitive chain) plain onnx.helper is equally good and dependency-free — scripts/surgery_gelu.py is the pattern to copy.
• Both approaches: prefer graph-only edits that leave weights untouched (load_external_data=False) — or inline first (§5) and edit the self-contained file.

3. Export contract (what every path must produce)

• opset explicit and ≤ 25 (Sentis window: 7–25). Outside it "might import", unpredictably.
• Fixed input shapes wherever possible — dynamic dims cause axis-index errors and block constant folding. Exception: KV-cache seq axes stay dynamic by design.
• KV cache as plain tensors: past_key_values.{i}.{key,value} in, present.{i}.* out. No If, no merged decoder. The C# loop owns cache passing (sentis-inference).
• No control flow (If/Loop/Scan/Sequence* are unsupported): see §4 for the structural alternatives; for accidental Ifs from dynamic shapes run onnxsim.simplify(..., overwrite_input_shapes={...}) to constant-fold them away.
• No quantized ONNX ops; fp32 or f16 only. f16 if the inline file would exceed 2 GB.

4. Structural transforms — when the model's SHAPE must change, not just its ops

a. Decoder-only SLM/LLM → KV-cache I/O

Preferred: re-export from source with a wrapper exposing past/present (§2a; verified incantation: reference/verified_exports.md gemma-3). If ONLY an ONNX exists: surgery-inject Concat(past_k, k, axis=2) / Concat(past_v, v, axis=2) on each layer's K/V paths and expose them as I/O (reference/graphsurgeon.md recipe — pattern is sound but NOT yet verified in this project; budget a full parity-ladder run).

b. Encoder-decoder seq2seq (STT class) → UNMERGED graphs

no_post_process=True, monolith=False (§2b) → separate first-step and loop-step decoders. Never ship decoder_model_merged (contains If). First-step vs loop-step selection happens in C#.

c. Stateful RNN/LSTM streaming (VAD class) → hoist state to tensor I/O

Rewrite the torch wrapper so h/c (or any recurrent state) are explicit inputs AND outputs; C# carries them across frames. If the graph is inherently iterative (SSM/linear-attention Loop/Scan recurrence) there is NO Sentis path — reject at scout stage. 🔴 DON'T export the inner submodule — export the sub-path the FULL model actually runs per step, INCLUDING its context/overlap window. VERIFIED trap (Silero-VAD v5): the full model m(x,sr) feeds its net 576 = 64-sample context (last 64 of the previous hop) ++ 512 new, not the bare 512. Exporting/driving the inner _model at 512 imported clean AND passed single-frame ONNX↔torch parity (Δ<1e-8) — yet every speech clip read ~0 (looked like a dead mic). Because a one-frame fixture compared the SAME wrong 512 call on both sides, it hid the missing context. The full model gave ~1. Fix: re-export at [1,576]; C# maintains the 64-sample context ring alongside the LSTM state. Lesson (→ §7): for stateful/streaming models the acceptance test MUST run a real input SEQUENCE (actual speech clip) end-to-end vs the FULL reference model — single-frame tensor parity is NOT proof.

d. Pre-processing → its own small graph

HF pipelines do feature extraction in Python (log-mel, resize, normalize) that the export doesn't contain. Build it as a separate ONNX / Sentis-Functional graph and verify numerically against the HF preprocessor class (parity ≤ 1e-3). Watch for Ifs sneaking in via reflect-Pad / torch.stft export — fold with onnxsim at fixed shape. Whisper log-mel: two verified paths + the STFT onesided=true bug → reference/verified_exports.md.

e. Post-processing → OUT of the graph

Export the bare network (YOLO: nms=False); NMS/decoding live in C# (Functional.NMS, manual greedy decode). Baked post-processing hits unsupported ops or batching bugs.

f. Promptable segmentation (SAM / SAM2 / EfficientSAM class) → TWO graphs

Split exactly as the SAM ONNX pattern; do NOT export the whole predictor.

• encoder.onnx: image[1,3,S,S] → image_embed + the high-res feature maps. Run once per image; the C# side caches the outputs.
• decoder.onnx: (image_embed, high_res_feats, point_coords[1,N,2], point_labels[1,N]) → low_res_masks[1,3,m,m] + iou[1,3]. Bake multimask_output=True / repeat_image=False (kills the bool-driven If); the positional encoding get_dense_pe() is input-free so it bakes to a constant. Feed coords in the model's input frame (x/W*S, y/H*S); labels as int32.
• Pre-processing (resize→S + normalize) and post (upscale mask logits + threshold>0) live in C# (§4d/§4e). Verified end-to-end (EfficientSAM3): triple-parity ≤1.4e-4; warm cost concentrates in the encoder (paid ONCE per image) while the per-click decoder is comparatively cheap — exactly why the two-graph split pays (profile + backend verdicts: sentis-profiling measurements §10). The SAM2/SAM3 memory-bank video tracking is a Loop/Scan dead end — export only the single-image prompt path.
• 🔴 SAM3 has a SECOND prompt path — TEXT/concept grounding — that is ALSO Sentis-viable (VERIFIED, EfficientSAM3 EV-M), exported as 2 more graphs:
  • text_encoder.onnx: input_ids[1,ctx] → language_features[ctx,1,256]. MobileCLIP mct = conv token-mixers over the sequence ⇒ class-G ZERO-pad is parity-critical (§8 G); tokenize host-side.
  • grounding.onnx: (image, language_features, language_mask) → DETR outputs pred_logits[1,Q,1], pred_boxes[1,Q,4] (cxcywh), pred_masks[1,Q,m,m], presence[1,1]. Fusion encoder + DETR decoder (box-RPB cross-attn, NOT deformable/grid_sample — so it imports) + seg head; sigmoid/presence-gate/ threshold/box-convert live in C#. The vision backbone runs INSIDE this graph (a point+text app pays it twice — split vision+head to cache if that matters).
  • 🔴 Export trap — empty geometric prompt kills the export (BOTH ways). Text-only uses a dummy Prompt with 0 boxes/0 points; the geometry encoder's 0-length box/point encode + concat_padded_sequences (index_put) makes TorchScript bake a [0,1,C]+[N,1,C] non-broadcast Add → ORT won't load, and returning a clean empty [0,bs,C] instead hits the index_put→Reshape exporter assert (shape_has_zero && minus_one_pos==-1). dynamo=True separately dies on a data-dependent guard in the DETR RPB-cache branch. FIX = export-only monkeypatch of geometry_encoder.forward: for the empty prompt, skip all box/point encode+concat and build the output directly from the learned cls_embed → proj/norm → the encoder cross-attn layers (numerically identical — concat of 0-length is a no-op — but emits none of the hostile nodes). Don't try null-prompt (errors) / 1-masked-box (≠ 0-box) / baking the geo token (image-dependent).
  • Checkpoint pick matters: the distilled encoders alone (stage-1 ckpts) give presence≈0 → zero detections — you need the jointly-tuned FULL ckpt, and the README's advertised text-encoder variant can be WRONG (EV-M actual = MobileCLIP-S0/ctx16, not the table's "S1"). Verify by loading + a real prompt.

5. Hard model-level constraints (check BEFORE ops)

• 🔴 INLINE the weights. Sentis 2.6.1 imports a sidecar-.onnx.data model with 0 errors but reads the weights corruptly → total garbage while ORT is bit-exact on the same files. Fix: scripts/inline_onnx.py; delete the .onnx.data + stale .meta. This bug masquerades as a core-op divergence — rule it out FIRST, it is a one-command fix. (⚠ Evidence: well-isolated on 2.6.1 — same-graph sidecar-vs-inline A/B, plus a forced ImportAsset(ForceUpdate) control that ruled out a stale-import artifact. Version-pinned: re-verify on package bump.)
• 2 GB single-file protobuf cap — inline only works ≤ 2 GB; f16 first if needed (verified: over-cap fp32 fail / f16 OK).
• No tensor > 8 dims; no sparse/string/complex tensors. bool converts to float/int on import (memory grows — prefer explicit numeric mask inputs).

6. Graph surgery — practice

1. Smallest possible diff. Decompose into primitives Sentis computes identically to ORT; don't rebuild whole subgraphs when one node is at fault.
2. New decomposition = copy the surgery_gelu.py pattern: scalar constants as initializers, splice the primitive chain in place of the node, never touch external data, save, re-scan. (Why Gelu at all: Sentis maps ONNX Gelu to its exact-erf kernel and silently ignores approximate='tanh' — supported-by-NAME ops can still be wrong. §7 catches this class.)
3. Topology edits (remove/reroute/inject/extract): scripts/gs_surgery.py subcommands (info / extract / fix-shape / rename-io) or hand-written graphsurgeon following reference/graphsurgeon.md.
4. After ANY edit, in order: scan_onnx.py (re-scan) → ort_parity.py (numerics) → Netron eyeball for the edited region. An edit that "should be equivalent" is a hypothesis until ORT says so.

7. Triple-parity gate — original vs ORT vs Sentis (model-agnostic, verified)

🔴 The ground truth is the ORIGINAL framework output (torch / HF), NOT ORT. ORT faithfully runs whatever graph you exported, and Sentis faithfully runs whatever it imported — so neither can tell you the export itself is faithful to the real model. The acceptance artifact is a three-way comparison: run the SAME fixed input through all three, from ONE serialized fixture, and report BOTH edges. "Sentis matches ORT" alone is not done — if the export is wrong, both match each other and both are wrong.

The triple-parity table (produce this, don't paraphrase it):

Edge	Compare	Tol (fp32)	Isolates	How
A export fidelity	original (torch/HF) vs ORT	~1e-3	bad baked pre/post, dropped-op behavior, export-time approximation, wrong default	ort_parity.py model.onnx --load-fixture f.npz (fixture out:: = ORIGINAL output)
B import fidelity	ORT vs Sentis	~1e-3 (bit-exact for pure surgery)	Sentis-only divergence: Gelu-tanh dropped attr, external-data sidecar, mis-mapped importer op	Unity Worker.Schedule on the SAME bytes → diff vs ORT output
A∘B deployment truth	original vs Sentis	A+B combined	what actually ships	transitive; must hold end-to-end

The two edges see DIFFERENT failure classes and neither can see the other's — that is why you report both, always, even when one is obviously green. (Verified: V-JEPA 2 ViT-L edge A = 2.8e-3.)

One fixture feeds all three. Capture it IN THE EXPORT SCRIPT (while the original model is still in memory): save {input, original_output} as an .npz with keys in::<name> / out::<name> (the out:: values are the ORIGINAL forward, NOT ORT's), PLUS the raw f32 bytes of each for Unity. Then Edge A uses --load-fixture, and Unity feeds the identical bytes for Edge B — tokenizer-independent, so a mismatch is never a tokenizer artifact.

If an edge FAILs → localize (bisection, cheapest suspect first):

1. Which edge? A-fail ⇒ fix the export before opening Unity. B-fail ⇒ Sentis import bug.
2. Localize: re-export with intermediate outputs (embeddings, after-layer-0, mid layer); compare stage-by-stage on the failing edge; first divergence = culprit region.
3. Isolate path vs stack: export a variant taking the intermediate directly (inputs_embeds), feed the reference values — divergence follows the culprit.
4. Rule out weights: run the base model — same failure ⇒ structural, not weights. Order B-fail suspects cheapest-first: external-data (§5, one command) → dropped attribute (§6.2) → mask/bool handling → int64→int32 (benign, not the bug).

8. Failure taxonomy (triage BEFORE assuming a new failure mode)

Class	Symptom	Fix	Verified case
A fused/contrib ops	import errors on com.microsoft::*	eager re-export (§2a) / optimum (§2b) / custom converter cases (reference/contrib_op_converter_cases.cs)	gemma-3
B control flow	If/Loop/Scan/Sequence* in graph	unmerged export (§4b) / state hoisting (§4c) / onnxsim fold for accidental If	whisper
C dropped attribute	0-error import, subtly-then-catastrophically wrong	decompose the node (§6.2)	Gelu-tanh
C′ BENIGN dropped attribute	import warns it ignored an attr, but output is correct	none if the ignored value == Sentis's default for the node's actual use — confirm via Edge B, don't preemptively decompose	LayerNormalization axis→-1, Resize cubic_coeff_a→-0.75
D external-data sidecar	0-error import, garbage from step one, ORT fine	inline_onnx.py (§5)	functiongemma
E baked post-processing	export flag baked NMS/decode → class A/B ops or batch bugs	export bare, C# post (§4e)	YOLO nms=True
F missing pre-processing	Unity feeds raw input → garbage	separate preprocessing graph (§4d)	whisper log-mel
G wrong input encoding / padding	0-error import, per-tensor parity OK, but real inputs give wrong/flat output	match the reference's EXACT input convention (padding VALUE, special tokens, channel order)	MobileCLIP text ZERO-pad

🔴 Class G — padding value is parity-critical for conv/patch token-mixers. A pure-attention transformer is padding-agnostic (pooled from the eos position, causal mask), so wrong padding hides. But a model with 1-D conv (or patch) token-mixers over the sequence reads the padding tokens — so the pad VALUE changes the output. VERIFIED (MobileCLIP2-S0 MCt text tower, 2.6.1): eos-padding (transformers CLIPTokenizer default) vs zero-padding (open_clip) gave completely different text embeddings → zero-shot ranking scrambled. Scan + single-tensor import + even single-fixture ORT↔Sentis parity all PASS; only a functional end-to-end test (real image vs labels) caught it — the §7 "run a real sequence, not one fixture" rule generalizes from stateful models to any sequence-conv model. Fix lives in the C# runtime (zero-init the id tensor, copy only real ids), not the graph — so state the padding contract in the handoff to sentis-inference.

⚠ Version pins: classes C and D were verified on Sentis 2.6.1 — re-verify on every package bump (C is a plausible upstream fix — 2.6.1 already routes Gelu-tanh→GeluFast, just not equal to pytorch_tanh; D's mechanism is isolated via same-graph inline A/B + forced-reimport control, §5).

Full per-model dossiers (incl. fine-tune context): sentis-training-pipeline/reference/models/. Conversion-side distillation: reference/verified_exports.md.

Bundled scripts (run them, don't reimplement)

• scripts/scan_onnx.py <model.onnx|HF-repo|--dir> — Sentis 2.6 compatibility scan, no weights (TWIN copy shared with sentis-model-scout — update both).
• scripts/inline_onnx.py <in.onnx> [out.onnx] — inline external .onnx.data (class D).
• scripts/surgery_gelu.py <model.onnx> — Gelu(tanh) → primitives (class C); the decomposition pattern to copy for new attribute-dropping ops.
• scripts/gs_surgery.py {info,extract,fix-shape,rename-io} — graphsurgeon topology toolkit.
• scripts/ort_parity.py — model-vs-model / fixture-based numerical parity (the §7 gate).

Reference files

• reference/sentis_supported_ops.md — official 2.6.1 op lists + caveats (snapshot; re-fetch on package upgrade; TWIN copy shared with sentis-model-scout).
• reference/verified_exports.md — per-model verified export incantations.
• reference/graphsurgeon.md — gs API notes + surgery recipes (incl. KV-injection sketch).
• reference/contrib_op_converter_cases.cs — teach Sentis's importer the 4 common contrib ops (embedded-package path; ORT parity still mandatory; TWIN copy shared with sentis-model-scout).

9. Standard order & self-check

SLM standard order: export (eager) + capture original-weights fixture → inline → surgery → re-scan → Edge A (torch/HF==ORT, §7) → Sentis import → Edge B (ORT==Sentis on the same bytes). Both edges of the triple-parity table (§7) are the acceptance artifact — skipping a step is how weeks get lost to a one-command bug.

• Tool chosen from §1 by situation, not habit; export flags copied from reference/verified_exports.md when a verified incantation exists.
• opset ≤ 25 explicit; fixed shapes (KV seq axes excepted); eager attention; explicit masks.
• No control flow relied on; unmerged decoder for seq2seq; state hoisted for RNN.
• Weights INLINE; file ≤ 2 GB (f16 if needed); no quantized ops; ≤ 8 dims.
• Triple-parity table (§7) reported with BOTH edges: Edge A original(torch/HF)==ORT confirmed BEFORE Unity, Edge B ORT==Sentis on the SAME raw bytes; fixture anchored on the ORIGINAL output (keys out:: = torch/HF forward, not ORT) and captured in the export script; every graph edit re-scanned + both edges re-checked.
• Pre-processing graph parity-checked vs the HF preprocessor; post-processing in C#.
• Handoffs stated: checkpoint in from sentis-training-pipeline; fixture + tensor-name contract out to sentis-inference.