sentis-model-scout

name

sentis-model-scout

description

Find and judge Hugging Face models for Unity Sentis (com.unity.ai.inference) 2.6 on-device use, BEFORE downloading weights. Use whenever the task is: searching HF for a model ("find a model for X that runs in Unity/Sentis"), deciding whether a candidate model will import into Sentis, comparing model candidates, checking a model's size/license/language fit for on-device, or scanning an ONNX graph for unsupported operators, or judging whether a FAILED export is repairable (torch eager re-export / optimum-cli / Olive) or a structural dead end. Runs a bundled scan script instead of guessing. Hands off to sentis-model-converter (conversion/fixes) once a candidate is chosen.

Sentis Model Scout — find it, judge it, THEN download it

A failed import wastes hours; these checks take minutes on the model card + a graph scan. Never "try it and see". Never trust the model card alone. Verdict order: cheap checks first (file listing, config.json, license) → graph scan (script) → only then download weights.

1. Search strategy (Hugging Face)

• Start from the task tag (automatic-speech-recognition, object-detection, voice-activity-detection, text-generation…) + sort by downloads/trending; prefer official orgs (openai, google, Ultralytics, Supertone…) over anonymous re-uploads.
• Check for existing unity-sentis-tagged repos first — someone may have already done the conversion AND shipped a self-contained .cs.
• ⚠ Prebuilt ONNX mirrors (onnx-community/*, transformers.js exports) are usually ORT-optimized = fused com.microsoft ops = Sentis-impossible as-is. Prefer the original PyTorch repo + self-export (the sentis-model-converter skill owns that path). A prebuilt ONNX is a candidate to scan, not an answer.

2. Judge WITHOUT downloading weights

1. File listing: total size per file (2 GB inline cap below), presence of a separate *.onnx.data (external data — Sentis mis-reads it; must inline), tokenizer/config files. ⚠ No ONNX at all (a research repo shipping only .pt/.pth, "ONNX export coming soon") → there is nothing to scan; you must self-export from the paper's GitHub, so judge Q4 by architecture (§4) and hand the source to sentis-model-converter. Also: an advertised param count is often just ONE sub-module ("0.68M encoder") while the full checkpoint bundles text encoder + memory + heads (measured: a "0.68M" EfficientSAM3 ckpt was multi-GB) — size Q1 against the deployable file, and note you only export the sub-path you need. Never git clone a multi-GB LFS weights repo; fetch the single smallest checkpoint via the HF API / resolve/main/<file> (GIT_LFS_SKIP_SMUDGE=1 for the code repo).
2. config.json → architecture: look it up in the verdict table (§4). The architecture often decides the path before any scan.
3. License — read BOTH code and weights: they can differ in one repo (permissive Apache-2.0 code but non-commercial CC BY-NC-SA pre-trained weights, for example). Prefer Apache-2.0/MIT. Known traps: strong-copyleft licenses (e.g. AGPL-3.0) make derivative weights inherit the license; some model families ship their own custom weight terms — check per repo.
4. Graph scan: run the bundled script on the .onnx (it loads the graph WITHOUT external weights, so even multi-GB models scan in seconds once the small .onnx is fetched):

   python scripts/scan_onnx.py <model.onnx | HF repo id>
   

   It reports: op domains (flags com.microsoft), unsupported ops (If/Loop/Scan/ Sequence*), opset range (7–25), external-data initializers, >8-dim/sparse/string tensors, and per-file sizes vs the 2 GB inline cap.
5. Authoritative lists — BUNDLED: reference/sentis_supported_ops.md holds the official 2.6.1 supported/unsupported operator lists (+ caveat table: backend-only ops, unsupported attributes like Resize/Pool params, LSTM-yes-but-GRU/RNN-no, quantized-op family all-out) and model-level constraints. The scan script has the same lists embedded → three-state verdict: supported / officially-unsupported / not-in-table (may still map via the importer, e.g. Gelu — verify by import + parity). Snapshot of the official 2.6 manual pages (supported-models.html, supported-operators.html) — re-fetch on package upgrade.
6. com.microsoft contrib ops are not always a dead end — the custom-converter path. If the scan flags {SimplifiedLayerNormalization, SkipSimplifiedLayerNormalization, RotaryEmbedding, GroupQueryAttention}, there are TWO fixes:
   • (a) default: eager re-export from the PyTorch original (sentis-model-converter §2a)
   • (b) advanced — APPLY THE BUNDLED CASES: embed com.unity.ai.inference as a local package and paste the verified case blocks from reference/contrib_op_converter_cases.cs into the operator switch of Editor/ONNX/ONNXModelConverter.cs (exact paste location, in-scope helpers, and the package-embedding step are in that file's header). The prebuilt ONNX then imports AS-IS. Mappings: SimplifiedLayerNorm→RMSNormalization · SkipSimplified→Add(+bias)+RMSNorm(4 outputs) · RotaryEmbedding→primitive decomposition (⚠ interleaved=1 unsupported) · GQA→full attention build (KV concat, head tiling, causal mask; outputs attn + present KV). Choose (b) when the PyTorch original is unavailable/painful to re-export (VLM-class exports), or many candidates share the same ORT-optimized style. Costs: you maintain a package fork pinned to a Sentis version, and ORT parity is still mandatory — an importer mapping can be subtly wrong exactly like a stock one. (Origin: verified in Sky-Kim/fastvlm-0.5b-unity.)

3. The four questions (a FAIL on any = candidate is out, or needs the listed fix)

#	Question	Hard facts	Fix if failed
1	Size fits on-device budget?	single inline .onnx ≤ 2 GB (protobuf cap; verified: over-cap Qwen fp32 failed, its f16 fit; sub-cap fp32 runs in-editor). Mobile/Web budget = per-project human decision — quantize + validate on target device	quantize upstream (f16 halves), or smaller model
2	License allows ship + redistribution?	prefer Apache-2.0/MIT; check code AND weights separately; derived weights inherit base terms	legal review, or different base
3	Target language actually works?	"multilingual" on the card ≠ works — measured Korean zero-shot FAILED most of the project command set on two separate models; both needed fine-tuning	narrow fine-tune (training-pipeline §1)
4	Sentis-importable graph?	scan verdicts from §2.4	fixability triage §4½ → eager re-export / unmerged export / inline / surgery (sentis-model-converter §2–§6)

"imports with 0 errors" ≠ "works" — attribute-level traps (Gelu-tanh) pass any op scan. The only real gate is ORT parity, owned by sentis-model-converter (§7).

4. Architecture verdict table (patterns verified in this project — judge by FAMILY, not name)

Architecture pattern (from config.json / graph shape)	Verdict	Path
Decoder-only transformer SLM/LLM (KV cache as I/O)	prebuilt ORT-optimized ONNX ❌ as-is / self-export ✅	eager re-export → inline → surgery → ORT parity, or custom package converter (§2.6)
Encoder-decoder seq2seq with past (STT class)	✅	unmerged export (avoids If), multiple graphs + manual KV in C#
Encoder-only embedding (BERT family)	✅	legacy exporter (dynamo=False), bake pooling/normalize into the wrapper
Two-tower CLIP (image + text encoders)	✅	export/scan each tower separately (vision_model.onnx + text_model.onnx); L2-normalize + cosine in C#. Prebuilt clean ONNX exists for some (MobileCLIP: plhery/mobileclip2-onnx). ⚠ conv-based text towers (MobileCLIP MCt) need the reference's exact padding — converter §8 class G. ⚠ license: Apple MobileCLIP weights = research-only
CNN detector (YOLO class)	✅	fixed shape, opset ≤ 25; check the NMS strategy (NMS-free output vs bake NMS)
Promptable segmentation (SAM / SAM2 / SAM3 / EfficientSAM / MobileSAM class)	✅ image path (point/box AND text/concept) / ❌ video-memory path	multi-graph split: image-encoder + prompt/mask-decoder (point/box) ✅; SAM3 text/concept adds a text-encoder + fusion+DETR grounding graph — ALSO viable (box-RPB DETR, not deformable), VERIFIED EfficientSAM3 (converter §4f, incl. the empty-prompt export fix). The SAM2/SAM3 memory-bank video tracking is Loop/Scan-class recurrence = no Sentis path (reject that path only). ⚠ pick the jointly-tuned FULL checkpoint (distilled-encoder-only ckpts → presence≈0, zero detections) and verify the text-encoder variant by loading it — README arch tables can be wrong.
Stateful RNN/LSTM streaming (VAD class)	✅	carry recurrent state tensors as I/O across frames — AND the per-step context/overlap window (Silero-VAD = 64-sample context + 512 hop = 576 input, not 512; converter verified_exports.md)
Multi-stage audio stack (mel → embedding → classifier head)	✅	compose stages in C# with ring buffers
Any fp32 model > 2 GB	⚠	quantize upstream to f16 FIRST (inline cap), then re-scan
transformers.js / ORT-optimized re-uploads	❌ as-is	find the original PyTorch repo, self-export
Linear-attention / hybrid SSM-style decoders	❌	Loop/Scan-class graph — no current Sentis path

4½. Fixability triage — a FAIL condemns THIS export, not necessarily the model

Most scan FAILs are artifacts of how the ONNX was exported, not of the architecture. Before rejecting a candidate, classify the failure below and report one of FOUR verdicts: works as-is / fixable (name the tool) / fixable only with source access / not fixable. The actual conversion is owned by sentis-model-converter; this table decides whether that handoff is worth making — and with which tool.

Scan failure	Repair path (tool)	Fixable?
com.microsoft fused contrib ops (ORT-optimized export)	optimum-cli export onnx --model <repo> --task <task> — exports UNFUSED by default — or eager torch re-export from the PyTorch original; or the custom-converter package (§2.6)	✅ virtually always — the architecture itself is Sentis-viable (verified: SmolLM2/Qwen-class)
Merged decoder If (seq2seq / decoder-with-past)	optimum-cli produces unmerged graphs (decoder_model.onnx + decoder_with_past_model.onnx); wire KV manually in C#	✅ (verified: Moonshine, Whisper-tiny)
If/Loop baked into the model's own logic (not a merge artifact)	rewrite the torch wrapper to hoist state/branches to plain tensor I/O, then re-export (the Silero-VAD stateful-LSTM pattern)	⚠ needs source + wrapper work; if the graph is inherently iterative → ❌
> 2 GB file / external .onnx.data	quantize upstream to f16 — Olive (float16/size passes) or torch .half() before export — then inline	✅ if f16 lands ≤ 2 GB; otherwise ❌ at this size (pick a smaller model)
Quantized ONNX ops (QLinear*, MatMulInteger…)	cannot import, ever; re-export fp32/f16 from torch, then use Sentis runtime quantization (Float16/Uint8) instead	✅ via re-export only — never directly
Opset outside 7–25	re-export pinning --opset (torch and optimum-cli both support)	✅ trivial
Structural Loop/Scan recurrence (SSM/linear-attention class)	none — no export tool removes structural recurrence	❌ reject

Tool selection (mechanics live in sentis-model-converter):

• torch (torch.onnx.export, eager) — maximum control; required whenever a custom wrapper must change model I/O (state hoisting, KV as tensors, baked pre/post-processing). BERT-class models want the legacy exporter (dynamo=False).
• optimum-cli — first choice for HF transformers models: task-aware, unfused graphs and unmerged decoders out of the box, --opset / dtype flags.
• Olive — size/dtype passes (float16 conversion) when you can't easily re-export from torch. ⚠ Its ORT transformer-optimization (fusion) passes ADD com.microsoft ops — the exact thing Sentis rejects. Use size/dtype passes only; never the fusion passes. INT4/INT8 passes are also out (quantized-op family unsupported) — quantize at Sentis runtime instead.

Whatever the tool: the repaired export re-enters the pipeline at §2.4 (re-scan) and the final gate is still ORT parity — a fixed graph can be subtly wrong exactly like a stock one.

5. Report format (what the agent should output)

Per candidate: Q1 size / Q2 license / Q3 language / Q4 graph verdicts + one-line evidence each

• fixability verdict (§4½: as-is / fixable-with- / fixable-only-with-source / not-fixable)
• next step (download & convert via sentis-model-converter (then fine-tune via sentis-training-pipeline if needed), or reject with reason). Multiple candidates → comparison table, recommendation, and what measurement would decide ties.

Bundled scripts & reference files

• scripts/scan_onnx.py — the §2.4 compatibility scan (TWIN copy shared with sentis-model-converter — update both).
• reference/sentis_supported_ops.md — official 2.6.1 supported/unsupported op lists snapshot (§2.5; TWIN copy shared with sentis-model-converter).
• reference/contrib_op_converter_cases.cs — the §2.6(b) custom-converter case blocks (TWIN copy shared with sentis-model-converter).

Self-check before recommending a model

• Ran the scan script (or justified why not) — didn't judge ops from memory.
• Checked license on BOTH code and weights; noted derivative-weight inheritance.
• Quoted size vs the 2 GB inline cap and the platform budget as separate concerns.
• Language claim backed by a measurement plan, not the model card.
• Every FAIL carries a fixability verdict (§4½) naming the repair tool — or why none applies.
• Handoff stated: which sentis-model-converter section applies next (fine-tuning: sentis-training-pipeline).