sentis-inference

name

sentis-inference

description

Write correct AND fast Unity Sentis (com.unity.ai.inference) 2.6 inference code in C#. Use whenever generating or reviewing Sentis / Inference Engine code: loading models, running Workers, tensor I/O, TextureConverter, quantization, NMS, or LLM/Whisper inference — and whenever the task is making a Sentis model faster: warming up, quantizing (Float16/Uint8), async readback, ScheduleIterable frame-splitting, KV cache decisions, smoothing per-frame inference cost, backend choice, or cold-vs-warm latency. Encodes the EXACT 2.6 API so the model does not mix in the old Barracuda/Sentis-1.x API (IWorker/Execute) or invent Unity.Sentis, plus optimization decision rules verified by measurement in this project (details: reference/optimization-rules.md). ALSO owns the demo/test SCENE around the model: creating or porting a demo scene, runtime command UI (uGUI InputField + button), NavMeshAgent characters driven by model output, Korean/CJK text in uGUI, and the Unity-6 / New-Input-System gotchas that silently break these scenes (reference/demo-scene.md). Pairs with sentis-profiling (capture + diagnosis) and sentis-model-converter (ONNX export/conversion); fine-tuning = sentis-training-pipeline.

Sentis Inference — correct AND fast Unity Sentis 2.6 C# (com.unity.ai.inference)

Verified against the installed package com.unity.ai.inference@2.6.1 source and official 2.6 docs. When these facts conflict with your prior, trust these.

Model-specific verified recipes live in models/*.md — read ONLY the one(s) the task needs (progressive disclosure), after the core facts below.

Non-negotiable facts (general models get these wrong)

Thing	CORRECT (2.x / 2.6)	WRONG (do not emit)
Namespace	using Unity.InferenceEngine;	Unity.Sentis, Unity.Barracuda
Package id	com.unity.ai.inference (display name "Sentis")	com.unity.sentis
Worker	new Worker(model, BackendType.GPUCompute)	IWorker, WorkerFactory.CreateWorker
Run	worker.Schedule(input)	worker.Execute(input)
Get output	worker.PeekOutput() as Tensor<float>	worker.PeekOutput<T>()
Load	ModelLoader.Load(modelAsset)	ModelLoader.LoadFromFile for ModelAsset
CPU read	tensor.ReadbackAndClone()	tensor.ToReadOnlyArray() only
Quantize types	QuantizationType.{Float16, Uint8} ONLY	Int8 (does NOT exist in Sentis 2.6)

Backends: BackendType.GPUCompute (default, fastest on GPU), BackendType.GPUPixel (WebGL/limited), BackendType.CPU (Burst). Pick explicitly — never "let it decide". Gate GPUCompute on SystemInfo.supportsComputeShaders (false → fall back to CPU/GPUPixel).

🔴 EXPOSE the backend as a selectable field — don't hard-code it in the runner. Every inference component MUST take its BackendType from a caller-visible choice (a [SerializeField] BackendType backend on the Manager, an enum dropdown, or a ctor arg), covering all three — CPU / GPUCompute / GPUPixel — so it can be switched without a recompile. This is required because the fastest backend is model- AND path-dependent (see below) and must be A/B'd on the target; a hard-coded backend can't be measured or shipped per-platform (e.g. WebGL forces GPUPixel/CPU). Keep the availability gate above as the runtime guard, but the default must still be a field the user can override. Pattern (verified, FastVLM-0.5B): [SerializeField] BackendType backend = BackendType.GPUCompute; → var be = (backend == BackendType.GPUCompute && !SystemInfo.supportsComputeShaders) ? BackendType.CPU : backend; → pass be into the runner. A raw bool useGpu is NOT enough — it hides GPUPixel; expose the full 3-way BackendType.

🔴 PER-GRAPH backends — a multi-graph pipeline can (and often should) run each graph on a DIFFERENT backend. The fastest backend is a property of the graph's shape, not the app, so one global backend is usually wrong for a pipeline that mixes graph types. Give the runner a backend PER Worker and pick each from that graph's measured profile (sentis-profiling §3):

• heavy, compute-bound, run-once-per-input graph → GPUCompute (big conv/attention: image encoders, ViT, seq prefill). GPU's parallel compute wins and the one-time cost amortizes.
• tiny / readback-dominated / per-step-in-a-loop graph → CPU (Burst). A seq=1 decode step, a small decoder, or any graph you read back every iteration: the per-step GPU dispatch + the GPU→CPU sync on its outputs (logits, KV) dwarf the actual compute, so CPU/Burst is faster AND avoids the round-trip. (Verified FastVLM-0.5B: vision → GPUCompute + decoder(prefill+decode) → CPU beat all-GPU decode decisively — measured evidence: sentis-profiling measurements §14. EfficientSAM3 the same: image-encoder/grounding → GPUCompute, mask-decoder → CPU.)
• Constraints when splitting: (1) two graphs on two backends that hand tensors across the boundary pay one readback/upload at the seam — fine if it's once (image-embed → decoder), bad if it's per-step, so keep a tight loop's graphs on ONE backend. (2) you cannot run ONE Worker on two backends; splitting sub-phases of the same model (e.g. prefill vs decode of one decoder) needs a 2nd Worker = a 2nd copy of its weights in memory — only worth it if the saving beats the extra RAM (FastVLM: a duplicate of the decoder weights was NOT worth the small prefill saving, so prefill stayed with decode on CPU). Expose one selectable BackendType field per Worker (e.g. visionBackend/decoderBackend), each gated on supportsComputeShaders. Benchmark both for small models — dispatch + readback overhead can make CPU win for very small graphs. ⚠ The winner is PATH-dependent, not just size-dependent: this project's 270M SLM won on CPU while its decode loop did per-step CPU readbacks (262k-vocab argmax + KV round-trips), but flipped to GPU once argmax + KV moved on-GPU (sentis-profiling reference/measurements.md §9b/§9c). Details: reference/optimization-rules.md §6.

🔴 CPU-fallback trap (manual: how-sentis-runs-a-model): ops unsupported on your backend silently fall back to CPU, forcing a GPU sync + tensor readback/re-upload around each such layer. A model with many fallback layers can be slower on GPUCompute than on plain CPU. If a "GPU" model is mysteriously slow, check its ops against supported-operators — or just use CPU.

The canonical 5-step workflow

using UnityEngine;
using Unity.InferenceEngine;          // 1. namespace

public class Inference : MonoBehaviour
{
    [SerializeField] ModelAsset modelAsset;
    Worker m_Worker;
    Tensor m_Input;

    void OnEnable()
    {
        var model = ModelLoader.Load(modelAsset);              // 2. load
        m_Worker  = new Worker(model, BackendType.GPUCompute); // 4. worker
        m_Input   = new Tensor<float>(new TensorShape(1, 3, 224, 224)); // 3. input
    }

    void Update()
    {
        m_Worker.Schedule(m_Input);                            // 5. schedule
        var output = m_Worker.PeekOutput() as Tensor<float>;  //    peek
        using var cpu = output.ReadbackAndClone();            //    read on CPU
        // ... read cpu[i]; do NOT read PeekOutput() directly off GPU
    }

    void OnDisable() { m_Worker.Dispose(); m_Input.Dispose(); } // ALWAYS dispose
}

Rules:

• Always Dispose() the Worker and every Tensor you new. Tensors are native resources; leaking them leaks GPU/native memory.
• PeekOutput() returns a non-owning view valid until the next Schedule — do not Dispose it; call ReadbackAndClone() to get an owned CPU copy.
• Multiple inputs → worker.Schedule(tensorArray), or worker.SetInput(name, tensor) per input then worker.Schedule(); multiple outputs → PeekOutput("output0").
• 🔴 Keep an output across Schedule calls → CopyOutput, not PeekOutput. worker.CopyOutput(name, ref myTensor) copies into a tensor YOU own/dispose (pass null → Sentis allocates; else dtype must match and capacity suffice — it reshapes). Holding a PeekOutput view past the next Schedule is a use-after-invalidate bug.
• For non-blocking readback use the async API (ReadbackAndCloneAsync / awaitable), not a blocking ReadbackAndClone() every frame — three forms: reference/optimization-rules.md §4.
• Render-pipeline integration: cb.ScheduleWorker(worker, input) — a CommandBuffer extension (also ScheduleWorkerIterable); build the CommandBuffer once and reuse it.

Make it fast — measure-first levers (full rules: reference/optimization-rules.md)

Verified in this project; profile before optimizing (capture + diagnosis workflow = sentis-profiling; measured evidence = its reference/measurements.md). Several "obvious" wins lost in practice — read the full rules before applying §3/§5/§8.

1. Warmup (always) — the first inference is substantially slower than warm (shader compile + alloc). Every inference component gets a Warmup() doing one dummy pass at load, hidden behind a loading screen. (§1)
2. Async readback (the usual real win) — ReadbackAndClone() blocks the CPU until ALL queued GPU work drains — a large main-thread cost even with a tiny output; async removes it. For any always-on per-frame model, go async regardless of output size. Three forms — await / request+poll / callback. (§4)
3. ScheduleIterable (smoothing) — spread one inference K layers/frame; drops the per-frame median dramatically. Smooths, does NOT speed up; completion frame still pays the readback → combine with async. (§4½) One-token-per-frame decode for SLMs. (§5¾)
4. Quantization = memory win, not a speed win — re-measured (Sentis 2.6.1/M1, fair A/B): Float16 & Uint8 ran E5 slightly SLOWER and YOLO neutral (§3); reliable payoff is ~2×/4× smaller, near-lossless. Measure per model AND package version; don't quantize for speed on desktop GPU. MUST Save→reload after QuantizeWeights (in-memory quantized model throws). (§3)
5. KV cache is NOT automatic — a naive CPU-round-trip cache measured SLOWER than no cache at short lengths; keep it on-GPU or skip it. Fixed prompt prefix → prefill once, reuse (bit-exact, tokenizer-seam trap). (§5–5½)
6. Backend & dims — small models may run faster on CPU (§6); dynamic input dims are not automatically a penalty — measure before re-exporting (§8). Whisper >30 s silently truncates → TranscribeStream (§7).

🔴 Inspect the model first — don't hard-code shapes for dynamic models

Many ONNX models have dynamic dims (batch_size, sequence_length, …). Read the real I/O contract off model.inputs / model.outputs instead of guessing:

var model = ModelLoader.Load(modelAsset);
foreach (var i in model.inputs)          // Model.Input { name, dataType, shape }
{
    DynamicTensorShape ds = i.shape;
    if (ds.IsStatic())
    {
        TensorShape fixedShape = ds.ToTensorShape();  // safe: fully known → allocate this
    }
    // else: dynamic dims — YOU pick concrete sizes (e.g. seq len) and document them
}

Rule: if !shape.IsStatic(), the code must choose and state the concrete dims it feeds (and pad/truncate accordingly) — never call ToTensorShape() on a dynamic shape (throws).

Texture / camera input (CV)

using var input = new Tensor<float>(new TensorShape(1, 3, H, W)); // NCHW default
TextureConverter.ToTensor(webcamTexture, input,
    new TextureTransform().SetTensorLayout(TensorLayout.NCHW));

TextureConverter.ToTensor(texture, tensor, transform) resamples (linear) to the tensor's W/H. Channel count comes from tensor dim. TextureTransform chains .SetTensorLayout(), .SetChannelSwizzle(), .SetCoordOrigin().

Audio / microphone input (STT, wake-word, VAD, audio classification)

🔴 Any model whose input is audio (waveform, mel/log-mel, MFCC, embeddings-from-audio) MUST be driven by REAL-TIME microphone capture — not only offline clips/fixtures. A file/fixture path is fine for parity tests, but the shipped inference component's primary input is the live mic. Building an audio model that only scores a pre-baked clip is INCOMPLETE (same bar as §MANDATORY DELIVERABLES).

Pattern (Unity Microphone, verified in this project's real-time "Hey Unity" wake-word):

• Capture: Microphone.Start(null, loop:true, lengthSec, sampleRate) → a looping AudioClip ring buffer. Request the model's rate (wake-word/STT = 16000); if the device forces another rate, resample. Guard Microphone.devices.Length == 0.
• Drain per frame in Update: int pos = Microphone.GetPosition(null); int n = pos - last; if (n < 0) n += clip.samples; then clip.GetData(buf, last) and advance last = pos. Feed the new samples into the model pipeline.
• Separate transport from DSP: expose a ProcessSamples(float[] pcm) the mic path calls — so the SAME pipeline is drivable by a .wav/fixture for parity tests (this is how you verify against the Python/ORT reference without a live voice).
• Stream, don't re-run whole clips: keep rolling buffers (raw → features → model window), advance by the model's hop each step; carry state across frames (see §"LLM / transformer decoders" for the stateful analogue). Port the reference pipeline's EXACT constants (sample-hop, feature-window, normalization) — they are parity-critical.
• 🔴 Prime / warm-up transient: seed buffers exactly like the reference (e.g. openWakeWord seeds a ones() mel buffer) AND suppress detections until the seed is flushed and the model's input window is fully real — otherwise the fill transient false-triggers (verified: a negative clip spiked far above threshold during warm-up and dropped far below once primed). Expose a Primed flag; gate OnWake on it.
• Debounce: after a detection, apply a cooldown (~1–1.5 s) before the next, so one utterance = one event.
• Mobile: Microphone needs the OS record permission (Android RECORD_AUDIO, iOS NSMicrophoneUsageDescription); request it before Start.
• STT (Whisper) streaming caveats (>30 s truncation, TranscribeStream) → models/whisper.md.

Quantization (decision rules)

Two valid types only: Float16 (≈2x smaller, near-lossless) and Uint8 (≈4x smaller, lossy — validate accuracy). No Int8 in Sentis. Upstream INT8/INT4 ONNX is NOT an alternative: the quantized-op family (QLinear*, MatMulInteger…) cannot import (sentis-model-converter §2c) — viable only if it decomposes to plain supported ops (reference/optimization-rules.md §3). Float16/Uint8 here is the ceiling.

var model = ModelLoader.Load(modelAsset);
ModelQuantizer.QuantizeWeights(QuantizationType.Uint8, ref model); // weights only
ModelWriter.Save(path, model);   // save a .sentis asset (Editor)

Decision rule: start Float16; move to Uint8 only if memory-bound, and always re-check task accuracy after. Quantization is a human decision, not "optimize it for me". 🔴 Save→reload is MANDATORY — running the in-memory quantized model directly throws KeyNotFound in Conv/layers. Speed is model-dependent (Float16 can SLOW conv-bound graphs): reference/optimization-rules.md §3.

Model editing — Functional API (NMS, custom heads)

Sentis edits models in C# via Functional / FunctionalTensor. Object-detection post-processing (NMS) lives in Functional.Vision.Detection. Use it to bake NMS into a YOLOv8/11 graph (output (1,84,8400), NMS required) so the Worker returns final boxes. There is no built-in YOLO, Whisper, or KVCache helper — you assemble these.

VERIFIED 2.6 graph-edit recipe (prepend pre-proc / append post-proc / add-remove I/O), smoke-tested in this project on YOLOv10n — appending a *2f op went 236 → 237 layers, output (1,300,6) preserved:

var model  = ModelLoader.Load(modelAsset);
var graph  = new FunctionalGraph();
var inputs = graph.AddInputs(model);                 // FunctionalTensor[] (model's inputs)
var outs   = Functional.Forward(model, inputs);      // run the original graph
var edited = Functional.Softmax(outs[0]);            // any FunctionalTensor op; operators (+ - * /) work too
graph.AddOutput(edited, "out");                      // register modified output
var newModel = graph.Compile();                      // SLOW + high-memory → do OFFLINE/editor, then Save .sentis

Compile() is expensive — bake once in the editor and ModelWriter.Save the result; don't compile at runtime.

Model-specific VERIFIED recipes (read only what the task needs)

Model	File	One-line summary
YOLOv10n (CV detect)	models/yolo.md	NMS-free, output (1,300,6) pixel-space; v8/11 need NMS baking
RF-DETR nano (CV detect)	models/rf-detr.md	NMS-free DETR, dets[1,300,4]cxcywh + logits; GridSample/TopK (no GPUPixel); judge parity on decoded dets
multilingual-e5-small (embedding)	models/e5.md	legacy exporter, baked mean-pool+L2, (1,384) normalized
MobileCLIP2-S0 (CLIP embedding)	models/mobileclip2.md	two-tower, /255-only preproc, [B,512]; text ZERO-pad (conv mixer); weights research-only
Whisper-tiny (STT)	models/whisper.md	4 ONNX, unmerged decoder, manual encoder/decoder KV, forced prefix
openWakeWord ("Hey Unity")	models/openwakeword.md	3 stacked graphs (mel→embed→clf [1,16,96]→[1,1]), 16 kHz stream, primed seed, CPU
Silero-VAD (VAD)	models/silerovad.md	[1,576]=64-ctx++512, LSTM state[2,1,128] carry, CPU backend (GPU slower), 16 kHz resample
Supertonic-3 (TTS)	models/supertonic.md	4 models, flow-matching loop, NFKD jamo indexer, 44.1 kHz
Piper (VITS TTS)	models/piper.md	input[1,128]ids+input_lengths+scales[3], offline G2P, baked noise, 22.05 kHz, CPU
Gemma-3-270m (SLM)	models/gemma3.md	eager re-export + Gelu surgery + INLINE weights; manual KV
Qwen3-0.6B (chat SLM)	models/qwen3.md	Apache-2.0, position_ids input, uniform-f16 export, host ChatML, CPU (GPU crashed)
FunctionGemma-270m-it (tool-calling)	models/functiongemma.md	fc prompt/parser, stop=50, KV prefix cache, non-blocking decode, OOS gate
FastVLM-0.5B (VLM caption)	models/fastvlm.md	3 graphs (vision 256 tok + Qwen2 decoder), per-graph backend split; custom-converter contrib ops; research-only
EfficientSAM3 EV-M (promptable seg)	models/efficientsam3.md	4 graphs, point/box (SAM decoder) + text/concept (DETR grounding); per-graph backend split; CLIP tokenizer; 1008² preprocess baked 2x-1

Tokenization — use the BUILT-IN Unity.InferenceEngine.Tokenization (don't hand-roll / don't add deps)

Sentis 2.6 ships a full HuggingFace-compatible tokenizer framework. VERIFIED: parses an HF tokenizer.json and produces token ids IDENTICAL to Python (tested XLM-R/Unigram for E5; CLIP byte-level BPE + RobertaProcessing — openai/clip-vit-base-patch32/tokenizer.json — reproduced the EfficientSAM3 repo's SimpleTokenizer ids exactly; also covers BPE/WordPiece/WordLevel, Metaspace/ByteLevel/Bert pre-tokenizers, Precompiled(SentencePiece) normalizer, TemplateProcessing post-processor). ⚠ padding to the model's fixed ctx is YOUR job and can be parity-critical (CLIP/mct conv text towers need ZERO-pad — sentis-model-converter §8 class G). ⚠ Coverage is broad but NOT total — the manual (tokenizer.md) lists HF components not yet implemented; if Parse throws on an exotic tokenizer.json, check that list before hand-rolling. This also covers Whisper's tokenizer.

using Unity.InferenceEngine.Tokenization;
using Unity.InferenceEngine.Tokenization.Parsers.HuggingFace;

// tokenizerJson = a TextAsset of the model's tokenizer.json (drop the .json in Assets)
ITokenizer tok = HuggingFaceParser.GetDefault().Parse(tokenizerJson.text);
IEncoding enc = tok.Encode("query: 따라와");   // (inputA, inputB=null, addSpecialTokens=true)
IReadOnlyList<int> ids = enc.GetIds();          // includes special tokens (e.g. XLM-R 0..2)
// also: enc.GetAttentionMask(), enc.GetTypeIds(), enc.GetSpecialMask(); tok.Decode(ids)

Then pad/truncate ids to the model's fixed seq length, build Tensor<int>(1,seq) for input_ids + attention_mask (XLM-R pad id = 1), and Schedule(idTensor, maskTensor).

LLM / transformer decoders

• No built-in KV cache — implement manually (carry past-key/value tensors as model inputs/outputs across the decode loop). See models/gemma3.md for the verified loop and reference/optimization-rules.md §5–5¾ for when a cache helps (it is NOT automatic).
• 🔴 Keep the KV cache ON-GPU — per-step ReadbackAndClone() of every present.* measured SLOWER than no cache at all (reference/optimization-rules.md §5). Official patterns below (reference/external-patterns.md §1) avoid the CPU round-trip entirely.
• ✅ On-GPU KV via CopyOutput — A/B VERIFIED (FunctionGemma, 18 layers). Per step replace nk = present.ReadbackAndClone() (a GPU→CPU sync × 2 × layers) with Tensor dk=null; worker.CopyOutput("present.N.key", ref dk); — null ref → exact-shape CloneEmpty on the backend (GPU→GPU MemCopy, no sync); feed dk as next step's past.N.key, dispose the buffer you owned last step. Null-alloc-per-step avoids CopyOutput's Reshape-capacity assert (preallocating a max-size buffer and letting the KV grow into it throws Tensor.Reshape: new length ... allocated on the backend). Measured faster per command with the per-step stall jitter removed — token-exact parity (numbers: measurements §9c-2). Keep it behind a useGpuKvCache-style toggle (A/B + parity fallback) on both the sync and coroutine decode loops.
• Token loop: tokenize (built-in tokenizer, above) → Schedule → argmax/sample logits → append → repeat to EOS or maxTokens. For big vocabs, bake argmax/sampling into the graph so you read back 1 int, not (1,1,vocab) (reference/external-patterns.md §2).
• 🔴 GPU-side ArgMax head — VERIFIED recipe (FunctionGemma, vocab 262 144). The naive path logits.ReadbackAndClone() clones the whole (1,seq,vocab) tensor to the CPU every token (worse: on the prefill pass that's seq×vocab floats) just to scan one row. Instead build a tiny standalone worker once and keep the wide logits on the device:

  static Worker BuildArgmaxHead(BackendType backend) {
      var g = new FunctionalGraph();
      var logits = g.AddInput<float>(new DynamicTensorShape(1, -1, Vocab)); // seq dynamic
      var ids = Functional.ArgMax(logits, 2, false);                        // (1, seq) int
      return new Worker(g.Compile(ids), backend);
  }
  // per step: feed the main worker's logits VIEW straight in (no CPU copy), read back seq ints, take last:
  m_Argmax.Schedule(m_Worker.PeekOutput("logits"));
  using var ids = (m_Argmax.PeekOutput() as Tensor<int>).ReadbackAndClone();
  int next = ids[0, ids.shape[1]-1];
  

  Run the head on the same backend as the decoder (keeps logits GPU-resident). Deterministic argmax → token-exact parity with the CPU scan (ORT-verified: all calls identical ON vs OFF). Measured GPUCompute: faster per command at zero accuracy cost — a free win, but the per-token compute (18 layers + the hidden×262144 unembed matmul) still dominates, so it is not a silver bullet; the next lever is slicing logits to the last position only before the unembed (export-side). Keep a useGpuArgmax toggle for A/B + parity fallback. Numbers: sentis-profiling reference/measurements.md §9c.
• Keep the model small (≤~0.5–1B) for real-time single-session demos.
• 🔴 CHAT SLM prompt formatting — build the ChatML string HOST-SIDE, then encode. For a chat/instruct decoder (system prompt, multi-turn), assemble the template yourself and feed it to the tokenizer; there is no in-graph template. VERIFIED (Qwen3-0.6B): the hand-built string <|im_start|>system\n{sys}<|im_end|>\n<|im_start|>user\n{user}<|im_end|>\n<|im_start|>assistant\n is byte-identical to Python tokenizer.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False) — diff the two once to lock the exact whitespace. The special tokens (<|im_start|>=151644, <|im_end|>=151645) tokenize correctly even with addSpecialTokens=false — the built-in tokenizer matches added-tokens in-text (same as FastVLM's ChatML). Stop on the chat EOS (<|im_end|>), not just <|endoftext|>.
  • Thinking models: Qwen3 emits <think>…</think> by default. For a small model / short-answer demo, prime the official non-thinking block (<|im_start|>assistant\n<think>\n\n</think>\n\n) so it answers directly; otherwise strip the think span before display. Show the user's raw text in the UI, feed the wrapped ChatML to the runner.
  • 🔴 Name trap: Qwen3-0.6B IS the chat model; Qwen3-0.6B-Base is the base. Don't run a chat model as raw completion (or vice-versa) — check for a chat_template.jinja in the export.

Verified EXTERNAL patterns → reference/external-patterns.md

Six source-verified patterns (official Unity samples / HF unity org) live in reference/external-patterns.md — read it before writing an LLM decode loop, a zero-readback CV pipeline, or mobile-targeted inference:

1. 🔴 On-GPU KV cache (two-worker handoff / CopyOutput double-buffer) — never readback KV per step
2. Bake argmax/sampling into the graph — read back 1 int, not (1,1,vocab)
3. Persistent pinned token tensor — zero per-token allocation
4. All-GPU CV pipeline (compute-shader pre-proc, in-graph NMS, RenderToTexture)
5. Burst post-processing on CPU tensors
6. Mobile traps (Android=Vulkan only, ONNX NMS runs on CPU, Mali Resize crash, few-MB design target)

Import formats (2.6)

ONNX (opset 7–25), LiteRT (formerly TFLite), and PyTorch ExportedProgram .pt2 (decomposed to Core ATen IR) all import directly. Drop the file in Assets/ → it becomes a ModelAsset. Constraints (import fails / converts silently): no tensors with >8 dims, no sparse / string / complex tensors; bool → float/int (memory↑); opset <7 or >25 may import but results are unpredictable. (Conversion traps live in sentis-model-converter.)

Ship the model: serialize (.sentis) + encrypt + load at runtime (VERIFIED)

Don't ship the raw ONNX. Serialize to a .sentis (FlatBuffers) — smaller, faster to load, validated. Editor: select the model → Inspector "Serialize to StreamingAssets", OR in code ModelWriter.Save(path, model). Both ModelWriter.Save/ModelLoader.Load have string path and Stream overloads. Verified in this project (YOLOv10n): .sentis round-trip (save→ModelLoader.Load(path)→run) reproduces the original output.

// editor/offline: bake once
ModelWriter.Save(Path.Combine(Application.streamingAssetsPath, "m.sentis"), model);
// runtime: load the serialized model (no ONNX in the build)
var model = ModelLoader.Load(Application.streamingAssetsPath + "/m.sentis");

Encrypt by wrapping the stream — VERIFIED end-to-end (AES CryptoStream, decrypt→load→run matched plaintext):

// save: write IV, then CryptoStream(encryptor) -> ModelWriter.Save(cryptoStream, model)
using var fs = new FileStream(path, FileMode.Create); fs.Write(aes.IV, 0, aes.IV.Length);
using var cs = new CryptoStream(fs, aes.CreateEncryptor(), CryptoStreamMode.Write);
ModelWriter.Save(cs, model);
// load: read IV, CryptoStream(decryptor) -> ModelLoader.Load(cryptoStream)
using var dcs = new CryptoStream(inFs, aes.CreateDecryptor(), CryptoStreamMode.Read);
var model = ModelLoader.Load(dcs);

Reading tensor data efficiently

Direct CPU indexing (cpu[i], tensor[0,i,j]) only works on a CPU tensor and is slow per-element — fine for a handful of values (argmax row, a few boxes), bad for bulk. For bulk, ReadbackAndClone() then DownloadToArray() once, or process with Burst / a compute shader / NativeArray. Tensors hold up to 8 dims (0-dim = scalar).

Sample / demo scenes: model-runner MODULE + Manager + authored UGUI (IMGUI only as a gated debug overlay)

🔴 MANDATORY DELIVERABLES — any "example / demo / sample / runnable inference" task ships ALL of these, not just a script:

1. A NEW dedicated scene built by an editor scene-builder (e.g. Assets/Scenes/<Model>Demo.unity) — never dump the demo into SampleScene or leave it scene-less. Save the scene asset.
2. Example inference code = the model-runner module (below).
3. All scene UI authored as real UGUI objects placed into that scene (Canvas + EventSystem + Camera + the HUD widgets), wired to the Manager's [SerializeField] fields by the builder — not created at runtime, never IMGUI.
4. .sentis serialization (§"Ship the model") — the builder/runner MUST serialize the built model to a .sentis via ModelWriter.Save and the runtime path MUST load the .sentis, not the raw ONNX. Include it even for a demo; it is the shipping-truth path and is non-optional.

A task that produces only a MonoBehaviour, or a scene with no UGUI, or skips .sentis, is INCOMPLETE — go back and finish it before claiming done.

Structure every runnable Sentis scene as three SEPARATED layers. Do not fuse inference + orchestration

• UI into one MonoBehaviour, and do not draw the UI with IMGUI (OnGUI / GUILayout / GUI.*) — use UGUI (UnityEngine.UI) authored into the scene. (Sole IMGUI exception: an existing debug overlay gated off behind a serialized bool — reference/demo-scene.md.)

1. Model-runner module — one class per model that owns the Worker + tensors and exposes a plain API (EnsureReady(), Warmup(), Transcribe() / Detect() / GenerateCall() …). No UI, no scene assumptions → reusable and unit-testable in isolation (one model per scene — never co-load; OOM).
2. Manager (MonoBehaviour) — composes the runner(s) + a reusable HUD module: sets the title, wires button callbacks to runner calls, pushes results to the UI. Holds serialized refs to both sides and is the ONLY layer that knows about both. Keep it working headless (log the report) when no HUD is assigned, so batch/CI runs still work.
3. UGUI authored into the scene — a reusable HUD/panel component (Canvas + Button + ScrollRect + InputField + Image bars) placed as REAL scene objects by an editor builder and wired to the Manager's [SerializeField] fields. The Manager attaches behaviour in Start() (button.onClick.AddListener(...)); it never new GameObject()s widgets at runtime.

🔴 Scene scaffolding — read reference/demo-scene.md before building or porting any scene. It holds the verified Unity-6 recipes and traps (absorbed from the retired sentis-demo-scene skill), each of which silently breaks the scene when skipped:

• code-built scenes via [MenuItem] — portable (no GUID refs) + agent-buildable; SerializedObject field wiring; cross-project porting (.meta files, skip the raw .onnx).
• camera requirement — a camera-less UGUI scene logs "No cameras rendering".
• NavMesh — legacy edit-time bake is EMPTY on Unity 6 → runtime NavMeshSurface bake.
• input handling — EventSystem needs InputSystemUIInputModule (value 1/2); all input code against UnityEngine.InputSystem, never UnityEngine.Input; legacy InputField is DEAD under New-only → activeInputHandler=Both + one restart; guard WASD reads while a text field is focused.
• Korean/CJK text — builtin font has no Hangul (tofu); swap to a dynamic OS font at RUNTIME in Awake (canonical stack in the reference; a serialized in-scene font can only be an asset).
• layout groups + timing readout; avatar chat-demo layout (URP magenta trap); gated debug overlay.

Reference files

• models/*.md — per-model VERIFIED recipes (table above; read only what the task needs).
• reference/optimization-rules.md — full optimization decision rules (§1–§8; stable IDs; §0/§2 exist only in the measurements log).
• reference/external-patterns.md — six source-verified external patterns (LLM loop, zero-readback CV, mobile).
• reference/demo-scene.md — demo/test scene scaffolding recipes and traps (builder, NavMesh, input, CJK font).
• Measured evidence behind every speed claim here: sentis-profiling reference/measurements.md (append-only §IDs; ratios/shares only — no absolute wall-clock values).

Self-check before claiming done

• using Unity.InferenceEngine; present, no Unity.Sentis.
• Worker + Schedule (not IWorker/Execute).
• Backend stated explicitly AND exposed as a selectable field (full 3-way CPU/GPUCompute/GPUPixel, not a bool useGpu); GPUCompute gated on SystemInfo.supportsComputeShaders; CPU-fallback risk considered. Multi-graph pipeline → backend chosen PER graph (heavy/once → GPU, tiny/readback/per-step-loop → CPU), each a selectable field.
• Every new Worker/new Tensor has a matching Dispose().
• Output read via ReadbackAndClone(), not off the live GPU view; output kept across Schedules uses CopyOutput, never a held PeekOutput view.
• Input shapes taken from model.inputs (dynamic dims resolved explicitly), not guessed.
• No QuantizationType.Int8.
• LLM decode loop: KV cache stays on-GPU (no per-step ReadbackAndClone of present.*); big-vocab logits reduced in-graph (argmax/sampling), not read back whole.
• Shipping a built model → serialize to .sentis (ModelWriter.Save) and load it at runtime, not the raw ONNX; Functional.Compile() baked offline, never at runtime.
• Used a model covered in models/*.md? → followed that recipe (don't re-derive I/O shapes from memory).
• Inference component has a Warmup(); always-on per-frame model reads back async.
• Any optimization claim: measured warm-vs-warm before/after on target (reference/optimization-rules.md self-check; capture/diagnosis via sentis-profiling).
• Runnable scene = model-runner module + Manager + authored UGUI (IMGUI only as a gated-off debug overlay); scene follows reference/demo-scene.md (code-built via MenuItem, camera present, NavMesh runtime bake, Input-System/InputField handler mode, CJK runtime OS font).
• Example/demo/sample task → shipped ALL four MANDATORY DELIVERABLES: a NEW saved scene, runner code, all UGUI authored into that scene, AND the .sentis ship path (previous item).
• Audio-input model → driven by REAL-TIME Microphone capture (not only clips); DSP behind a ProcessSamples() seam for parity tests; warm-up transient primed/suppressed; detection debounced.