sentis-training-pipeline

name

sentis-training-pipeline

description

Training pipeline for Unity Sentis 2.6 on-device inference: find the training recipe in the model's HuggingFace repo, reshape the user's data to that recipe's schema, fine-tune, and choose the quantization method and ORDER. Use whenever the task is: fine-tuning any vision / audio / SLM model on a dataset, preparing a dataset for a specific model, or evaluating a fine-tune without fooling yourself (held-out split, separated capability axes). Encodes the recipe-from-repo rule, template byte-identity, the quantization ORDER, and Apple-MPS training economics that a general model gets wrong. ONNX conversion / graph surgery / import debugging = sentis-model-converter; C# runtime = sentis-inference.

Sentis Training Pipeline (HF recipe → data shaping → fine-tune → checkpoint handoff)

SCOPE (4-part split): this skill = Part 1 (HF recipe discovery → data shaping → fine-tuning) + quantization decision rules. Parts 2–3 (ONNX conversion + Sentis graph surgery) = sentis-model-converter skill. Part 4 (Unity C# inference) = sentis-inference skill. Recommended EXECUTION order is still conversion-first: let sentis-model-converter validate the export path with base/pretrained weights, then run the fine-tune here and swap in the checkpoint — the export path is identical either way.

STRUCTURE — how this skill stays fixed-size: this file contains only the GENERAL procedure and the trap TAXONOMY. Everything model-specific (verified commands, measured numbers, model-shape constants) lives in reference/models/<model>.md case files. Adding a new model = adding ONE case file; this file changes ONLY if a genuinely NEW trap class is discovered. When working on a model, read its case file if one exists; if none exists, follow the general procedure and WRITE the case file as you go (template at the bottom of this file).

Verified against Olive / Ultralytics / optimum docs. These facts override priors. The value of this skill is the recipe-from-repo rule and the training economics, not generic hyperparameter taste.

The pipeline & where the model's knowledge fails

HF recipe → data shaping → fine-tune → quantize ORDER → [sentis-model-converter] → [sentis-inference]
    ▲            ▲             ▲           ▲
repo is truth  schema rules  format    ORDER rule      ← model can't know these

A general model reaches for generic HF-Trainer boilerplate and quantizes in the wrong order.

0. Model selection — done BEFORE this skill (see sentis-model-scout)

Candidate finding & judging (size vs the 2 GB inline cap, license on code AND weights, language reality-check, graph op-scan) lives in the sentis-model-scout skill. This skill starts AFTER a candidate passed that gate: it owns recipe discovery, data shaping, fine-tuning, and the held-out evaluation of the fine-tune. Everything from "checkpoint" to "runs correctly in Unity" belongs to sentis-model-converter (conversion, surgery, ORT parity).

1. HF recipe discovery — the training recipe lives in the model's repo, not in priors

Before writing ANY training code, mine the model's HuggingFace repo (and the vendor's official repos/notebooks) for the recipe. Generic HF-Trainer boilerplate is how fine-tunes silently break (wrong template → train/inference mismatch). Collect, in order:

1. Model card: recommended fine-tune method, hyperparameters, official notebook/blog links, license (code AND weights — re-check even after sentis-model-scout).
2. Repo files that ARE the recipe: chat_template.jinja / tokenizer_config.json (prompt format — always render via tokenizer.apply_chat_template, never hand-build the template), generation_config.json (stop/eos ids), preprocessor_config.json (audio/vision preprocessing), config.json (arch: layers / kv-heads / head_dim → needed again at export and as C# constants).
3. Official training code: the vendor's own tool or notebook beats generic code — find it from the model card / official repo (a vendor CLI, an official Colab/notebook, or the framework's own fine-tune path). Prefer the vendor recipe's hyperparameters over taste; deviate deliberately, not by default. A matching reference/models/<model>.md case file, if one exists, already records the verified tool + gotchas for that family; if none exists, follow the general procedure and write one.
4. Write the chosen recipe down (source URL + exact template/schema) BEFORE shaping data — the dataset transform (§2) and the C# prompt builder (sentis-inference) must both derive from THIS single source.

2. Data shaping — transform the user's raw data into the recipe's schema

The recipe dictates the schema; the user's data almost never matches it. Own the transform:

1. Identify the target schema from §1 (detector = images + label files + data manifest; audio-STT = fixed sample-rate + transcript pairs; SLM = chat-template-rendered text, completion-only labels with prompt tokens masked to -100; TTS/VITS = metadata.csv (id|text[|speaker]) + wav/ at the model sample rate + a phoneme-id config.json, and the G2P (espeak vs neural-IPA) is a hard contract with the acoustic model; wake-word = feature-space binary classification — clips run through the FROZEN shared front-end models to feature arrays, and only a tiny classifier head trains).
2. Convert, then VALIDATE by rendering: script the transform, render a handful of examples and eyeball them — a mis-rendered template trains fine and fails at inference. For chat models assert the invariant: train input must be byte-identical to what inference will build — dump the rendered prompt once and reuse the same bytes in the C# prompt builder.
3. Split held-out BEFORE training — new phrasings, not shuffled duplicates (see "Separated eval"). For tool-calling models include out-of-scope negatives.
4. Sanity-scan the shaped set: per-class counts, empty labels, degenerate boxes, duration/sample-rate outliers, column-mapping mistakes in text templates.

3. Fine-tuning — general rules (verified per-model runs: reference/models/)

• Feasibility classes: small CV models on ~hundreds of images train fast enough on one GPU/Apple-MPS to be live-demo feasible. Anything LLM/seq2seq = pre-bake (too slow for a live demo, even at sub-1B). VITS-class TTS = pre-bake, full fine-tune resumed from a gender-matched checkpoint (~2000–3000 steps once the base matches the target's gender/pitch — a mismatched base does NOT converge with more steps). Wake-word models = train only the tiny classifier head on the frozen shared front-ends (via the vendor's official training pipeline, typically GPU/Colab), then bring just the resulting ONNX back to Sentis/Unity.
• Narrow-domain rule: narrow task → tiny data (~hundreds of examples), LoRA, very few epochs. Do not over-train — one epoch has been enough to take a narrow task from failing to passing. Include paraphrase diversity and out-of-scope negatives.
• 🔴 Apple-MPS traps (cost, not correctness): per-epoch generation eval dominates wall-time; batching generation makes it WORSE (memory cliff at moderate batch sizes); long sessions thermally throttle. Fix: train-only, checkpoint every epoch, defer real eval to the ORT-parity + Unity check (sentis-model-converter §7) — those run on CPU/ORT, independent of MPS state.
• Template byte-identity (from §2) is the invariant most fine-tunes break — re-verified on the TRAINED checkpoint at §6.3.

4. Quantization — decision rules (model gets the ORDER wrong)

• Standard order = fine-tune FIRST, then quantize (PTQ). Do not quantize a base model before fine-tuning and expect an accuracy win.
• QLoRA is the ONE inversion: quantize the base to 4-bit, then train LoRA adapters on top. This is a memory-efficiency trick for training, NOT a quality win — adapters merge back before deployment.
• AWQ = activation-aware, weight-only quantization; protects the ~1% of weights tied to high-activation channels. Olive supports it via AutoAWQ (4-bit) — ⚠ but NOT for a Sentis target: the quantized ONNX it emits cannot import (next bullet). Relevant only for non-Sentis runtimes.
• For a Sentis target, ModelQuantizer Float16/Uint8 is the ONLY quantization path — no Int8, and upstream INT8/AWQ/4-bit ONNX cannot import, ever (quantized-op family QLinear*, MatMulInteger… — sentis-model-converter §2c). Start Float16; move to Uint8 only if memory-bound; re-validate accuracy after. (C# call: sentis-inference.)

5. Checkpoint handoff → sentis-model-converter (conversion is NOT this skill's job)

Everything between "trained checkpoint" and "runs correctly in Unity" — exporter choice (torch/optimum/Olive/graphsurgeon), structural rewrites (KV-cache I/O, unmerged decoders, state hoisting), graph surgery, weight inlining, the trap taxonomy, and the ORT-parity ladder — lives in sentis-model-converter. Do not improvise export flags here; that skill has the verified incantations (sentis-model-converter/reference/verified_exports.md).

Hand over, explicitly:

1. The checkpoint (merged weights if LoRA — adapters merge back before export).
2. Arch constants from §1's recipe (layers / kv-heads / head_dim / vocab) — the export wrapper and the C# side both need them.
3. The rendered-prompt fixture (exact bytes from §2's byte-identity check) — it becomes the ORT-parity fixture, so the conversion is tested on the task's real inputs.
4. Target I/O contract (fixed shapes, expected pre/post split) if the demo constrains it.

Conversion-first (SCOPE note): the converter should already have validated the export path on base weights before training finishes. The per-model case files below (reference/models/) remain the full dossiers — the converter's verified_exports.md carries only their conversion-side distillation; keep both updated together.

6. Verification is mandatory (training side)

Fine-tuning fails silently — loss goes down while the deployed model is wrong. Enforce, in order:

1. Data shaping: render + eyeball transformed samples; held-out split made BEFORE training.
2. Fine-tune: task metric before/after on held-out (improved, not memorized?).
3. Re-verify template byte-identity on the TRAINED checkpoint before handing it over (§5).
4. Conversion/import verification (re-scan, ORT parity, Sentis import) is enforced by sentis-model-converter — the fine-tune is not "done" until that gate passes on the fine-tuned weights, not just the base weights. Delegate the edits to the agent; enforce the checks as procedure — never trust "it should work."

Separated eval — never report one number (AgentFlux pattern)

Evaluate on a HELD-OUT set of NEW phrasings (generalization, not memorization) and report capability axes SEPARATELY (e.g. for tool-calling: function selection / argument extraction / OOS rejection). General findings (measured; per-axis verdicts in reference/eval_log.md):

• The primary-task gain generalizes; quote the held-out verdict, never the curated demo score.
• Secondary extraction (arguments/slots) generalizes worse — diversify phrasings; expect a gap.
• ⚠ A capability NOT in the positives can REGRESS hard (OOS rejection went from perfect to badly broken after a narrow fine-tune; "more negatives + 1 epoch" verified NOT to fix it in a single run). Plan a runtime gate (confidence/whitelist) rather than assuming training fixes it.

Reference files

• reference/models/<model>.md — per-model case files: the full dossiers (template below). Coverage rule: 1:1 with sentis-inference/models/ (same file names). A model deployed WITHOUT a fine-tune still gets a short STATUS case file (as-is/zero-shot verdict + why — license / no released recipe / runtime tuning suffices — + the would-be recipe and handoffs), so this skill answers "can/should I fine-tune X?" for every deployed model. When a new model doc lands in sentis-inference/models/, add the matching case file here.
• reference/eval_log.md — per-axis held-out eval VERDICTS behind "Separated eval" (no raw scores).
• reference/olive.md — Olive notes (size/dtype passes only).
• reference/scripts/ — bundled VERIFIED helper scripts the case files cite (piper launcher + noise-baking export; openwakeword local + streaming-feature training).
• Graph scripts (scan_onnx.py, inline_onnx.py, surgery_gelu.py, gs_surgery.py, ort_parity.py) live in sentis-model-converter/scripts/ — run them from there; this skill keeps no local copies.

Case-file template (reference/models/<model>.md)

Keep each file self-contained and dated: (1) model id + arch constants (layers/kv-heads/head_dim/ vocab, size fp32/f16); (2) recipe source (URLs, template files used); (3) data schema + shaping notes; (4) verified fine-tune command + scale/epochs + measured results; (5) verified export incantation (exact flags) — mirror it into sentis-model-converter/reference/verified_exports.md; (6) traps hit, BY TAXONOMY CLASS (A–G, defined in sentis-model-converter §8), with the fix that worked; (7) parity numbers (ORT vs Sentis) + eval numbers (held-out, separated).

Self-check before claiming done

• Recipe sourced from the model's HF repo (template/preprocessor/official code), not boilerplate.
• Data transformed to the recipe schema; rendered samples eyeballed; train input byte-identical to inference input; held-out split (incl. negatives) made BEFORE training.
• Quantization ORDER correct (fine-tune→PTQ; QLoRA only as memory trick); no QuantizationType.Int8.
• Held-out eval reported on SEPARATED capability axes, not one curated number.
• Handoff package complete (§5): merged checkpoint + arch constants + rendered-prompt fixture + I/O contract, passed to sentis-model-converter; its ORT-parity gate run on the FT weights.
• If this model had no case file: reference/models/<model>.md written with what was learned.