sentis-profiling

name

sentis-profiling

description

Measure and diagnose Unity Sentis 2.6 (com.unity.ai.inference) inference performance. PRIMARY path = an agent-drivable code warm-bench (editor Unity_RunCommand) plus the Unity_Profiler_* MCP tools — these yield NUMBERS; the AI Assistant integration (6.4+) is a secondary, human-driven qualitative aid. Use whenever the task is: getting warm/cold latency numbers for a Sentis model, A/B-ing quantization or backends, finding the real bottleneck (compute vs readback vs KV round-trip), explaining a frame spike / stall / GC, detecting CPU-fallback layers, checking FPS budget, or deciding whether async readback / ScheduleIterable is defending FPS. Trigger even for vague asks like "why is my game slow when the model runs", or measuring/validating on a REAL build target (Android via adb, macOS standalone) — the editor only proxies a desktop ship, not a phone. Owns this project's measurement log (reference/measurements.md). Pairs with sentis-inference (the optimization levers the diagnosis points to).

Sentis Profiling — measure AND diagnose in code; Assistant = optional human-side aid

Measure with code; diagnose from profiler samples + §5 ground truth. The reliable, agent-drivable path is a code warm-bench (§2) plus the Unity_Profiler_* MCP tools — they produce NUMBERS you can A/B. The AI Assistant profiler feature (§4) is secondary: a human clicks "Ask Assistant" on a sample and gets guidance, not measurement. Either way, capture quality (§1) and Sentis ground truth (§5) decide whether the answer is real, and every claimed win must survive a warm-vs-warm before/after on the same device. This skill owns the project's measurement log: reference/measurements.md (ratios/shares only — no absolute wall-clock values). Fixes the diagnosis points to live in sentis-inference (reference/optimization-rules.md).

1. Know your baseline / capture deliberately (do this FIRST)

Per-model warm-cost verdicts (qualitative classes — numbers are measured fresh per target) live in reference/measurements.md §0/§9a–9b. Before touching anything, measure the baseline. Rules that make a capture meaningful:

• Warm vs cold: the FIRST inference is substantially slower (shader compile + alloc). Always warm ≥5–8 passes before timing, and know which frame/iteration was cold — mixing them produces confident nonsense.
• Editor timing is RELATIVE: editor frames are dominated by EditorLoop, and medians are inflated by EditorLoop/MCP interference — take the min as the cleaner warm figure, or capture a build (Autoconnect Profiler) for absolute per-layer times.
• Profiler MCP capture (agentic read path): Unity_Profiler_GetFrameTopTimeSamples, GetSampleTimeSummary, GetFrameGcAllocations etc. read a recorded session programmatically — use these to inspect frames without the human "Ask Assistant" UI. Sentis work shows in the CPU Usage module; Download/Upload samples = CPU↔GPU transfer.

2. Measure in code — the PRIMARY path (Unity_RunCommand warm-bench)

This is how this project's numbers were actually produced. An editor Unity_RunCommand loads the model(s), warms, and times warm iterations. It needs no scene and no human.

// internal class CommandScript : IRunCommand  — inside Execute(result):
double T() => UnityEditor.EditorApplication.timeSinceStartup * 1000.0;   // ms
var w = new Worker(ModelLoader.Load(asset), BackendType.GPUCompute);
for (int i = 0; i < 8; i++) { w.Schedule(input); using var o = (w.PeekOutput() as Tensor<float>).ReadbackAndClone(); } // warm
var times = new List<float>();
for (int i = 0; i < 30; i++) {
    double t0 = T();
    w.Schedule(input);
    using var o = (w.PeekOutput() as Tensor<float>).ReadbackAndClone();  // blocking readback = forces GPU drain = true warm latency
    times.Add((float)(T() - t0));
}
times.Sort(); float median = times[times.Count/2], min = times[0];       // report BOTH; min is the cleaner warm figure in-editor

🔴 Two RunCommand traps (both hit this project):

• System.Diagnostics.Stopwatch is type-forwarded out of the RunCommand context → compile error. Use EditorApplication.timeSinceStartup (double seconds, high-res).
• result.Log substitutes {0} but NOT {0:F3} — format specifiers print literally. Pre-format with value.ToString("F3", CultureInfo.InvariantCulture) and concatenate.

🔴 A synchronous RunCommand loop CANNOT bench GPUCompute (verified MobileCLIP, 2.6.1): its delayed-dispose reaper needs a frame boundary between Schedules, so a tight for-loop of Schedule+readback NREs in ComputeTensorDataReaper.ExecuteAsyncDisposeCommandBufferAndClear (ExecuteCommandBuffer buffer null). Worse, one crashed GPU command corrupts the global reaper → even later CPU Schedule NREs until a recompile/domain-reload resets it. Bench GPU per-frame in Play mode (§2b pattern: one Schedule/Update), not a sync loop. And an unfocused editor throttles play-mode Update to a crawl (headless MCP runs never focus it) → set Application.runInBackground = true in the bench's OnEnable, else it looks hung.

🔴 A sync RunCommand loop also LIES about the CPU/Burst backend (verified FastVLM-0.5B, 2.6.1): Burst jobs compile asynchronously, and a tight RunCommand loop times the ops before Burst finishes compiling → the slow managed/safety-mode path, inflated by an order of magnitude (measured FastVLM: the sync-bench CPU numbers were pure artifact vs the same model play-mode-warm — measured pair: measurements §14). It cost a WRONG backend conclusion because CPU was sync-benched while GPU was play-mode-benched — two harnesses = invalid A/B. Rule: bench BOTH backends the SAME way, in play-mode, warm (or force BurstCompiler.Options.EnableBurstCompileSynchronously = true before a sync CPU bench). A Semaphore.WaitForSignal-free, suspiciously-round-and-huge CPU time is the tell. (measurements §14.)

🔴 Fair A/B = INTERLEAVED round-robin, not sequential. Timing all of variant A then all of B biases the later one (GPU is hot / thermally throttled by then). Load all variants, warm each, then in the timing loop rotate for i: for k in variants: time(w[k]) so every variant shares the same thermal/driver state each round. Measured proof: timing order ALONE shifted the same E5 f16 A/B by as much as the effects you're typically hunting — so only the interleaved protocol is trustworthy (measured pair: measurements §3).

Quantization A/B: QuantizeWeights(ref model) → ModelWriter.Save(path) → ModelLoader.Load(path) (the Save→reload is MANDATORY) → bench the reloaded model. Always pair with an accuracy check on the SAME input (embedding cosine / detection IoU / transcript match), not just latency.

🔴 Separate compute from transfer (the decomposition trick). When ReadbackAndClone IS the GPU-sync, you can't time compute by wrapping Schedule alone. Trick: after Schedule, read the small output that forces the full drain FIRST (that call = compute + its own transfer), THEN read the other/bigger tensors (GPU is now idle → their readback is pure transfer). Used on Whisper: read logits first (=compute), then the KV tensors (=pure transfer) → proved the KV CPU round-trip was a minor share of the loop, so the "keep KV on-GPU" refactor was NOT worth the risk (numbers: measurements §6). Decompose like this before building any "fix" — a clever fix that addresses a sliver of the budget is not worth building.

2b. Built-in Profiler snapshot — the agentic capture→read recipe (VERIFIED)

When you want the built-in Profiler's per-sample breakdown (not just wall-clock ms from §2), capture and read it entirely from MCP — no human at the editor:

1. Drive the model per-frame so it lands in captured frames: a tiny MonoBehaviour holding a Worker, warmed in OnEnable, that Schedule+readback once in Update. A once-per-image model won't show up otherwise — you must make it run every frame for the capture window.
2. Enable recording BEFORE Play (one Unity_RunCommand, edit mode): UnityEditor.ProfilerDriver.enabled = true; UnityEngine.Profiling.Profiler.enabled = true; ProfilerDriver.ClearAllFrames();
3. Play, wait ~2–3 s (Bash sleep) so ~30+ frames record.
4. Get the frame range: ProfilerDriver.firstFrameIndex / lastFrameIndex (RunCommand).
5. Read a MID frame with the MCP tools (avoid the cold first frames). Exact tool names VERIFIED this session (the unity-mcp registry truncates+hashes them, so match by prefix):
   • Unity_Profiler_GetFrameTopTimeSam… (GetFrameTopTimeSamplesSummary; args frameIndex, targetFrameTime=16.6) → top level (PlayerLoop = real work vs EditorLoop = editor noise; expect PlayerLoop to dominate).
   • Unity_Profiler_GetFrameSelfTimeSa… (GetFrameSelfTimeSamplesSummary; arg frameIndex) → leaf hotspots = the real signal.
   • drill/aggregate helpers: Unity_Profiler_GetSampleTimeSummary (frameIndex + threadName + sampleId) and Unity_Profiler_GetFrameRangeTopTimeSummary (startFrameIndex, lastFrameIndex, targetFrameTime) for a multi-frame view.
6. Interpret the leaf self-time (ties to §5):
   • Semaphore.WaitForSignal dominating self-time = the main thread BLOCKING on the GPU → compute-bound, GPU-side; the cost is real GPU work behind the blocking readback, NOT CPU. (Measured: EfficientSAM3 encoder frame = WaitForSignal dominating frame self-time — measurements §10.)
   • Repeated Download/Upload leaf samples = CPU-fallback ping-pong (§3). Their ABSENCE (as here) proves a weak GPU-vs-CPU speedup is genuine heavy compute, not fallback.
   • On a GPU backend the per-layer GPU time collapses into that one WaitForSignal on the CPU timeline — these CPU-sample tools won't itemize GPU layers; use the GPU module for that.
7. Stop Play, ProfilerDriver.enabled = false. 🔴 MCP capture gotchas (both hit in this project): a Unity_RunCommand that pops an editor dialog fails with "User interactions are not supported" — NewScene/OpenScene on a dirty scene prompts to save, so keep the bench scene saved/clean or delete stray scene files via the filesystem. And SerializedProperty.enumValueIndex is the enum's NAME-ordinal, not its value — set intValue (or rely on the field default) when wiring a BackendType via SerializedObject.

2c. Measuring on a REAL device (Android VERIFIED; editor is only a partial proxy)

Everything above (§2/§2b) also runs on the target — but the editor's numbers only transfer to a desktop standalone, NOT to a phone. Get real numbers ON the device before trusting mobile perf.

• 🔑 editor ≈ desktop-standalone, editor ≠ mobile. Same EfficientSAM3: M1 editor ≈ macOS standalone player (near-identical), but the Android phone was several times slower (measurements §11/§12). Trust the editor for a desktop ship; re-measure on the actual phone for mobile.
• Backend availability is platform-gated (decide per §3, but know what EXISTS on the target): GPUCompute needs compute shaders (Vulkan / GLES3.1+ / Metal / D3D); WebGL has none → GPUPixel or CPU only (bench GPUPixel in-editor as the WebGL proxy). On-device order seen: GPU > CPU > GPUPixel on both Android and macOS (measurements §11/§12).
• How to get numbers on device (verified path): build a dev player that runs the model and Debug.Logs latency+correctness with a greppable tag → BuildPipeline.BuildPlayer (Unity_RunCommand) → adb install -r / launch (Bash) → adb logcat -d | grep '\[TAG\]' (macOS standalone: run the .app binary with -logFile, read Player.log — never -nographics, it kills the GPU backends). The remote Unity-Profiler→Unity_Profiler_*-over-network path is NOT verified.
• Re-run the correctness smoke test on-device (bake the reference as a .bytes TextAsset, compare): EfficientSAM3 was bit-exact on the phone across all 3 backends (refDiff 0.0000) — but you only know because you checked; uint8 accuracy especially is model-dependent (§5). Warm on the target GPU first (device shader-compile is bigger than editor). Generic build/packaging gotchas (IL2CPP time, APK size vs summary.totalSize, macOS cameraUsageDescription, Android Input-Handling) are plain Unity, not Sentis — see the project memory note, not this skill.

3. Diagnostics you read from a capture (backend / fallback / FPS / ScheduleIterable)

• CPU fallback: on a GPU backend, unsupported layers silently fall back to CPU — visible as Download/Upload pairs bracketing a layer (GPU→CPU, run, CPU→GPU). One or two at graph edges is normal I/O; repeated pairs mid-model = ping-pong that can dominate the frame. Fix: re-export without the unsupported op (sentis-model-converter), or move the whole model to CPU so there's nothing to ping-pong with.
• Which backend fits — decide by capture signature, not dogma. GPU busy + few transfers → GPUCompute is right. Many fallback pairs, or a tiny/readback-dominated model → try CPU (Burst). GPUPixel only for WebGL. Always A/B warm-vs-warm on target (§2) — and RE-A/B after changing the readback pattern: the 270M SLM won on CPU with per-step CPU readbacks but flipped to GPU once argmax+KV moved on-GPU (measurements §9b/§9c). Full sweep (§9b): small/light models → CPU; big transformers + E5 → GPU.
• FPS defense = async readback first: an always-on per-frame model with a blocking readback on the main thread is THE FPS killer — measured a large per-frame main-thread cost collapsing to ~zero by going async, even with a tiny output (measurements §4). Then verify the readback wait is gone from the main thread. Remaining over-budget cost = compute → next lever.
• Frame budget: report median AND worst-frame vs 16.6 ms @60 fps / 33.3 ms @30 fps — averages hide spikes, and spikes are what players feel.
• ScheduleIterable signature: when working, one big inference is smeared into K small even slices across frames (per-frame median dropped dramatically — measurements §4½) plus one completion-frame spike if the readback still blocks → combine with async. It SMOOTHS frame time; total GPU compute is unchanged — it does not make inference faster or finish sooner.

3b. Diagnosis → the lever, and CLOSE THE LOOP (profiling isn't done until the win is re-measured)

A diagnosis is only worth something if it names the fix AND you prove it. This skill decides WHICH lever the capture signature implies; the HOW lives in sentis-inference (reference/optimization-rules.md) or sentis-model-converter. Apply one change, then re-measure warm-vs-warm (§2) on the same device — a lever that doesn't beat baseline is REVERTED, not shipped; record the verdict in reference/measurements.md. (Verified rule, re-confirmed here: not every "optimization" wins.)

Capture signature (from §3 / §5)	Lever to reach for	Implemented in
WaitForSignal dominates, few transfers ⇒ compute-bound	backend A/B; then the only real cuts are a smaller model / lower input resolution / fewer layers (re-export). Quantize does NOT help speed.	inference / converter
blocking-readback stall on an always-on per-frame model	async readback (ReadbackRequest + harvest a later frame)	sentis-inference
one big inference spikes the frame	ScheduleIterable frame-split (+ async) — SMOOTHS, doesn't shorten	sentis-inference
repeated Download/Upload mid-model = CPU-fallback ping-pong	re-export without the unsupported op, or move the whole model to CPU	sentis-model-converter
tiny / readback-dominated model slower on GPU	run THAT model on CPU (Burst)	sentis-inference
memory-bound / >2 GB but latency fine	quantize f16/uint8 (MEMORY win; re-check accuracy)	inference / converter

🔴 Quantization is a memory lever, not a speed lever (re-confirmed on EfficientSAM3: encoder f16 = zero speed change, measurements §10). Only pull it when memory — not latency — is the constraint, and always re-check accuracy.

Worked closed loop (EfficientSAM3, reference/measurements.md §10): capture ⇒ encoder WaitForSignal dominating the frame, no fallback pairs ⇒ compute-bound. Levers tried + RE-MEASURED: decoder GPU→CPU (faster, kept — tiny/readback-dominated); encoder f16 (no speed win ⇒ kept for MEMORY only, half size, masks identical); lower-input-res = the sole remaining compute lever, deferred (needs re-export + accuracy check). Backend/quant cannot move the encoder's compute floor — the profiler proved where the floor is.

4. AI Assistant — SECONDARY, qualitative diagnosis (Unity 6.4+, human-driven)

Use when a human is at the editor and wants a narrative read of a capture. It records nothing — you capture (§1), it explains, and its output is guidance to verify against §5, never a measurement. Two entry points (both need com.unity.ai.assistant 2.13; the in-Profiler button needs Unity 6.4+):

• Assistant window ("where is time going overall?"): submit a perf question; it lists saved/active sessions → Analyze. None → it routes to Open Profiler to record.
• Profiler → select a sample → Ask Assistant ("explain THIS spike"): a prompt appears with the sample attached. Always edit it to add one line of Sentis context — model, backend, what the sample is — e.g. "Sentis 2.6 YOLOv10n on GPUCompute; the selected sample is the per-frame ReadbackAndClone — is this a GPU-sync stall or transfer?" That one sentence changes answer quality more than anything else. Then bisect by selecting other samples (readback sample, first-inference frame, a fallback layer's Download/Upload pair, a GC spike frame). Cross-check every conclusion against §5.

5. Verify any answer (Assistant's OR your own) against Sentis ground truth

• GPU ~idle + CPU high ⇒ blocking readback stall, not compute. ReadbackAndClone cost is the CPU waiting for queued GPU work, not transfer size (large even on a tiny 300×6 output — measurements §4). Fix = async readback, not a smaller output.
• A slow first frame is warmup, not a regression (cold is substantially slower) — check the frame #.
• Editor EditorLoop dominance is expected, not a finding; absolute claims need a build.
• Small steady GC (≲1 KB/frame) is normal; ReadbackAndClone is not a heavy GC source — don't start an allocation hunt on generic advice; measure GC first (it was negligible here).
• Quantization speed claims → §3b (memory lever, not speed) — and uint8 accuracy is model-dependent (safe on the conv detector, broke the fine-tuned Whisper transcript). Never assume 4× is free. (measurements §3/§3a)

6. Traps

• Code bench > Assistant for numbers. Assistant can't be driven agentically and gives qualitative guidance; §2's bench gives reproducible numbers. Reach for Assistant only for a human-in-editor narrative read.
• Bench-protocol traps all live in §2 — re-read it before benching: the Stopwatch/result.Log RunCommand traps, the GPU sync-loop reaper NRE, Burst-uncompiled CPU inflation, sequential-A/B bias.
• Analysis quality = capture quality. Cold/warm mixed, editor-only treated as absolute, or 3-frame captures produce confident nonsense. §1 first, always.

Reference files

• reference/measurements.md — THE project measurement log: the measured evidence behind the rules above lives there under an append-only, stable §ID (never renumbered), recorded as RATIOS / cost shares / budget fractions — never absolute wall-clock values (they don't transfer across devices/packages). SKILL.md files state directions only and cite the log.

Self-check before claiming a measurement/diagnosis

• Warmed ≥5–8 passes; know which run was cold (never timed the first call).
• Timed with EditorApplication.timeSinceStartup; reported median AND min (editor) or used a build for absolute; env (device/editor-vs-build/backend) stated.
• A/B variants timed INTERLEAVED, not sequentially.
• Accuracy re-checked with a fitting metric (cosine/IoU/transcript); quantized models went Save→reload before benching.
• Separated compute vs readback (drain-first trick) before proposing a readback/KV "fix".
• Readback advice: stall = GPU-sync wait → async, not smaller output.
• Backend justified by capture signature + A/B, not by default; CPU-fallback pairs checked.
• Frame-budget claims use median AND worst frame; ScheduleIterable read as smoothing.
• New numbers recorded in reference/measurements.md.