on-device-verification

name

on-device-verification

description

Prove a converted or quantized LiteRT model on the actual device via the CompiledModel API - confirm GPU residency, compare device output against the source model, and diagnose device-only failures such as silent CPU fallback, whole-graph compile ceilings, and fp16 range breaks. Use after conversion or quantization, when device output is wrong or NaN, when a clean graph fails to compile only on device, or when GPU and CPU outputs are suspiciously identical.

On-device verification

A device result is done when three things hold:

1. the model compiles and runs on the accelerator you claim it runs on,
2. the output matches the source model numerically and on a task-level gate,
3. the record names the device, the runtime version, and the residency line. A number without those is not reproducible and not a result.

Host-side checks (the CompiledModel checker used in gpu-clean-conversion) exercise the host GPU. The device has its own shader compiler, its own precision behavior, and its own memory ceiling — every failure mode in the table below was hit by a model that had already passed on the host.

Loop

1. Dump references from the source model, once. Fixed inputs — one real sample plus fixed-seed random — saved as .npy next to the recipe (dump_*_ref.py). These are ground truth for every later step; regenerate them only when the source model changes.

2. Run the same inputs on the device: CPU first, then GPU. One argument switches the accelerator:

from ai_edge_litert.compiled_model import CompiledModel
from ai_edge_litert.hardware_accelerator import HardwareAccelerator

model = CompiledModel.from_file(
    "model.tflite", hardware_accel=HardwareAccelerator.GPU)  # or .CPU

The device-CPU run is the control. If it already diverges from the source dump, the problem is the conversion, not the GPU — go back to gpu-clean-conversion. A full worked example of the A/B lives in this repo at samples/litert/speech_recognition/convert/verify_tflite.py.

Ask for the strict accelerator. Compiling with HardwareAccelerator.CPU | HardwareAccelerator.GPU permits partial delegation and hides fallback; use the combined mode only to discover which ops fell back after a strict GPU compile fails.

3. Read the delegate log before reading any numbers.

Replacing N out of M node(s) with delegate ... X partitions

Record N/M and the partition count. N < M or X > 1 means part of the graph runs on the CPU — decide whether that is acceptable before quoting any accuracy or latency number.

4. Gate the output three ways.

• Numeric: correlation and max abs diff against the source dump. Expect genuine GPU fp16 to drift in the last digits — exact equality is a symptom, not a pass (see below).
• Task: argmax match, IoU, token-for-token greedy decode — whatever the model is for.
• Artifact, for generative models: the decoded text / audio / image. An artifact gate catches argmax ties that numeric tolerance misses.

5. Record it: device, runtime version, N/M + partitions, correlation, max abs diff, task result — one row per device in the recipe README.

The silent CPU fallback

The most common false positive: everything runs and the numbers match perfectly. If the GPU output is bit-identical to the device CPU fp32 output, it almost certainly did not run on the GPU. Perfect equality is the tell, not the goal. Cross-check the residency line; only when N/M is full and the outputs drift in the last digits are you looking at a real GPU run.

Device-only failures

What you see	What it is, what to do
GPU compile fails on device for a graph that is op-clean and passed the host check	A whole-graph compile ceiling, not a bad op — a fused graph can fail where each half compiles. Split at a natural block boundary (conv frontend / transformer encoder), verify split == monolith bit-exact on the host, ship the halves
The file refuses to load at all	The >2 GB flatbuffer limit. Split, or quantize below it
Full residency, wrong numbers	Bisect by intermediates: re-export with block-boundary tensors as extra outputs and find the first one that diverges. The cause is usually a reduction (mean, variance, Σx²) in fp16, not the op you suspect. If materializing a tap fixes the numbers, that localizes the bug to a fusion boundary — record it as a finding, not a nuisance
Wrong-but-plausible output, and every hypothesis costs a slow re-export	Micro-probe instead: export ~1 KB graphs — the suspect subexpression and each candidate fix — and run them through the device harness you already have. Minutes per hypothesis instead of a re-export per hypothesis
NaN or garbage only on the GPU, in a model with large-magnitude residuals or deep modulation paths	An fp16 range break. Confirm the attribution by forcing fp32 on the delegate where the API exposes it (~2× memory, slower); then fix it properly with the fp16-safe rewrites in gpu-clean-conversion, or keep the offending block on CPU and run the rest on GPU
The process dies while loading or running a large float model	A memory ceiling, not a model bug. Weights + activations + delegate buffers must fit in available memory. Do not run a multi-GiB fp32 build in-process on a phone "just to check" — quantize or split first, then verify the smaller thing
A deep transformer shipped as an fp16 graph is bit-exact on desktop and noise on the device CPU	Android ARM XNNPACK computes native fp16; desktop XNNPACK upcasts to fp32. Deep residual streams compound the difference to collapse. Ship fp32 graphs for CPU inference on device; fp16 is a GPU-side format
Attention quality collapses only for a small-head submodule	head_dim is the fp16-fragile axis, not token count: the same graph at head_dim 64 held corr 0.998 where head_dim 16 fell to 0.86. Pad heads to ≥32 or keep the small-head module on CPU
A [1,N,C] token tensor that is a graph output and feeds other consumers comes back corrupted	3-D fan-out corruption — the later branch is clobbered, it cascades, and it reads exactly like an fp16 wall downstream (4-D NCHW maps with the same fan-out are fine). Keep token tensors as sole leaf outputs (or keep them 4-D) and push per-token heads to the host — exact, since per-token ops commute with the gather

Watch for

• Two references, two verdicts. The source-framework dump is the truth; the device CPU run is the control that isolates the GPU. A GPU-vs-CPU comparison alone can pass while both are wrong.
• The fp32-forcing knob is for attribution, not shipping. It tells you precision is the cause; the fix is a rewrite or hybrid placement.
• First inference includes shader compilation. Correctness on the first run is fine; never quote first-run latency, and never let a latency number travel without its residency line.
• One device proves correctness, not portability. GPU compilers differ per vendor — a compile ceiling on one chip may not exist on another. "Runs on Android GPUs" means a device matrix (your own devices, or a farm service such as AI Edge Portal), recorded as one row each.
• One runtime proves it for that runtime. A delegate rejection or miscompute is a fact about the runtime version you measured: ops have been dropped between minor versions, a miscompute's victim output has moved between versions, and mixing accelerator and core libraries across versions silently falls back to CPU. When a wall appears after an upgrade, bisect the runtime pin on the real graph — micro-probes have repeatedly failed to reproduce walls that only fire in full-graph context, so a negative micro-probe is not a refutation.
• Localize miscomputes with single-output graphs, never fan-out taps. A multi-output tapped probe is itself exposed to output-aliasing bugs and has produced a confidently wrong culprit; in single-output form every op was exact and the assembly was the bug.
• Sweep the delegate options to classify a miscompute. Run the same graph across precision, buffer-storage, and backend options: a bit-identical wrong result across all of them places the bug in the shared graph-compilation layer and rules out precision/storage in one pass. And know what the precision flag can do: forcing fp32 rescues overflow→NaN cases only — it does not fix precision compounding (the delegate still reduces in fp16), so "fp32 didn't help" does not exonerate fp16.
• Time the enqueue and the readback as separate counters. run() is asynchronous; timing it alone has reported a 4× GPU win that did not exist. A large readback time is usually the deferred compute, not the transfer. Corollary economics: per-call overhead makes small per-step graphs (KV-cache decoders re-uploading state every token) a net GPU loss — estimate calls × per-call overhead against the CPU time before re-exporting for GPU; the crossover sits around hundreds of nodes per call.
• The desktop build is the CPU reference, not a GPU sieve — desktop Python runtimes exercise CPU/XNNPACK only, which is exactly what makes them the right numerical reference. For the device loop, a minimal push-run-pull binary (tflite in, output tensor out) iterates in seconds without an app rebuild.

Output layout

Verification is part of the model recipe, not a side script:

models/<family>/<model>/converted/
  dump_*_ref.py            source-model reference dumps (.npy)
  verify_*.py              parity vs those references, accelerator as a flag
  README.md                per-device table: device | accelerator |
                           N/M nodes, partitions | corr | max abs | task gate

Keep the dump and the verify separately runnable: references are dumped once on the host, verification re-runs on every device and after every model change.