On-device verification
A device result is done when three things hold:
- the model compiles and runs on the accelerator you claim it runs on,
- the output matches the source model numerically and on a task-level
gate,
- the record names the device, the runtime version, and the residency
line. A number without those is not reproducible and not a result.
Host-side checks (the CompiledModel checker used in gpu-clean-conversion)
exercise the host GPU. The device has its own shader compiler, its own
precision behavior, and its own memory ceiling — every failure mode in the
table below was hit by a model that had already passed on the host.
Loop
1. Dump references from the source model, once. Fixed inputs — one
real sample plus fixed-seed random — saved as .npy next to the recipe
(dump_*_ref.py). These are ground truth for every later step; regenerate
them only when the source model changes.
2. Run the same inputs on the device: CPU first, then GPU. One
argument switches the accelerator:
from ai_edge_litert.compiled_model import CompiledModel
from ai_edge_litert.hardware_accelerator import HardwareAccelerator
model = CompiledModel.from_file(
"model.tflite", hardware_accel=HardwareAccelerator.GPU) # or .CPU
The device-CPU run is the control. If it already diverges from the source
dump, the problem is the conversion, not the GPU — go back to
gpu-clean-conversion. A full worked example of the A/B lives in this
repo at samples/litert/speech_recognition/convert/verify_tflite.py.
Ask for the strict accelerator. Compiling with
HardwareAccelerator.CPU | HardwareAccelerator.GPU permits partial
delegation and hides fallback; use the combined mode only to discover
which ops fell back after a strict GPU compile fails.
3. Read the delegate log before reading any numbers.
Replacing N out of M node(s) with delegate ... X partitions
Record N/M and the partition count. N < M or X > 1 means part of the
graph runs on the CPU — decide whether that is acceptable before quoting
any accuracy or latency number.
4. Gate the output three ways.
- Numeric: correlation and max abs diff against the source dump.
Expect genuine GPU fp16 to drift in the last digits — exact equality is
a symptom, not a pass (see below).
- Task: argmax match, IoU, token-for-token greedy decode — whatever
the model is for.
- Artifact, for generative models: the decoded text / audio / image.
An artifact gate catches argmax ties that numeric tolerance misses.
5. Record it: device, runtime version, N/M + partitions,
correlation, max abs diff, task result — one row per device in the recipe
README.
The silent CPU fallback
The most common false positive: everything runs and the numbers match
perfectly. If the GPU output is bit-identical to the device CPU fp32
output, it almost certainly did not run on the GPU. Perfect equality is
the tell, not the goal. Cross-check the residency line; only when N/M
is full and the outputs drift in the last digits are you looking at a
real GPU run.
Device-only failures
| What you see |
What it is, what to do |
| GPU compile fails on device for a graph that is op-clean and passed the host check |
A whole-graph compile ceiling, not a bad op — a fused graph can fail where each half compiles. Split at a natural block boundary (conv frontend / transformer encoder), verify split == monolith bit-exact on the host, ship the halves |
| The file refuses to load at all |
The >2 GB flatbuffer limit. Split, or quantize below it |
| Full residency, wrong numbers |
Bisect by intermediates: re-export with block-boundary tensors as extra outputs and find the first one that diverges. The cause is usually a reduction (mean, variance, Σx²) in fp16, not the op you suspect. If materializing a tap fixes the numbers, that localizes the bug to a fusion boundary — record it as a finding, not a nuisance |
| Wrong-but-plausible output, and every hypothesis costs a slow re-export |
Micro-probe instead: export ~1 KB graphs — the suspect subexpression and each candidate fix — and run them through the device harness you already have. Minutes per hypothesis instead of a re-export per hypothesis |
| NaN or garbage only on the GPU, in a model with large-magnitude residuals or deep modulation paths |
An fp16 range break. Confirm the attribution by forcing fp32 on the delegate where the API exposes it (~2× memory, slower); then fix it properly with the fp16-safe rewrites in gpu-clean-conversion, or keep the offending block on CPU and run the rest on GPU |
| The process dies while loading or running a large float model |
A memory ceiling, not a model bug. Weights + activations + delegate buffers must fit in available memory. Do not run a multi-GiB fp32 build in-process on a phone "just to check" — quantize or split first, then verify the smaller thing |
| A deep transformer shipped as an fp16 graph is bit-exact on desktop and noise on the device CPU |
Android ARM XNNPACK computes native fp16; desktop XNNPACK upcasts to fp32. Deep residual streams compound the difference to collapse. Ship fp32 graphs for CPU inference on device; fp16 is a GPU-side format |
| Attention quality collapses only for a small-head submodule |
head_dim is the fp16-fragile axis, not token count: the same graph at head_dim 64 held corr 0.998 where head_dim 16 fell to 0.86. Pad heads to ≥32 or keep the small-head module on CPU |
A [1,N,C] token tensor that is a graph output and feeds other consumers comes back corrupted |
3-D fan-out corruption — the later branch is clobbered, it cascades, and it reads exactly like an fp16 wall downstream (4-D NCHW maps with the same fan-out are fine). Keep token tensors as sole leaf outputs (or keep them 4-D) and push per-token heads to the host — exact, since per-token ops commute with the gather |
| A recurrent/streaming graph gives correct output on call 1 and drifts on repeated calls |
Fused-LSTM-style variable tensors persist across invoke() on a reused interpreter — and a fresh-interpreter-per-call verify script structurally cannot see it. Call reset_all_variables() before every invoke (cost ≈ 0). Related: the CompiledModel loader rejects variable tensors outright, so such graphs are Interpreter-only |
Watch for
- Two references, two verdicts. The source-framework dump is the
truth; the device CPU run is the control that isolates the GPU. A
GPU-vs-CPU comparison alone can pass while both are wrong.
- The fp32-forcing knob is for attribution, not shipping. It tells
you precision is the cause; the fix is a rewrite or hybrid placement.
- First inference includes shader compilation. Correctness on the
first run is fine; never quote first-run latency, and never let a
latency number travel without its residency line.
- One device proves correctness, not portability. GPU compilers
differ per vendor — a compile ceiling on one chip may not exist on
another. "Runs on Android GPUs" means a device matrix (your own
devices, or a farm service such as AI Edge Portal), recorded as one
row each.
- One runtime proves it for that runtime. A delegate rejection or
miscompute is a fact about the runtime version you measured: ops have
been dropped between minor versions, a miscompute's victim output
has moved between versions, and mixing accelerator and core
libraries across versions silently falls back to CPU. When a wall
appears after an upgrade, bisect the runtime pin on the real graph —
micro-probes have repeatedly failed to reproduce walls that only fire
in full-graph context, so a negative micro-probe is not a refutation.
- Localize miscomputes with single-output graphs, never fan-out
taps. A multi-output tapped probe is itself exposed to
output-aliasing bugs and has produced a confidently wrong culprit;
in single-output form every op was exact and the assembly was the
bug.
- Sweep the delegate options to classify a miscompute. Run the same
graph across precision, buffer-storage, and backend options: a
bit-identical wrong result across all of them places the bug in the
shared graph-compilation layer and rules out precision/storage in one
pass. And know what the precision flag can do: forcing fp32 rescues
overflow→NaN cases only — it does not fix precision compounding
(the delegate still reduces in fp16), so "fp32 didn't help" does not
exonerate fp16.
- Time the enqueue and the readback as separate counters.
run()
is asynchronous; timing it alone has reported a 4× GPU win that did
not exist. A large readback time is usually the deferred compute, not
the transfer. Corollary economics: per-call overhead makes small
per-step graphs (KV-cache decoders re-uploading state every token) a
net GPU loss — estimate calls × per-call overhead against the CPU
time before re-exporting for GPU; the crossover sits around
hundreds of nodes per call.
- The desktop build is the CPU reference, not a GPU sieve — desktop
Python runtimes exercise CPU/XNNPACK only, which is exactly what makes
them the right numerical reference. For the device loop, a minimal
push-run-pull binary (tflite in, output tensor out) iterates in
seconds without an app rebuild.
Output layout
Verification is part of the model recipe, not a side script:
models/<family>/<model>/converted/
dump_*_ref.py source-model reference dumps (.npy)
verify_*.py parity vs those references, accelerator as a flag
README.md per-device table: device | accelerator |
N/M nodes, partitions | corr | max abs | task gate
Keep the dump and the verify separately runnable: references are dumped
once on the host, verification re-runs on every device and after every
model change.
1---2name: on-device-verification3description: Prove a converted or quantized LiteRT model on the actual device via the CompiledModel API - confirm GPU residency, compare device output against the source model, and diagnose device-only failures such as silent CPU fallback, whole-graph compile ceilings, and fp16 range breaks. Use after conversion or quantization, when device output is wrong or NaN, when a clean graph fails to compile only on device, or when GPU and CPU outputs are suspiciously identical.4---56# On-device verification78A device result is done when three things hold:9101. the model compiles and runs on the accelerator you claim it runs on,112. the output matches the source model numerically **and** on a task-level12 gate,133. the record names the device, the runtime version, and the residency14 line. A number without those is not reproducible and not a result.1516Host-side checks (the CompiledModel checker used in `gpu-clean-conversion`)17exercise the host GPU. The device has its own shader compiler, its own18precision behavior, and its own memory ceiling — every failure mode in the19table below was hit by a model that had already passed on the host.2021## Loop2223**1. Dump references from the source model, once.** Fixed inputs — one24real sample plus fixed-seed random — saved as `.npy` next to the recipe25(`dump_*_ref.py`). These are ground truth for every later step; regenerate26them only when the source model changes.2728**2. Run the same inputs on the device: CPU first, then GPU.** One29argument switches the accelerator:3031```python32from ai_edge_litert.compiled_model import CompiledModel33from ai_edge_litert.hardware_accelerator import HardwareAccelerator3435model = CompiledModel.from_file(36 "model.tflite", hardware_accel=HardwareAccelerator.GPU) # or .CPU37```3839The device-CPU run is the control. If it already diverges from the source40dump, the problem is the conversion, not the GPU — go back to41`gpu-clean-conversion`. A full worked example of the A/B lives in this42repo at `samples/litert/speech_recognition/convert/verify_tflite.py`.4344Ask for the strict accelerator. Compiling with45`HardwareAccelerator.CPU | HardwareAccelerator.GPU` permits partial46delegation and hides fallback; use the combined mode only to discover47*which* ops fell back after a strict GPU compile fails.4849**3. Read the delegate log before reading any numbers.**5051```52Replacing N out of M node(s) with delegate ... X partitions53```5455Record `N/M` and the partition count. `N < M` or `X > 1` means part of the56graph runs on the CPU — decide whether that is acceptable before quoting57any accuracy or latency number.5859**4. Gate the output three ways.**6061- **Numeric**: correlation and max abs diff against the source dump.62 Expect genuine GPU fp16 to drift in the last digits — exact equality is63 a symptom, not a pass (see below).64- **Task**: argmax match, IoU, token-for-token greedy decode — whatever65 the model is for.66- **Artifact**, for generative models: the decoded text / audio / image.67 An artifact gate catches argmax ties that numeric tolerance misses.6869**5. Record it**: device, runtime version, `N/M` + partitions,70correlation, max abs diff, task result — one row per device in the recipe71README.7273## The silent CPU fallback7475The most common false positive: everything runs and the numbers match76perfectly. If the GPU output is **bit-identical to the device CPU fp3277output, it almost certainly did not run on the GPU.** Perfect equality is78the tell, not the goal. Cross-check the residency line; only when `N/M`79is full *and* the outputs drift in the last digits are you looking at a80real GPU run.8182## Device-only failures8384| What you see | What it is, what to do |85|---|---|86| GPU compile fails on device for a graph that is op-clean and passed the host check | A whole-graph compile ceiling, not a bad op — a fused graph can fail where each half compiles. Split at a natural block boundary (conv frontend / transformer encoder), verify split == monolith bit-exact on the host, ship the halves |87| The file refuses to load at all | The >2 GB flatbuffer limit. Split, or quantize below it |88| Full residency, wrong numbers | Bisect by intermediates: re-export with block-boundary tensors as extra outputs and find the first one that diverges. The cause is usually a reduction (mean, variance, Σx²) in fp16, not the op you suspect. If materializing a tap *fixes* the numbers, that localizes the bug to a fusion boundary — record it as a finding, not a nuisance |89| Wrong-but-plausible output, and every hypothesis costs a slow re-export | Micro-probe instead: export ~1 KB graphs — the suspect subexpression and each candidate fix — and run them through the device harness you already have. Minutes per hypothesis instead of a re-export per hypothesis |90| NaN or garbage only on the GPU, in a model with large-magnitude residuals or deep modulation paths | An fp16 range break. Confirm the attribution by forcing fp32 on the delegate where the API exposes it (~2× memory, slower); then fix it properly with the fp16-safe rewrites in `gpu-clean-conversion`, or keep the offending block on CPU and run the rest on GPU |91| The process dies while loading or running a large float model | A memory ceiling, not a model bug. Weights + activations + delegate buffers must fit in available memory. Do not run a multi-GiB fp32 build in-process on a phone "just to check" — quantize or split first, then verify the smaller thing |92| A deep transformer shipped as an fp16 graph is bit-exact on desktop and noise on the device CPU | Android ARM XNNPACK computes **native fp16**; desktop XNNPACK upcasts to fp32. Deep residual streams compound the difference to collapse. Ship fp32 graphs for CPU inference on device; fp16 is a GPU-side format |93| Attention quality collapses only for a small-head submodule | head_dim is the fp16-fragile axis, not token count: the same graph at head_dim 64 held corr 0.998 where head_dim 16 fell to 0.86. Pad heads to ≥32 or keep the small-head module on CPU |94| A `[1,N,C]` token tensor that is a graph output **and** feeds other consumers comes back corrupted | 3-D fan-out corruption — the later branch is clobbered, it cascades, and it reads exactly like an fp16 wall downstream (4-D NCHW maps with the same fan-out are fine). Keep token tensors as sole leaf outputs (or keep them 4-D) and push per-token heads to the host — exact, since per-token ops commute with the gather |95| A recurrent/streaming graph gives correct output on call 1 and drifts on repeated calls | Fused-LSTM-style **variable tensors persist across `invoke()`** on a reused interpreter — and a fresh-interpreter-per-call verify script structurally cannot see it. Call `reset_all_variables()` before every invoke (cost ≈ 0). Related: the CompiledModel loader rejects variable tensors outright, so such graphs are Interpreter-only |9697## Watch for9899- **Two references, two verdicts.** The source-framework dump is the100 truth; the device CPU run is the control that isolates the GPU. A101 GPU-vs-CPU comparison alone can pass while both are wrong.102- **The fp32-forcing knob is for attribution, not shipping.** It tells103 you precision is the cause; the fix is a rewrite or hybrid placement.104- **First inference includes shader compilation.** Correctness on the105 first run is fine; never quote first-run latency, and never let a106 latency number travel without its residency line.107- **One device proves correctness, not portability.** GPU compilers108 differ per vendor — a compile ceiling on one chip may not exist on109 another. "Runs on Android GPUs" means a device matrix (your own110 devices, or a farm service such as AI Edge Portal), recorded as one111 row each.112- **One runtime proves it for that runtime.** A delegate rejection or113 miscompute is a fact about the runtime version you measured: ops have114 been *dropped* between minor versions, a miscompute's victim output115 has *moved* between versions, and mixing accelerator and core116 libraries across versions silently falls back to CPU. When a wall117 appears after an upgrade, bisect the runtime pin on the real graph —118 micro-probes have repeatedly failed to reproduce walls that only fire119 in full-graph context, so a negative micro-probe is not a refutation.120- **Localize miscomputes with single-output graphs, never fan-out121 taps.** A multi-output tapped probe is itself exposed to122 output-aliasing bugs and has produced a confidently wrong culprit;123 in single-output form every op was exact and the *assembly* was the124 bug.125- **Sweep the delegate options to classify a miscompute.** Run the same126 graph across precision, buffer-storage, and backend options: a127 bit-identical wrong result across all of them places the bug in the128 shared graph-compilation layer and rules out precision/storage in one129 pass. And know what the precision flag can do: forcing fp32 rescues130 overflow→NaN cases only — it does **not** fix precision compounding131 (the delegate still reduces in fp16), so "fp32 didn't help" does not132 exonerate fp16.133- **Time the enqueue and the readback as separate counters.** `run()`134 is asynchronous; timing it alone has reported a 4× GPU win that did135 not exist. A large readback time is usually the deferred compute, not136 the transfer. Corollary economics: per-call overhead makes small137 per-step graphs (KV-cache decoders re-uploading state every token) a138 net GPU loss — estimate `calls × per-call overhead` against the CPU139 time before re-exporting for GPU; the crossover sits around140 hundreds of nodes per call.141- **The desktop build is the CPU reference, not a GPU sieve** — desktop142 Python runtimes exercise CPU/XNNPACK only, which is exactly what makes143 them the right numerical reference. For the device loop, a minimal144 push-run-pull binary (tflite in, output tensor out) iterates in145 seconds without an app rebuild.146147## Output layout148149Verification is part of the model recipe, not a side script:150151```152models/<family>/<model>/converted/153 dump_*_ref.py source-model reference dumps (.npy)154 verify_*.py parity vs those references, accelerator as a flag155 README.md per-device table: device | accelerator |156 N/M nodes, partitions | corr | max abs | task gate157```158159Keep the dump and the verify separately runnable: references are dumped160once on the host, verification re-runs on every device and after every161model change.
google-ai-edge