accuracy-safe-quantization

name

accuracy-safe-quantization

description

Shrink a converted LiteRT model with ai-edge-quantizer (fp16 / int8 / int4) without losing accuracy, verifying parity against the float source after every step. Use when choosing a quantization recipe for a new model, when a quantized model fails to load, degrades on a task benchmark, or degenerates over long generations, or when deciding between dynamic-range, weight-only, and blockwise variants.

Accuracy-safe quantization

A quantization is done when three things hold, in this order:

1. it exports and the file shrinks by what the recipe predicts,
2. output parity with the float source holds on a task-level check, not just a smoke test,
3. the quantized model still passes the deployment check on the target runtime and device.

Quantization rewrites the graph, so step 3 is a fresh obligation every time: re-run the same CompiledModel verification you used to accept the float conversion (see the gpu-clean-conversion skill), then the on-device numerical check.

All recipes below are ai-edge-quantizer (pip install ai-edge-quantizer), plain Python, no build step. Worked examples live in this repo under models/bonsai/bonsai_image_4b/converted/ and models/qwen/qwen3_tts/converted/.

Choosing a lane

Start with the lightest recipe that meets the size budget, and move down only on evidence:

Budget / model	Recipe
~2× smaller, zero risk	fp16 float-casting. Weights cast to fp16, compute stays float. On a GPU that already computes in fp16 this is close to free numerically — verify anyway
~4× smaller — encoders, conv nets, diffusion blocks	Dynamic-range int8 channelwise. int8 weights, float activations; this shape rides the GPU delegate