getitune-optimizing-a-model

# Optimizing (quantizing) a model with getitune `getitune` applies **post-training quantization (PTQ)** via [NNCF](https://github.com/openvinotoolkit/nncf) to shrink an exported OpenVINO model and speed up inference. Quantization runs on an **OpenVINO** model (an exported `.xml`), producing an INT8 version. Run everything from `library/`. ## Workflow ```python from getitune.engine import create_engine # Load an exported OpenVINO model, then quantize it ov_engine = create_engine( model="/path/to/exported_model.xml", data="/path/to/dataset", ) ov_engine.optimize() # INT8 post-training quantization via NNCF int8_metrics = ov_engine.test() # validate the quantized model predictions = ov_engine.predict() # run inference with the quantized model ``` 1. **Start from an exported OpenVINO model** (`.xml`). If you only have a checkpoint, export it first with the `getitune-exporting-a-model` skill. - Done when: `create_engine(model="....xml", data=...)` builds an `OVEngine`. 2. **Provide a calibration dataset.** Calibration images are taken automatically from the training subset; 200-500 images is the recommended calibration size. - Done when: `optimize()` runs without a "not enough calibration data" issue. 3. **Run `optimize()`.** This replaces the engine's model in place with the INT8 version. - Done when: the call completes and subsequent `test()`/`predict()` use INT8. 4. **Re-validate accuracy** with `test()` and compare against the FP32/FP16 baseline; a small accuracy drop is expected in exchange for size/latency. - Done when: the INT8 metric is within your acceptable tolerance of baseline. ## Comparing against the original model After `optimize()` the engine holds the INT8 model. To re-check the original FP32/FP16 model, either pass the original `.xml` path directly to `.test()` / `.predict()`, or create the engine again from the original `.xml`. ## Notes - Quantization is OpenVINO/NNCF-based and applies to exported IR models — it is not a training-time step. - **Only OpenVINO IR (`.xml`) is supported.** An ONNX model must be converted to OpenVINO IR first before it can be optimized. - In the Geti application this is exposed as the `quantize` job (`application/backend/app/execution/quantization/`); library `optimize()` is the same capability without the job/queue wrapper. ## Verify ```bash # from library/ just lint just test-unit -- -k optimize # when you touched optimization code ``` ## Related skills - `getitune-exporting-a-model` — produce the OpenVINO `.xml` to quantize. - `getitune-running-inference` — run inference with the quantized model.