litert-runtime

This skill provides step-by-step guidance for building an Android app that runs a `.tflite` model with the [LiteRT CompiledModel API](https://ai.google.dev/edge/litert/android) (`com.google.ai.edge.litert:litert` 2.x; [source repository](https://github.com/google-ai-edge/litert/blob/main/README.md)) on the CPU or the GPU, with the runtime bundled in the APK or from Google Play services. Don't use the deprecated `org.tensorflow.lite.Interpreter` API. For open text language models using `.litertlm` files, suggest the `litert-lm` skill. ## Prerequisites - A Kotlin Android project (`minSdk` must be `24` or higher, as declared by the LiteRT 2.3.0 AARs). - The runtime that Step 1 picks: the Google Play services runtime (`com.google.android.gms:play-services-litert`, the default; see [LiteRT in Google Play services](references/play-services.md)), or the bundled runtime (`implementation("com.google.ai.edge.litert:litert:2.3.0")` and `implementation("com.google.ai.edge.litert:litert-gpu:2.3.0")` from Google Maven). For the bundled runtime, from 2.3.0 the GPU accelerator is the second artifact: with `litert` alone the app builds, and `CompiledModel.create` with `CompiledModel.Options(Accelerator.GPU)` throws `LiteRtException` because the APK has no GPU accelerator library. With `CompiledModel.Options(Accelerator.GPU, Accelerator.CPU)` it doesn't throw: the model runs on the CPU while the app reports the GPU, and only Logcat shows it (`GPU accelerator could not be loaded and registered`). No manifest entry is needed: `litert-api`, which `litert` depends on, declares the OpenCL driver libraries (`uses-native-library`) in its own manifest. - A verified `.tflite` model and its input contract (input dimensions, mean, standard deviation, channel order, and tensor layout). For models with a verified recipe, inspect the [LiteRT model recipes](https://github.com/google-ai-edge/litert-samples/blob/main/models/README.md). ## Critical mandates - **Single-thread confinement** : One single-thread executor or confined coroutine dispatcher must own the `CompiledModel` instance. Creation, inference, and closing must execute on that same background thread, never on the main thread. - **Buffer lifecycle** : Allocate `TensorBuffer` instances once using `createInputBuffers` and `createOutputBuffers`, reuse them across inferences, and close all input and output buffers before closing the `CompiledModel`. - **Synchronous timing** : `model.run` enqueues work on the GPU and returns before GPU execution completes. Always measure `model.run` and output buffer readback (`outputs[0].readFloat`) together. - **Warm-up and exception safety** : Execute one warm-up inference inside the helper constructor's `init` block so first-run GPU shader compilation doesn't block the first user interaction, and call `close` in `catch` before rethrowing so neither `CompiledModel` nor `TensorBuffer` handles leak if buffer allocation or warm-up throws. - **Runtime choice** : Before Step 1, you must read [LiteRT in Google Play services](references/play-services.md) and choose the runtime. Use the Google Play services runtime unless the app needs the NPU, custom operators, or a Support library class that this skill can't replace (such as `MetadataExtractor`), or also targets devices without Google Play services. Never put both runtimes in the same build variant. - **Migration** : When the app already uses `org.tensorflow.lite` code, you must read the "Migrating from legacy TFLite Interpreter" section in full before you edit any file. Migrated code must not import any `org.tensorflow.lite` class (including `DataType`), every legacy GPU choice (`GpuDelegate`, or the Support library's `Model.Device.GPU`) becomes `CompiledModel.Options(Accelerator.GPU, Accelerator.CPU)`, and an NNAPI choice maps to that same GPU option. ## Detailed steps ### Step 1: Configure Gradle and place the model file Search the project for `libs.versions.toml`, `gradle.properties`, `app/build.gradle.kts`, and any existing `org.tensorflow.lite` imports before editing build files. Then pick the runtime: - **Google Play services runtime (default)** : Use it unless one of the bundled runtime cases applies, also when the request doesn't name a runtime. You must read [LiteRT in Google Play services](references/play-services.md) now: its dependencies replace the ones in this step, and its `Environment` replaces the `environment` property of Step 2. Use version `17.0.0` exactly, and run `./gradlew assembleDebug` (not `--dry-run`) after you add it. Switch to the bundled runtime only when that build reports `Could not find com.google.android.gms:play-services-litert:17.0.0`, and tell the user, also when the user asked for Google Play services. Never leave the build broken. - **Bundled runtime** : Use it only when the app needs the NPU, custom operators, or a Support library class that this skill can't replace (such as `MetadataExtractor`), when the app also targets devices without Google Play services, or when the user asks for it. `FileUtil`, `ImageProcessor`, `TensorImage`, and `TensorLabel` don't count: replace them as [LiteRT in Google Play services](references/play-services.md) describes. For the bundled runtime, configure `minSdk` and add the LiteRT runtime and GPU accelerator dependencies in `app/build.gradle.kts`: android { defaultConfig { minSdk = 24 } } dependencies { implementation("com.google.ai.edge.litert:litert:2.3.0") implementation("com.google.ai.edge.litert:litert-gpu:2.3.0") } Place bundled models at `app/src/main/assets/model.tflite` (AGP stores `.tflite` files uncompressed by default). Deliver a model that exceeds the base APK size limit through [Play for On-Device AI](references/android/google/play/on-device-ai.md) as an AI pack, or download it into `context.filesDir` (`bundletool --local-testing` installs AI packs locally without Google Play). Read an install-time AI pack through `AssetManager`. Load a fast-follow or on-demand AI pack or a downloaded file from its on-disk `filePath` using `CompiledModel.create(filePath, CompiledModel.Options(accelerator), environment)` in place of the asset call in Step 2. Obtain the pack's directory from the AI Delivery library (`com.google.android.play:ai-delivery`) using `AiPackManagerFactory.getInstance(context).getPackLocation(name)?.assetsPath()`, which returns `null` until the pack has finished downloading. With 2.3.0, nothing is added to `gradle.properties`: `litert` 2.3.0 declares the namespace `com.google.ai.edge.litert.impl` and its dependency `litert-api` 2.3.0 declares `com.google.ai.edge.litert`, so AGP 9.x builds the project. With 2.2.0 or 2.1.6, both libraries declare `com.google.ai.edge.litert`, and AGP 9.x stops at `processDebugMainManifest` (AGP 8.x reports it as a warning and builds); for those two versions, add `android.uniquePackageNames=false` to `gradle.properties`, and remove it when moving to 2.3.0 because it also hides the same clash between any other two libraries. Excluding `litert-api` instead doesn't work: `CompiledModel` and `Accelerator` are in `litert-api`, and the build then fails at `compileDebugKotlin`. ### Step 2: Implement the inference helper class Isolate `CompiledModel` and its `TensorBuffer` instances in a dedicated `AutoCloseable` helper class free of Android UI types: import android.content.Context import com.google.ai.edge.litert.Accelerator import com.google.ai.edge.litert.CompiledModel import com.google.ai.edge.litert.Environment import com.google.ai.edge.litert.TensorBuffer const val INPUT_SIZE = 224 * 224 * 3 private val environment by lazy { Environment.create() } class Classifier(context: Context, accelerator: Accelerator) : AutoCloseable { private val model = CompiledModel.create( context.assets, "model.tflite", CompiledModel.Options(accelerator), environment ) private val inputs = mutableListOf<TensorBuffer>() private val outputs = mutableListOf<TensorBuffer>() init { try { inputs += model.createInputBuffers() outputs += model.createOutputBuffers() infer(FloatArray(INPUT_SIZE)) } catch (e: Exception) { close() throw e } } fun infer(input: FloatArray): FloatArray { inputs[0].writeFloat(input) model.run(inputs, outputs) return outputs[0].readFloat() } override fun close() { (inputs + outputs).forEach { it.close() } model.close() } } `Accelerator.CPU` requires nothing from the device. `Accelerator.GPU` compiles the graph for the GPU; when the GPU can't execute the model, the constructor throws `LiteRtException` (from `create` for an operator the GPU doesn't support, and on the Android emulator from `create` or from buffer allocation). A missing asset or a file that isn't a valid model also throws `LiteRtException` from `create`. The buffers are created once with the model, reused for every inference, and closed before the model. One `Environment` serves every model in the process and stays open, so LiteRT loads the GPU accelerator library once rather than on every `create` call. The constructor ends with one warm-up inference (the first GPU run includes shader compilation) and closes the model and its buffers if any step in `init` throws. `INPUT_SIZE` is the model's input element count: `writeFloat` throws on a longer array and writes a shorter one without an error, so a wrong size produces an incorrect result. For `int8` or `uint8` tensors, use `writeInt8` and `readInt8` on `TensorBuffer` instead of `writeFloat` and `readFloat`. When memory matters, use the following options: - **Load from a path** : `create(context.assets, ...)` keeps a copy of the asset's bytes in the native heap while the model is open. Copy the asset once into `context.noBackupFilesDir` (a downloaded or AI pack file is already on disk) and call `create(filePath, ...)` so the file is mapped and reclaimable. Version the staged filename, bump the version when the bundled model changes, and delete the previous version's file. Copy to a temporary file and rename it so a copy interrupted by process death is never loaded: import android.content.Context import com.google.ai.edge.litert.Accelerator import com.google.ai.edge.litert.CompiledModel import java.io.File fun stageModelAsset( context: Context, assetName: String = "model.tflite", version: String = "v1" ): File = File(context.noBackupFilesDir, "$version-$assetName").also { file -> if (!file.isFile) { context.noBackupFilesDir.listFiles()?.forEach { if (it.name.endsWith("-$assetName")) it.delete() } val tmp = File("${file.path}.tmp") context.assets.open(assetName).use { input -> tmp.outputStream().use { output -> input.copyTo(output) } } check(tmp.renameTo(file)) } } fun memoryOptions( context: Context, accelerator: Accelerator = Accelerator.CPU, assetName: String = "model.tflite", version: String = "v1" ): CompiledModel.Options = CompiledModel.Options(accelerator).apply { val prefix = "$assetName-" val cache = File(context.codeCacheDir, "$prefix$version.xnnpack_cache") context.codeCacheDir.listFiles()?.forEach { if (it.name.startsWith(prefix) && it.name.endsWith(".xnnpack_cache") && it != cache) { it.delete() } } cpuOptions = CompiledModel.CpuOptions( numThreads = 4, xnnPackFlags = 0x23, xnnPackWeightCachePath = cache.path ) } - **Weight cache (`xnnPackWeightCachePath`)**: Written on the first load (\~1x an FP32 or \~2x an FP16 model's file size) and mapped on later loads, moving packed FP32 weights off the heap. Version the cache filename and delete outdated caches. - **Transient indirection buffer (`xnnPackFlags = 0x23`)** : Default `0x3` plus flag `0x20`, replacing per-operator convolution pointer tables with one shared buffer (byte-identical outputs, \~1% latency overhead at 4 threads, \~3% single-threaded). Because the value replaces the default, `0x20` alone leaves `int8` and `uint8` operators to fallback kernels. `cpuOptions` is ignored on the GPU. - **Close after a task**: Closing frees the runtime workspace; recreating with a warm cache takes \~20 ms, whereas the first inference after any load pays a full warm-up run, so close when the next use isn't imminent. ### Step 3: Wire the ViewModel and Compose screen Drive the helper from a single-thread executor inside an `AndroidViewModel`, and fall back to `Accelerator.CPU` when GPU compilation fails: import android.app.Application import androidx.lifecycle.AndroidViewModel import com.google.ai.edge.litert.Accelerator import com.google.ai.edge.litert.LiteRtException import java.util.concurrent.Executors import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.SupervisorJob import kotlinx.coroutines.asCoroutineDispatcher import kotlinx.coroutines.cancel import kotlinx.coroutines.flow.MutableStateFlow import kotlinx.coroutines.flow.StateFlow import kotlinx.coroutines.flow.asStateFlow import kotlinx.coroutines.flow.update import kotlinx.coroutines.launch data class UiState( val ready: Boolean = false, val accelerator: Accelerator = Accelerator.CPU, val result: List<Float>? = null, val error: String? = null ) class MainViewModel(app: Application) : AndroidViewModel(app) { private val executor = Executors.newSingleThreadExecutor() private val scope = CoroutineScope( SupervisorJob() + executor.asCoroutineDispatcher() ) private var classifier: Classifier? = null private val _uiState = MutableStateFlow(UiState()) val uiState: StateFlow<UiState> = _uiState.asStateFlow() fun load(preferred: Accelerator = Accelerator.GPU) { scope.launch { if (classifier != null) return@launch val appContext = getApplication<Application>() val selected = try { classifier = Classifier(appContext, preferred) preferred } catch (e: LiteRtException) { if (preferred != Accelerator.CPU) { try { classifier = Classifier(appContext, Accelerator.CPU) Accelerator.CPU } catch (cpuError: LiteRtException) { _uiState.value = UiState(error = cpuError.message) return@launch } } else { _uiState.value = UiState(error = e.message) return@launch } } _uiState.value = UiState(ready = true, accelerator = selected) } } fun classify(input: FloatArray) { scope.launch { try { classifier?.infer(input)?.let { result -> _uiState.update { it.copy(result = result.toList(), error = null) } } } catch (e: LiteRtException) { _uiState.update { it.copy(error = e.message) } } } } override fun onCleared() { scope.launch { classifier?.close() }.invokeOnCompletion { scope.cancel() } executor.shutdown() } } One single-thread executor owns the model: creation, inference, and closing execute only on it, never on the main thread. The `ViewModel` keeps its own scope because `viewModelScope` is canceled before `onCleared` runs, so a close launched there wouldn't run; `onCleared` cancels `scope` once `close` completes. `load` is a no-op when a model is already loaded across configuration changes; a failed `load` leaves `classifier` `null`. `classify` reports any `LiteRtException` (in package `com.google.ai.edge.litert`) in `error` and keeps the model. The following Compose screen takes the picture (`bitmap`) from the photo picker or CameraX and runs `preprocess` in the click handler (a few milliseconds for 224x224; move `preprocess` onto the executor for images exceeding 224x224). `viewModel` and `collectAsStateWithLifecycle` come from `androidx.lifecycle:lifecycle-viewmodel-compose` and `androidx.lifecycle:lifecycle-runtime-compose`: import android.graphics.Bitmap import androidx.compose.foundation.layout.Column import androidx.compose.material3.Button import androidx.compose.material3.Text import androidx.compose.runtime.Composable import androidx.compose.runtime.LaunchedEffect import androidx.compose.runtime.getValue import androidx.lifecycle.compose.collectAsStateWithLifecycle import androidx.lifecycle.viewmodel.compose.viewModel import com.google.ai.edge.litert.Accelerator @Composable fun MainScreen( bitmap: Bitmap, viewModel: MainViewModel = viewModel() ) { val state by viewModel.uiState.collectAsStateWithLifecycle() LaunchedEffect(Unit) { viewModel.load(Accelerator.GPU) } Column { state.error?.let { Text(it) } Button( onClick = { viewModel.classify(preprocess(bitmap)) }, enabled = state.ready ) { Text("Run (${state.accelerator})") } state.result?.let { scores -> Text("Top class: ${scores.indices.maxBy { i -> scores[i] }}") } } } ### Step 4: Preprocess inputs to match the model contract Convert the `Bitmap` into the model's input with `preprocess`. Match the model's exact input dimensions, normalization mean and standard deviation, channel order, and memory layout (this example uses NHWC, RGB, scaled to `-1.0..1.0`). Mismatched `mean` or `std` values produce corrupted predictions that look like a broken model: import android.graphics.Bitmap fun preprocess( bitmap: Bitmap, size: Int = 224,