# LiteRT-LM
LiteRT-LM is Google's **production-ready** orchestration layer to run LLMs with
LiteRT, engineered for **high-performance**, **cross-platform** execution.
๐ [Product Website](https://ai.google.dev/edge/litert-lm) | ๐ฌโจ
[Chat Demo](https://google-ai-edge.github.io/LiteRT-LM/web_demos/chat/index.html) | ๐โจ [Embedding Demo](https://google-ai-edge.github.io/LiteRT-LM/web_demos/embedding_search/index.html)
## ๐ฅ What's New: v0.18.0
- **โจ New Model Release (EmbeddingGemma 2):** Shipped multimodal EmbeddingGemma 2
supporting text, vision, and audio embeddings with Matryoshka dimension
truncation across Python, Kotlin, Swift, Web (JavaScript), and C++. Check
out our
[DevSite documentation](https://developers.google.com/edge/litert-lm/embedding_models)
for more details!
- **๐ ๏ธ CLI & Developer Experience:** Added fast model imports (`litert-lm import`)
and an OpenAI-compatible `/v1/embeddings` endpoint (`litert-lm serve`) for
multimodal embedding models.
- **๐ Model Info API:** Added `ModelInfo` and `litert-lm describe` for full
model introspectionโcovering metadata, capabilities, and runtime
requirements before loading.
- **โก NPU & GPU Acceleration:** Enabled dynamic on-demand KV cache growth on
NPU and attention mask pruning optimizations on GPU.
๐ Try Gemma4-E4B with MTP on Linux, macOS, Windows or Raspberry Pi with the
[LiteRT-LM CLI](https://ai.google.dev/edge/litert-lm/cli):
```bash
litert-lm run \
--from-huggingface-repo=litert-community/gemma-4-E4B-it-litert-lm \
gemma-4-E4B-it.litertlm \
--backend=gpu \
--enable-speculative-decoding=true \
--prompt="What is the capital of France?"
```
## ๐ Key Features
- ๐ฑ **Cross-Platform Support**: Android, iOS, Web, Desktop, and IoT (e.g.
Raspberry Pi).
- ๐ **Hardware Acceleration**: Peak performance via GPU and NPU accelerators.
- ๐๏ธ **Multi-Modality**: Support for vision and audio inputs.
- ๐ง **Tool Use**: Function calling support for agentic workflows.
- ๐ **Broad Model Support**: Gemma, Llama, Phi-4, Qwen, and more.
--------------------------------------------------------------------------------
## ๐ Production-Ready for Google's Products
LiteRT-LM powers on-device GenAI experiences in **Chrome**, **Chromebook Plus**,
**Pixel Watch**, and more.
You can also try the
[Google AI Edge Gallery](https://github.com/google-ai-edge/gallery) app to run
models immediately on your device.
**Install the app today from Google Play** | **Install the app today from App Store**
:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------:
<a href='https://play.google.com/store/apps/details?id=com.google.ai.edge.gallery'><img alt='Get it on Google Play' height="120" src='https://play.google.com/intl/en_us/badges/static/images/badges/en_badge_web_generic.png'/></a> | <a href="https://apps.apple.com/us/app/google-ai-edge-gallery/id6749645337?itscg=30200&itsct=apps_box_badge&mttnsubad=6749645337" style="display: inline-block;"> <img src="https://toolbox.marketingtools.apple.com/api/v2/badges/download-on-the-app-store/black/en-us?releaseDate=1771977600" alt="Download on the App Store" style="width: 246px; height: 90px; vertical-align: middle; object-fit: contain;" /></a>
### ๐ฐ Blogs & Announcements
Link | Description
:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------
[Bringing Gemma 4 12B to your Laptop: Unlocking Local, Agentic Workflows with Google AI Edge](https://developers.googleblog.com/bringing-gemma-4-12b-to-your-laptop-unlocking-local-agentic-workflows-with-google-ai-edge/) | Bring agentic, multimodal AI capabilities to everyday laptops, enabling local data processing and visual insight generation.
[Blazing-fast on-device GenAI with LiteRT-LM](https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm/) | Unlock Gemma 4's full potential with blazing speed and incredible efficiency using newly added Swift, JavaScript, and Flutter APIs.
[Accelerating Gemma 4: faster inference with multi-token prediction drafters](https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/) | An overview of how Multi-Token Prediction (MTP) drafters are making Gemma 4 models up to 3x faster at inference.
[Bring state-of-the-art agentic skills to the edge with Gemma 4](https://developers.googleblog.com/bring-state-of-the-art-agentic-skills-to-the-edge-with-gemma-4/) | Deploy Gemma 4 in-app and across a broader range of devices with stellar performance and broad reach using LiteRT-LM.
[On-device GenAI in Chrome, Chromebook Plus and Pixel Watch](https://developers.googleblog.com/on-device-genai-in-chrome-chromebook-plus-and-pixel-watch-with-litert-lm/) | Deploy language models on wearables and browser-based platforms using LiteRT-LM at scale.
[On-device Function Calling in Google AI Edge Gallery](https://developers.googleblog.com/on-device-function-calling-in-google-ai-edge-gallery/) | Explore how to fine-tune FunctionGemma and enable function calling capabilities powered by LiteRT-LM Tool Use APIs.
[Google AI Edge small language models, multimodality, and function calling](https://developers.googleblog.com/google-ai-edge-small-language-models-multimodality-rag-function-calling/) | Latest insights on RAG, multimodality, and function calling for edge language models.
--------------------------------------------------------------------------------
## ๐ Quick Start
### ๐ Key Links
- ๐ [Technical Overview](https://ai.google.dev/edge/litert-lm/overview)
including performance benchmarks, model support, and more.
- ๐ [LiteRT-LM CLI Guide](https://ai.google.dev/edge/litert-lm/cli) including
installation, getting started, and advanced usage.
### โก Quick Try (No Code)
Try LiteRT-LM immediately from your terminal without writing a single line of
code using [`uv`](https://docs.astral.sh/uv/getting-started/installation/):
```bash
uv tool install litert-lm
litert-lm run \
--from-huggingface-repo=google/gemma-3n-E2B-it-litert-lm \
gemma-3n-E2B-it-int4 \
--prompt="What is the capital of France?"
```
--------------------------------------------------------------------------------
### ๐ Supported Language APIs
Ready to get started? Explore our language-specific guides and setup
instructions.
Language | Status | Best For... | Documentation
:------------------- | :-------------- | :---------------------- | :------------
**Python** | โ
Stable | Prototyping & Scripting | [Python Guide](https://ai.google.dev/edge/litert-lm/python)
**Kotlin** | โ
Stable | Android apps & JVM | [Kotlin Guide](https://ai.google.dev/edge/litert-lm/android)
**Swift** | ๐ Early Preview | Native iOS & macOS | [Swift Guide](https://ai.google.dev/edge/litert-lm/swift)
**JavaScript (web)** | ๐ Early Preview | Browser environments | [JavaScript Guide](https://ai.google.dev/edge/litert-lm/js)
**Flutter** | ๐ Community | Cross-platform mobile | [Flutter Guide](https://ai.google.dev/edge/litert-lm/flutter)
**C++** | โ
Stable | High-performance native | [C++ Guide](https://ai.google.dev/edge/litert-lm/cpp)
#### ๐๏ธ Build From Source
This [guide](./docs/getting-started/build-and-run.md) shows how you can compile
LiteRT-LM from source. If you want to build the program from source, you should
checkout the stable
[](https://github.com/google-ai-edge/LiteRT-LM/releases/latest)
tag.
--------------------------------------------------------------------------------