<div align="center">
<p align="center">
<img src="https://raw.githubusercontent.com/LMCache/LMCache-Ascend/main/docs/logos/lmcache-ascend-logo.png" width="720" alt="lmcache-ascend logo">
</p>
<h3 align="center">
LMCache-Ascend Plugin
</h3>
[](https://github.com/LMCache/LMCache-Ascend/actions/workflows/code-quality.yml)
[](https://deepwiki.com/LMCache/LMCache-Ascend)
<br />
<p align="center">
| <a href="https://www.hiascend.com/en/"><b>About Ascend</b></a>
| <a href="https://blog.lmcache.ai/"><b> LMCache Blog</b></a>
| <a href="https://docs.lmcache.ai/"><b>Documentation</b></a>
| <a href="https://join.slack.com/t/lmcacheworkspace/shared_invite/zt-36x1m765z-8FgDA_73vcXtlZ_4XvpE6Q"><b> Slack</b></a>
| <a href="https://deepwiki.com/LMCache/LMCache-Ascend"><b>LMCache-Ascend Wiki</b></a>
</p>
</div>
--------------------------------------------------------------------------------
## Overview
LMCache-Ascend is a community maintained plugin for running LMCache on the Ascend NPU.
> **Maintenance notice**: The stable in-process version (`LMCacheAscendConnectorV1Dynamic`) is maintained on the [`dsv4_support_045`](https://github.com/LMCache/LMCache-Ascend/tree/dsv4_support_045) branch and is **tightly coupled with LMCache 0.4.5** — pair it with the LMCache 0.4.5 release line, not newer versions. The `main` branch moves to MP mode (`LMCacheMPConnector`) and will no longer maintain the in-process mode.
## Prerequisites
To use LMCache-Ascend on the NPU hardware, please make sure the following prerequisites are satisfied.
- **Hardware**: Atlas A2/A3 series (supported); Atlas A5 series (planned support); Atlas 300I Duo (not supported).
- **OS**: Linux-based.
- **Software**:
- **Python**: >= 3.10
- **CANN Toolkit**: >= 8.5.0
- **Ascend Driver**: >= 25.5
- **PyTorch**: >= 2.8.0
- **vLLM**: >= v0.23.0 & **vLLM-Ascend**: >= v0.23.0
- **Container preparation**: see the official [vLLM-Ascend tutorials](https://docs.vllm.com.cn/projects/ascend/en/latest/tutorials/models/index.html) for preparing the base environment (NPU driver, CANN toolkit, and container images).
### Compatibility Matrix
Please ensure your environment matches the versions below.
| LMCache-Ascend | LMCache | vLLM Version |
| :--- | :--- | :--- |
| **main** | **dev** | **>=v0.23.0** |
| **v0.5.6** | **v0.5.6** | **>=v0.23.0** |
## Getting Started
The minimal deployment is three steps: install the two packages, start the `lmcache server`, then start vLLM with the connector.
### 1. Install
```bash
git clone https://github.com/LMCache/LMCache.git
cd LMCache
pip install -v --no-build-isolation -e .
cd ..
git clone --recurse-submodules https://github.com/LMCache/LMCache-Ascend.git
cd LMCache-Ascend
pip install -v --no-build-isolation -e .
```
> `third_party/hcomm` must match the container's CANN version — see [docs/mp_mode/lmcache_driven.md](docs/mp_mode/lmcache_driven.md) for the full installation guide (environment preparation, container startup, source installation).
### 2. Start the LMCache server
```bash
lmcache server \
--chunk-size 128 \
--l1-size-gb 50 \
--eviction-policy LRU
```
Wait for `LMCache INFO: LMCache zmq cache server is running on localhost:5555`.
`--chunk-size 128` matches the vLLM block size and suits single-KV-format models (e.g. Qwen3); DeepSeek-V4 requires `4096` — see [docs/mp_mode/lmcache_driven.md](docs/mp_mode/lmcache_driven.md) for the full deployment guide.
### 3. Start vLLM with the connector
```bash
vllm serve /path/to/Qwen3-32B \
--served-model-name qwen3 \
--max-model-len 40960 \
--max-num-batched-tokens 8192 \
--gpu-memory-utilization 0.9 \
--max-num-seqs 32 \
--tensor-parallel-size 8 \
--no-enable-prefix-caching \
--port 8900 \
--kv-transfer-config '{
"kv_connector": "LMCacheMPConnector",
"kv_role": "kv_both"
}'
```
The server address defaults to `localhost:5555` on both sides; on NPU the transfer mode defaults to `lmcache_driven`, so no extra config is needed here. A successful start shows `LMCache INFO: Creating transfer context (device_type=npu, mode=lmcache_driven)` for every worker.
For the `engine_driven` transfer mode (workers copy KV from a named SHM pool), see [docs/mp_mode/engine_driven.md](docs/mp_mode/engine_driven.md).
## Documentation
- [docs/mp_mode/README.md](docs/mp_mode/README.md) — MP mode overview.
- [docs/mp_mode/lmcache_driven.md](docs/mp_mode/lmcache_driven.md) — `lmcache_driven` full deployment guide (environment preparation, container, installation, server, vLLM, verification).
- [docs/mp_mode/engine_driven.md](docs/mp_mode/engine_driven.md) — `engine_driven` deployment guide (Qwen3 example).
- [docs/mp_mode/qwen3_5.md](docs/mp_mode/qwen3_5.md) — hybrid Mamba models (for example Qwen3.5, Qwen3.6, and Qwen3.8): same flags as the LMCache recipe, and where Ascend differs.