| name | turboquant-mlx |
| description | TurboQuant — compression KV-cache pour inférence MLX sur Apple Silicon (3-5× via rotation PolarQuant + quantization, Google Research mars 2026). 6 chemins d'intégration (mlx-vlm, mlx-openai-server, mlx-optiq, llama.cpp, SwiftLM, helgklaizar v1). Patch MoE-aware Qwen3.5. Déployé dans ~/ai-servers/. Charge on-demand sur fichiers turboquant ou servers.yaml. |
| paths | ["~/ai-servers/**","**/servers.yaml","**/*turboquant*"] |
TurboQuant MLX — KV-Cache Compression (rule on-demand)
Provenance : transféré du skill intellisoins-mlx:turboquant-mlx (SKILL.md) le 2026-05-24. Pas de references upstream (fichier unique). Pour TurboQuant intégré au plugin Metal, voir aussi ~/.claude/rules/vllm-metal.md. Setup local : ~/.claude/rules/local-ai-stack.md.
Google Research (March 25, 2026) — training-free, data-oblivious vector quantization
compressing KV-cache to 1-3 bits per value. Core algorithm: PolarQuant (Cartesian-to-Polar
transformation for unbiased dot-product estimation). Up to 5.3x memory reduction, quality preserved.
Note (2026-04-10): QJL (Quantized Johnson-Lindenstrauss error correction) a été abandonné
dans la v1 flagship après validation communautaire — introduit de la variance qui dégrade
la qualité softmax. PolarQuant seul donne de meilleurs résultats.
Paper: "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate"
(Zandieh, Daliri, Hadian, Mirrokni — Google Research). arXiv:2504.19874. Accepted ICLR 2026.
Repos & Integration Paths
| Path | Tool | Method | Best for |
|---|
| A | mlx-vlm (v0.4.4+) | --kv-bits 3.5 --kv-quant-scheme turboquant | Multimodal (vision), quick setup |
| B | mlx-openai-server + monkey-patch | turboquant_patch.py → TurboQuantKVCacheV2 | Production servers, aictl |
| C | mlx-optiq | pip install mlx-optiq + Python API | Standalone library |
| D | llama.cpp (fork) | --cache-type-k tbq3_0 --cache-type-v tbq3_0 | GGUF models |
| E | SwiftLM | Native Swift, OpenAI-compatible | Swift apps |
| F | helgklaizar v1 flagship | apply_turboquant_cache() drop-in mlx_lm + Asymmetric K/V | Python standalone, pip install -e ., PolarQuant pur |
| G | arozanov v2 | MixedQuantKVCache(k_bits=8, v_bits=4) | Mixed-precision, custom fork of mlx-lm |
Source repos
| Repo | Focus | API |
|---|
| helgklaizar/turboquant-mlx (hyphen, v1 flagship, actif) | PolarQuant pur, Asymmetric K/V, Boundary V, Attention Sink, OpenAI server, EXO | apply_turboquant_cache(model, k_theta_bits, v_theta_bits, fp16_sink_size) |
| helgklaizar/turboquant_mlx (underscore, legacy) | 1-3 bit V1/V2/V3 caches, QJL inclus | TurboQuantKVCacheV2(...) + turboquant.patch.apply() |
| rachittshah/mlx-turboquant | Clean standalone PoC, fractional bits (2, 3, 3.5, 4), bench suite | Standalone |
| flovflo/turboquant-mlx-qwen35-kv | Qwen3.5-35B-A3B specific, MoE-aware, benchmarked | Modèle HF |
| arozanov/mlx-lm | Mixed-Precision KV cache (fork of mlx-lm) | MixedQuantKVCache(k_bits, v_bits) / VOnlyTurboQuantCache |
Dernier push v1 flagship: 2026-04-10 — "ECO-MLX Hub integration, src-layout migration".
When to Use
- Long-context inference where KV-cache becomes the memory bottleneck
- Models with full attention layers (100% KV-cache): Gemma 3, Qwen3, Llama, Mistral
- NOT beneficial for hybrid architectures (GatedDeltaNet, Mamba) where <30% of layers use KV-cache
Architecture Compatibility
TurboQuant compresses standard scaled dot-product attention KV-caches.
MoE/hybrid models require selective layer patching (only KVCache layers, not ArraysCache/SSM layers).
| Architecture | KV-cache layers | Benefit | Patch mode |
|---|
| Gemma 4 E2B / E4B (dense + PLE) | 100% | High | All layers |
| Gemma 4 26B-A4B (MoE) | Varies (inspect) | High | MoE-aware (selective) |
| Gemma 4 31B (dense) | 100% | High | All layers |
| Gemma 3 / MedGemma (full attention) | 40/40 (100%) | High | All layers |
| Qwen3 dense (8B, 30B-A3B) | 100% | High | All layers |
| Llama, Mistral, Phi | 100% | High | All layers |
| Qwen3.5-35B-A3B (DeltaNet + Attention) | 10/40 (25%) | Significant | MoE-aware (see below) |
| Nemotron-H (Mamba2 + MoE + Attention) | ~12% | Marginal | MoE-aware |
Qwen3.5 MoE — Mixed Attention Architecture
Qwen3.5-35B-A3B has full_attention_interval=4: 10 full-attention layers use KVCache,
30 GatedDeltaNet layers use ArraysCache(size=2) (conv + SSM state, NOT key-value pairs).
Critical: replacing ALL 40 layers with TurboQuantKVCacheV2 crashes — ArraysCache uses
__getitem__/__setitem__, incompatible with update_and_fetch().
The model defines make_cache() returning a heterogeneous list:
def make_cache(self):
return [ArraysCache(size=2) if l.is_linear else KVCache() for l in self.layers]
MoE-aware patch — replace only KVCache instances:
if hasattr(model, "make_cache"):
from mlx_lm.models.cache import KVCache
caches = model.make_cache()
return [
TurboQuantKVCacheV2(head_dim=head_dim, bits=bits,
group_size=group_size, seed=42 + i)
if isinstance(c, KVCache) else c
for i, c in enumerate(caches)
]
Despite only 10/40 layers compressed, benchmarks show significant gains at longer contexts
because the full-attention layers dominate memory at scale.
Inspect model layer types
import mlx_lm
model, _ = mlx_lm.load("mlx-community/model-name")
if hasattr(model, "make_cache"):
from mlx_lm.models.cache import KVCache, ArraysCache
caches = model.make_cache()
kv = sum(1 for c in caches if isinstance(c, KVCache))
other = len(caches) - kv
print(f"KVCache: {kv}, Other: {other} — TurboQuant applies to {kv} layers")
else:
print(f"All {len(model.layers)} layers use KVCache — full TurboQuant benefit")
Modes
| Mode | Bits | Compression | Speed impact | Quality |
|---|
| V2 4-bit | 4 | ~3x | ~30% slower | Identical |
| V2 3.5-bit | 3.5 | ~4.6x | ~30% slower | Very close (recommended) |
| V2 3-bit | 3 | ~4x | ~30% slower | Very close |
V3 (Lloyd-Max, 3.5-bit) exists but is 5-6x slower — not recommended for server use.
Fractional bits (3.5): asymmetric quantization — 3-bit for Keys + 4-bit for Values.
Superior to uniform allocation at the same total bit budget. Best quality/compression ratio.
Quantization de K (Keys) vs V (Values) : Constat Clé
- Quantization de K : dégrade fortement le greedy decode en dessous de 4 bits. Les Keys déterminant les poids d'attention, le Softmax y est extrêmement sensible.
- Quantization de V : très robuste jusqu'à 3 bits, car l'interpolation pondérée du produit avec les valeurs est résiliente aux petites perturbations.
- Recommandation : privilégier la quantization mixte avec K en 8-bit et V en 4-bit (ou K=8, V=3 dans la version asymétrique de helgklaizar).
Benchmarks
Gemma 4 31B (dense, 128K context)
| Metric | Without TurboQuant (FP16 KV) | TurboQuant 3.5-bit | Delta |
|---|
| KV Memory | 13.3 GB | 4.9 GB | -63% |
| Peak Memory | 75.2 GB | 65.8 GB | -9.4 GB |
| Quality | Baseline | Preserved | ~0 loss |
Context window impact (24 GB Mac, model 4-bit ~18 GB):
- Without TurboQuant: ~16K tokens max before OOM
- With TurboQuant 3.5-bit: ~100K+ tokens viable
MedGemma 27B (dense, 200 tokens gen)
| Config | tok/s | Cache size | Compression |
|---|
| Standard fp16 | 20.9 | 144 MB | 1x |
| TQ V2 4-bit | 14.2 | 39 MB | 3.1x |
| TQ V2 3-bit | 14.3 | 32 MB | 3.8x |
KV-cache per token (MedGemma 27B fp16): ~496 KB. At 16K context: 8.3 GB → ~2.1 GB at 3-bit.
Qwen3.5-35B-A3B (MoE, 2048 prompt / 8 gen tokens, 3 trials)
| Backend | Prompt tok/s | Gen tok/s | Gen wall (s) | Cache |
|---|
| Baseline | 514.34 | 35.67 | 5.67 | 80.12 MB |
| mlx_quant | 516.13 | 38.30 | 5.16 | 44.77 MB |
| TurboQuant 3-bit | 679.14 | 44.83 | 4.20 | 45.10 MB |
vs Baseline: +32% prompt, +26% decode, -26% wall time, -44% cache memory.
At shorter contexts (128 tokens), gains are primarily memory (-12% cache) with minimal speed difference.
TurboQuant shines at 1K+ context tokens where KV-cache becomes the bottleneck.
Path F: helgklaizar v1 Flagship — Drop-in mlx-lm (simplest Python)
Repo helgklaizar/turboquant-mlx (2026-04-10).
Plug direct dans mlx_lm via 2 lignes, sans modifier la logique interne. PolarQuant pur (pas de QJL).
Features exclusives v1
- Asymmetric K/V: keys précis (
k_theta_bits=8), values agressivement compressés (v_theta_bits=3)
- Boundary V: premières 2 et dernières 2 couches laissées non-compressées (récupère ~90% précision perdue)
- Attention Sink: premiers N tokens (ex: 128) en fp16 pour préserver instruction-following
- Dynamic Chunking Buffer: cache par segments de 64 tokens (VRAM minimal en longue génération)
- EXO Cluster Ready: compatible inférence décentralisée Apple Silicon
Installation
git clone https://github.com/helgklaizar/turboquant-mlx.git
cd turboquant-mlx
pip install -e .
Usage Python
from mlx_lm import load
from turboquant_mlx.plugins.cache_plugin import apply_turboquant_cache
model, tokenizer = load("mlx-community/Meta-Llama-3-8B-Instruct-4bit")
apply_turboquant_cache(model, k_theta_bits=8, v_theta_bits=3, fp16_sink_size=128)
Serveur OpenAI-compatible
python scripts/run_server.py --model mlx-community/Meta-Llama-3-8B-Instruct-4bit --port 8080
Compatible avec Jan, Chatbox, Open WebUI (pointer vers http://localhost:8080/v1).
Non compatible LM Studio (runtime bundled, pas d'accès à la couche mlx_lm cache).
Compatibilité validée (Needle-in-a-Haystack 100%)
| Modèle | 3-bit | Notes |
|---|
| DeepSeek-R1-Distill-8B-4bit | ✅ | Reasoning complexe préservé |
| Mistral-Nemo-12B-4bit | ✅ | Layout robuste pour PolarQuant |
| Llama-3-8B / 3.2-1B | ✅ | Dot-product flawless |
| Qwen2.5-7B / Qwen3.5-35B-A3B | ✅ | Via lazy KV init (testé M4 Max 64GB) |
| gemma-2-2b-it-4bit | ❌ | Embeddings natifs clashent avec transfo polar 3-bit |
Mémoire Llama 3 8B (FP16 vs TurboQuant 3-bit)
| Contexte | FP16 | TurboQuant 3-bit | Économie | Compression |
|---|
| 4K | 64 MB | 12 MB | 81% | 5.3x |
| 16K | 256 MB | 48 MB | 81% | 5.3x |
| 64K | 1024 MB | 192 MB | 81% | 5.3x |
| 128K | 2048 MB | 384 MB | 81% | 5.3x |
Path G: arozanov v2 Mixed-Precision (mlx-lm fork)
Repo arozanov/mlx-lm (fork).
Utilise MixedQuantKVCache pour implémenter une quantization à précision mixte au niveau du cache KV.
Installation
git clone https://github.com/arozanov/mlx-lm.git
cd mlx-lm
pip install -e .
Usage Python (MixedQuantKVCache)
from mlx_lm import load
from mlx_lm.models.cache import MixedQuantKVCache
model, tokenizer = load(
"mlx-community/Meta-Llama-3-8B-Instruct-4bit",
cache_class=MixedQuantKVCache,
cache_kwargs={"k_bits": 8, "v_bits": 4}
)
Option V-Only TurboQuant Cache (compatible stock mlx-lm)
Alternative de arozanov compatible avec la version stock (officielle) de mlx-lm (sans fork).
- Configuration : K (Keys) est conservé en fp16 (aucune dégradation du softmax) et V (Values) est quantisé en PolarQuant 3-bit.
- Bénéfice : Zéro perte sur la précision d'attention tout en réduisant de ~50% la taille globale du cache KV.
from mlx_lm import load
from mlx_lm.models.cache import VOnlyTurboQuantCache
model, tokenizer = load(
"mlx-community/Meta-Llama-3-8B-Instruct-4bit",
cache_class=VOnlyTurboQuantCache,
cache_kwargs={"v_bits": 3}
)
Path A: mlx-vlm Native Integration (simplest)
mlx-vlm v0.4.4+ includes TurboQuant with custom Metal kernels (fused quantize/dequantize).
pip install mlx-vlm>=0.4.4
Optimisations Metal
- Butterfly-pulled-out optimization : 4.5x speedup sur V-attention en factorisant la WHT (Walsh-Hadamard Transform).
- SIMD-group reductions : 1.85x speedup via
simd_sum au lieu d'une réduction en arbre.
- Flash-attention scaffold : fused SDPA (Scaled Dot-Product Attention) direct sur cache compressé.
Generate (text + vision)
mlx_vlm.generate \
--model mlx-community/gemma-4-e4b-it-8bit \
--image ordonnance.jpg \
--prompt "Extrais les médicaments" \
--kv-bits 3.5 \
--kv-quant-scheme turboquant
Server (OpenAI-compatible)
mlx_vlm.server \
--model mlx-community/gemma-4-e4b-it-8bit \
--port 8080 \
--kv-bits 3.5 \
--kv-quant-scheme turboquant
Endpoint: http://localhost:8080/v1/chat/completions
Gemma 4 models for mlx-vlm
mlx-community/gemma-4-e2b-it-8bit
mlx-community/gemma-4-e4b-it-8bit
unsloth/gemma-4-E4B-it-UD-MLX-4bit
unsloth/gemma-4-26B-A4B-it-UD-MLX-4bit
unsloth/gemma-4-31B-it-UD-MLX-4bit
Path B: mlx-openai-server Monkey-Patch (production)
Installation
Source package (no pip). Copy into the target venv:
git clone --depth 1 https://github.com/sharpner/turboquant-mlx.git /tmp/turboquant-mlx
cp -r /tmp/turboquant-mlx/turboquant <venv>/lib/python3.12/site-packages/turboquant
Dependencies: mlx + mlx-lm (already in MLX inference venvs).
Path B (cont.): Server Integration (mlx-openai-server)
Two files in ~/ai-servers/:
turboquant_patch.py
Monkey-patches make_prompt_cache → TurboQuantKVCacheV2 and applies SDPA dispatch.
import turboquant_patch
turboquant_patch.apply(bits=3)
Env vars: TURBOQUANT_BITS (default 3), TURBOQUANT_GROUP_SIZE (default 64), TURBOQUANT_DISABLED=1 to bypass.
launchers/turboquant_server.py
Wrapper that resolves app module namespace collision, patches, then launches the server CLI.
Launcher example
export TURBOQUANT_BITS=3
exec python ~/ai-servers/launchers/turboquant_server.py launch \
--model-path mlx-community/model-name \
--port 8080 --host 127.0.0.1 ...
LRU Prompt Cache compatibility confirmed: update_and_fetch, deepcopy, trim, is_trimmable, nbytes all work.
Standalone Benchmark
Dense models (MedGemma, Llama, Qwen3 dense)
import mlx_lm
from turboquant.cache_v2 import TurboQuantKVCacheV2
import turboquant.patch as tq_patch
tq_patch.apply()
model, tokenizer = mlx_lm.load("mlx-community/model-name")
head_dim = model.args.head_dim
tq_cache = [
TurboQuantKVCacheV2(head_dim=head_dim, bits=3, group_size=64, seed=42 + i)
for i in range(len(model.layers))
]
MoE/hybrid models (Qwen3.5-35B-A3B)
import mlx_lm
from mlx_lm.models.cache import KVCache
from turboquant.cache_v2 import TurboQuantKVCacheV2
import turboquant.patch as tq_patch
tq_patch.apply()
model, tokenizer = mlx_lm.load("mlx-community/Qwen3.5-35B-A3B-4bit")
head_dim = model.args.head_dim
bits, group_size = 3, 64
caches = model.make_cache()
tq_cache = [
TurboQuantKVCacheV2(head_dim=head_dim, bits=bits, group_size=group_size, seed=42 + i)
if isinstance(c, KVCache) else c
for i, c in enumerate(caches)
]
Local Setup (Michael's ai-servers)
Files: ~/ai-servers/turboquant_patch.py (monkey-patch), ~/ai-servers/launchers/turboquant_server.py (wrapper).
Package installed in: ~/.venvs/mlx-qwen35/lib/python3.12/site-packages/turboquant/ (V1, V2, V3 caches).
Server management: aictl start|stop|restart|logs <server-name>.
| Server | Port | TurboQuant | Config |
|---|
| MedGemma 27B | 8080 | Active (V2 3-bit) | servers.yaml → launchers/turboquant_server.py |
| Qwen3.5-35B-A3B | 8087 | Pending MoE patch | servers.yaml → launchers/qwen35-35b.sh |
To enable on Qwen3.5: modify turboquant_patch.py for MoE-aware detection (see Architecture Compatibility),
then switch qwen35-35b.sh to use turboquant_server.py.
Troubleshooting
| Issue | Cause | Fix |
|---|
AttributeError: __setitem__ on MoE | All layers replaced with TurboQuantKVCacheV2 | Use MoE-aware patch (selective KVCache replacement) |
AttributeError: self_attn | Hybrid model without standard attention | Check compatibility table |
app module not found | Namespace collision | Use turboquant_server.py wrapper |
| No memory gain | <30% full attention layers | Model may not benefit enough |
| Quality degradation | 3-bit too aggressive | Try TURBOQUANT_BITS=4 |
| Keepalive restart loop | Server crash on model load | aictl stop <name> immediately, check logs |
