mlx-serving

name	mlx-serving
version	1.0.0
description	This skill should be used when the user asks about "MLX serving", "mlx_lm.server", "oMLX", "Apple Silicon LLM serving", or "local LLM on Mac" — and when troubleshooting symptoms like model fails to load, OOM during load or inference, server hangs or crashes at batch>1, tool calls returning as plaintext content, throughput regression, or choosing between mlx-lm and oMLX. Also applies to oMLX feature-flag tuning ("turboquant_kv", "dflash", "MTP", "specprefill", "thinking_budget", "max-concurrent-requests", "force_sampling"), OptiQ proxy for models exceeding RAM, Llama-4 ChunkedKVCache batch handling, Llama-3 tool-call JSON format ("name"/"parameters"), and bench-driven validation of serving configs. For Apple Silicon (M-series) only — not for cloud LLM hosting (Bedrock, OpenAI API, Anthropic API), not for non-MLX backends (llama.cpp, Ollama, vLLM), not for model training.

Apple Silicon LLM Serving (MLX / oMLX)

Operational guidance for running and tuning local LLM inference on Apple Silicon (M-series). Covers two common backends: Apple's mlx-lm and the oMLX server built on top of MLX.

Both expose OpenAI-compatible HTTP APIs, so client code is portable. The differences are in features, packaging, and the failure modes you hit when pushing models near the limits of unified memory.

---

Pre-Flight: Verify Environment

Before debugging any serving issue, confirm the basics:

# Hardware: confirm Apple Silicon and unified-memory size
system_profiler SPHardwareDataType | grep -E "Model|Chip|Memory"

# Activate the venv where mlx-lm is installed before the next two probes,
# otherwise they hit system Python and may report nothing useful.
python -c 
python -c 


omlx --
omlx diagnose menubar   


lsof -nP -iTCP -sTCP:LISTEN | grep -E