mlx-lm-server

name	mlx-lm-server
description	Use when working on the mlx-lm-server project - a FastAPI + React app for LLM inference using Apple's MLX framework. Covers OpenAI-compatible backend API, React frontend, MLX model loading, streaming responses, and testing.

MLX LM Server

A full-stack application for running LLM inference locally on Apple Silicon using Apple's MLX framework. FastAPI backend with OpenAI-compatible API + React frontend with real-time chat UI.

Project Structure

mlx-lm-server/
├── backend/                    # FastAPI backend (Python)
│   ├── main.py                # App entry point, CORS, router mounting
│   ├── config.py              # Settings via pydantic-settings (MLX_ env prefix)
│   ├── models.py              # Pydantic request/response schemas (OpenAI-compatible)
│   ├── routers/
│   │   └── chat.py            # All API endpoints (/v1/chat/completions, /v1/models, etc.)
│   ├── services/
│   │   └── mlx_service.py     # MLX model loading, generation, streaming
│   └── tests/                 # pytest tests
├── frontend/                   # React + TypeScript + Vite
│   ├── src/
│   │   ├── components/        # Chat, ChatInput, MessageBubble, SettingsPanel
│   │   ├── hooks/             # useChat (state management)
│   │   ├── services/          # api.ts (SSE streaming client)
│   │   └── types/             # TypeScript interfaces (OpenAI-compatible)
│   └── package.json
├── pyproject.toml             # Python dependencies (uv)
└── README.md

Running the Project

Backend (must be running for frontend to work)

cd /Users/vishu/dev/vishal/mlx-lm-server
uv run uvicorn backend.main:app --reload --port 8000

Frontend (separate terminal)

cd /Users/vishu/dev/vishal/mlx-lm-server/frontend
pnpm dev

Health Check

curl http://localhost:8000/health

Running Tests

Backend (79 tests)

cd /Users/vishu/dev/vishal/mlx-lm-server
uv run pytest -v

Frontend (48 tests)

cd /Users/vishu/dev/vishal/mlx-lm-server/frontend
npx vitest run

Run Single Test File

uv run pytest backend/tests/test_mlx_service.py -v
npx vitest run src/hooks/__tests__/useChat.test.ts

Key Code Patterns

MLX Service (backend/services/mlx_service.py)

IMPORTANT: The mlx_lm API has changed. Do NOT use temp or top_p kwargs directly.

# WRONG - will fail with "unexpected keyword argument 'temp'"
generate(model, tokenizer, prompt=prompt, temp=0.7, top_p=0.9)

# CORRECT - use make_sampler
from mlx_lm.sample_utils import make_sampler
sampler = make_sampler(temp=0.7, top_p=0.9)
generate(model, tokenizer, prompt=prompt, sampler=sampler)

Streaming: Use stream_generate() not generate(stream=True):

from mlx_lm import stream_generate
for response in stream_generate(model, tokenizer, prompt=prompt, sampler=sampler):
    yield response.text  # Each token as a string

System Messages: The _prepare_messages() method prepends system message content to the first user message. Many models (like Mistral) don't support system role directly.

OpenAI-Compatible API (backend/routers/chat.py)

The backend implements OpenAI-compatible endpoints:

Chat Completions

POST /v1/chat/completions

Request Body:

{
  "model": "mlx-community/Mistral-7B-Instruct-v0.3-4bit",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"}
  ],
  "max_tokens": 2048,
  "temperature": 0.7,
  "top_p": 0.9,
  "stream": true
}

Non-Streaming Response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "mlx-community/Mistral-7B-Instruct-v0.3-4bit",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "Hello! How can I help you?"},
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 5,
    "total_tokens": 15
  }
}

Streaming Response: The streaming response uses SSE format with data: lines:

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1234567890,"model":"mlx-community/Mistral-7B-Instruct-v0.3-4bit","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1234567890,"model":"mlx-community/Mistral-7B-Instruct-v0.3-4bit","choices":[{"index":0,"delta":{"content":" world"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1234567890,"model":"mlx-community/Mistral-7B-Instruct-v0.3-4bit","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Models

GET /v1/models
POST /v1/models/load
DELETE /v1/models/{model_id}

List Models Response:

{
  "object": "list",
  "data": [
    {
      "id": "mlx-community/Mistral-7B-Instruct-v0.3-4bit",
      "object": "model",
      "created": 1234567890,
      "owned_by": "mlx-lm-server"
    }
  ]
}

Frontend SSE Parser (frontend/src/services/api.ts)

The parser handles OpenAI streaming format:

const data = trimmedLine.slice(6);

if (data === "[DONE]") {
  return;  // Stream complete
}

const chunk = JSON.parse(data) as ChatCompletionChunk;
yield chunk;

React State (frontend/src/hooks/useChat.ts)

CRITICAL: State updates must be immutable. Do NOT mutate objects in place.

// WRONG - mutates in place, causes duplicate tokens
lastMsg.content += chunk.delta;

// CORRECT - creates new object
updated[updated.length - 1] = {
  ...lastMsg,
  content: lastMsg.content + chunk.choices[0].delta.content,
};

Empty Messages Filter

The useChat hook filters out empty assistant placeholder messages before sending to API:

...messages
  .filter((m) => m.content !== "")
  .map((m) => ({ role: m.role, content: m.content })),

API Endpoints

Method	Endpoint	Description
GET	/	Server status
GET	/health	Health check with model info
GET	/status	Server status
POST	/v1/chat/completions	OpenAI-compatible chat completions
GET	/v1/models	List loaded models (OpenAI format)
POST	/v1/models/load	Load a model
DELETE	/v1/models/{model_id}	Unload a model

Configuration

Backend (backend/config.py)

Settings use MLX_ env prefix:

• MLX_DEFAULT_MODEL - Default model ID
• MLX_HOST - Server host (default: 0.0.0.0)
• MLX_PORT - Server port (default: 8000)
• MLX_DEBUG - Debug mode

Frontend (frontend/vite.config.ts)

• Dev server on port 5173
• Proxies /v1 to backend on port 8000

Common Tasks

Adding a New API Endpoint

1. Add request/response models to backend/models.py
2. Add route handler to backend/routers/chat.py
3. Add tests to backend/tests/test_endpoints.py

Adding a New React Component

1. Create component in frontend/src/components/
2. Export from index or import directly
3. Add tests in frontend/src/components/__tests__/

Changing Model Defaults

1. Update DEFAULT_SETTINGS in frontend/src/hooks/useChat.ts
2. Update settings.default_model in backend/config.py

OpenAI SDK Compatibility

This server is compatible with OpenAI SDKs. Example usage:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"  # Local server doesn't require auth
)

response = client.chat.completions.create(
    model="mlx-community/Mistral-7B-Instruct-v0.3-4bit",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"}
    ],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Known Issues & Gotchas

1. MLX API: Use sampler=make_sampler(temp=..., top_p=...) not temp/top_p kwargs
2. System messages: Some models don't support system role - backend prepends to first user message
3. React state: Always create new objects, never mutate in place
4. SSE format: Use data: {json}\n\n with data: [DONE] terminator (OpenAI format)
5. Empty messages: Filter out empty assistant placeholders before API calls
6. pnpm builds: Run pnpm approve-builds esbuild if install fails