server-inference

# Server Inference Use this workflow for the FastAPI server and API-compatible inference. ## First Checks 1. Identify the server command, model, port, request endpoint, request body, and expected response. 2. Start with health/model-list checks before debugging generation. 3. Separate server startup failures from request-handling failures. 4. Keep streaming and non-streaming repros separate. ## Startup ```bash uv run mlx_vlm.server \ --model <model-or-path> \ --port 8080 ``` Useful startup flags include `--adapter-path`, `--trust-remote-code`, `--log-level`, `--enable-thinking`, `--thinking-budget`, `--draft-model`, `--draft-kind`, `--kv-bits`, `--kv-quant-scheme`, `--max-kv-size`, and `--vision-cache-size`. ## Minimal Checks ```bash curl http://127.0.0.1:8080/health curl http://127.0.0.1:8080/v1/models curl http://127.0.0.1:8080/metrics ``` Minimal chat request: ```bash curl -s http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "<model-or-path>", "messages": [{"role": "user", "content": "Say hello."}], "max_tokens": 32, "stream": false }' ``` Minimal Responses API request: ```bash curl -s http://127.0.0.1:8080/v1/responses \ -H 'Content-Type: application/json' \ -d '{ "model": "<model-or-path>", "input": "Say hello.", "max_output_tokens": 32 }' ``` ## Endpoint Surface (beyond chat/responses) The server exposes more than chat — cover or route to these as needed: - **Anthropic Messages API:** `/v1/messages`, plus `/v1/messages/count` for token counting. - **Audio:** `/v1/audio/speech` (TTS), `/v1/audio/transcriptions` and `/v1/audio/translations` (STT). - **Images:** `/v1/images/generations` and `/v1/images/edits` (diffusion image models). - **Cache & metrics:** `/v1/cache/stats`, `/v1/cache/reset`, `/v1/metrics`. - **Models:** `/v1/models` lists loaded models and model candidates in the shared Hugging Face cache, with a `loaded` boolean indicating which are in memory — see `Skill("mlx-vlm-skills:hf-cache-models")`. Match the model kind to the endpoint (an image endpoint needs a diffusion/image model, audio endpoints need an audio model); a mismatch returns a clear 4xx, not a crash. ## Debugging Rules - Capture the server command, server logs, request body, response status, and response body. - Test `/v1/models` after preload/load changes. - For OpenAI client issues, reproduce with `curl` before blaming the client SDK. - For streaming bugs, save raw event chunks and compare with non-streaming. - For structured outputs, isolate the JSON schema and confirm whether the same request works without schema constraints. - For tool calls, inspect native support and the parser inferred from the loaded processor's actual chat template. Follow [Tool Calling](../add-new-model/SKILL.md#tool-calling) for parser registration and validation. ## Validation - For route/schema changes, run `uv run pytest mlx_vlm/tests/test_server.py -q`. - For structured output changes, include `uv run pytest mlx_vlm/tests/test_generate.py -q -k 'thinking_aware or json_schema'`. - For tool parser changes, add compact regressions to `TestProcessToolCalls` in `mlx_vlm/tests/test_server.py` and run the shared tool streaming tests in that module, plus any existing tests for affected parsers. Reuse consolidated tests instead of creating a new test file. - If the result is a user-facing bug report, switch to `Skill("mlx-vlm-skills:reproducible-github-issues")`.