llm-models

# Local LLM Model Management All model management lives in `llm-manager/` at the repo root. The llm-manager is responsible for the full lifecycle: downloading, serving, and advertising models to pi inside the container. ## Key Files | File | Purpose | Committed? | |------|---------|-----------| | `llm-manager/models.conf` | Shared registry — commented-out examples only | ✅ Yes | | `llm-manager/models.local.conf` | Personal active models for this machine | ❌ No (gitignored) | | `llm-manager/models.local.conf.example` | Template to copy | ✅ Yes | | `llm-manager/models.json` | Pi provider config — auto-generated by llm.sh | ❌ No (gitignored) | | `docker/models.json.example` | Human-readable template showing the format | ✅ Yes | | `llm-manager/llm.sh` | Lifecycle manager script | ✅ Yes | | `llm-manager/models/<id>/` | Downloaded model weights | ❌ No (gitignored) | | `llm-manager/fixes/<id>.sh` | Post-download fix scripts (per model) | ✅ Yes | ## How Pi Discovers Local Models `llm-manager/models.json` is **auto-generated** by `llm.sh` on every `start` and `stop`. It contains one provider entry per currently running model server. The **`local-llm` Terteform plugin** (bundled) reads `GET /v1/models` from the running server at container startup and injects the provider config into pi automatically. If no local model is running the plugin's health check fails, the plugin is skipped, and pi starts normally with cloud providers only. **No personal model names or config ever appear in committed files.** ## The conf / local.conf Split **Rule:** `models.conf` is public — it must contain **only commented-out examples**. Active (uncommented) model entries belong exclusively in `models.local.conf`, which is git-ignored and never committed. `llm.sh` merges both files at runtime: `models.conf` first, then `models.local.conf` (if present). ## Activating a Model (adding to your machine) 1. Copy the example file if `models.local.conf` doesn't exist yet: ```bash cp llm-manager/models.local.conf.example llm-manager/models.local.conf ``` 2. Open `llm-manager/models.local.conf` and uncomment the desired model line, or add a new one using the column format: ``` # id type hf_repo port extra my-model-id mlx some-user/some-repo 8001 - ``` 3. Download and start it: ```bash make llm-download MODEL=<id> make llm-start MODEL=<id> ``` Starting generates `llm-manager/models.json` automatically. 4. Launch terte — pi will have the model available as a provider. ## Adding a New Model to the Shared Registry When a new model should be **documented** (not activated) for all users: 1. Add a commented-out entry to `llm-manager/models.conf` under the appropriate section. 2. If the model needs a post-download fix (e.g. symlinks, patching), create `llm-manager/fixes/<id>.sh`. It receives `$1 = MODEL_DIR`. 3. Commit `models.conf` (and the fix script if any). **Do not commit an uncommented entry.** ## Column Format ``` id type hf_repo port extra ``` | Column | Values | Notes | |--------|--------|-------| | `id` | any slug | No spaces — use hyphens. Must match `fixes/<id>.sh` if a fix exists. | | `type` | `mlx` \| `gguf` | Runtime to use | | `hf_repo` | `user/repo` | HuggingFace repository | | `port` | integer | Port the server listens on (must be unique). Port 8001 is the default. | | `extra` | `-` or filename | MLX: always `-`. GGUF: exact `.gguf` filename to download. | ## Make Targets ```bash make llm-list # show all registered models + download/running status make llm-download MODEL=<id> # download from HuggingFace (+ run fix script if any) make llm-start MODEL=<id> # start server + regenerate models.json make llm-stop MODEL=<id> # stop server + regenerate models.json make llm-restart MODEL=<id> # stop + start make llm-health MODEL=<id> # curl /v1/models — check server is responding make llm-status MODEL=<id> # full status: path, port, running, disk usage make llm-logs MODEL=<id> # tail live server log make llm-remove MODEL=<id> # delete downloaded weights from disk (prompts) ``` ## Deactivating a Model To stop using a model on this machine without deleting the weights: ```bash make llm-stop MODEL=<id> ``` This stops the server and regenerates `llm-manager/models.json` (removing the model from it). If no models remain running, the file is deleted entirely — terte will start pi without any local provider. To also free disk space: `make llm-remove MODEL=<id>` ## Server URLs (from inside the Docker container) ``` http://host.docker.internal:<port>/v1 ``` This bypasses the Squid proxy (host-gateway routing, stays on the Mac). ## Runtimes | Type | Binary | Install | |------|--------|---------| | `mlx` | `/opt/homebrew/opt/mlx-lm/bin/mlx_lm.server` | `brew install mlx-lm` | | `gguf` | `/opt/homebrew/bin/llama-server` | `brew install llama.cpp` |