llm-models

name	llm-models
description	Manages local LLM models in Terteform. Use when asked to add, remove, activate, or deactivate a local model, download or start/stop a model server, register a new model in the registry, understand the models.conf / models.local.conf split, or how pi inside the container discovers local LLM servers.

Local LLM Model Management

All model management lives in llm-manager/ at the repo root. The llm-manager is responsible for the full lifecycle: downloading, serving, and advertising models to pi inside the container.

Key Files

File	Purpose	Committed?
llm-manager/models.conf	Shared registry — commented-out examples only	✅ Yes
llm-manager/models.local.conf	Personal active models for this machine	❌ No (gitignored)
llm-manager/models.local.conf.example	Template to copy	✅ Yes
llm-manager/models.json	Pi provider config — auto-generated by llm.sh	❌ No (gitignored)
docker/models.json.example	Human-readable template showing the format	✅ Yes
llm-manager/llm.sh	Lifecycle manager script	✅ Yes
llm-manager/models/<id>/	Downloaded model weights	❌ No (gitignored)
llm-manager/fixes/<id>.sh	Post-download fix scripts (per model)	✅ Yes

How Pi Discovers Local Models

llm-manager/models.json is auto-generated by llm.sh on every start and stop. It contains one provider entry per currently running model server.

The local-llm Terteform plugin (bundled) reads GET /v1/models from the running server at container startup and injects the provider config into pi automatically. If no local model is running the plugin's health check fails, the plugin is skipped, and pi starts normally with cloud providers only.

No personal model names or config ever appear in committed files.

The conf / local.conf Split

Rule: models.conf is public — it must contain only commented-out examples. Active (uncommented) model entries belong exclusively in models.local.conf, which is git-ignored and never committed.

llm.sh merges both files at runtime: models.conf first, then models.local.conf (if present).

Activating a Model (adding to your machine)

1. Copy the example file if models.local.conf doesn't exist yet:

   cp llm-manager/models.local.conf.example llm-manager/models.local.conf
   

2. Open llm-manager/models.local.conf and uncomment the desired model line, or add a new one using the column format:

   # id                              type  hf_repo                          port   extra
   my-model-id                       mlx   some-user/some-repo              8001   -
   

3. Download and start it:

   make llm-download MODEL=<id>
   make llm-start    MODEL=<id>
   

   Starting generates llm-manager/models.json automatically.

4. Launch terte — pi will have the model available as a provider.

Adding a New Model to the Shared Registry

When a new model should be documented (not activated) for all users:

1. Add a commented-out entry to llm-manager/models.conf under the appropriate section.
2. If the model needs a post-download fix (e.g. symlinks, patching), create llm-manager/fixes/<id>.sh. It receives $1 = MODEL_DIR.
3. Commit models.conf (and the fix script if any). Do not commit an uncommented entry.

Column Format

id                  type   hf_repo                        port   extra

Column	Values	Notes
id	any slug	No spaces — use hyphens. Must match fixes/<id>.sh if a fix exists.
type	mlx | gguf	Runtime to use
hf_repo	user/repo	HuggingFace repository
port	integer	Port the server listens on (must be unique). Port 8001 is the default.
extra	- or filename	MLX: always -. GGUF: exact .gguf filename to download.

Make Targets

make llm-list                           # show all registered models + download/running status
make llm-download MODEL=<id>            # download from HuggingFace (+ run fix script if any)
make llm-start    MODEL=<id>            # start server + regenerate models.json
make llm-stop     MODEL=<id>            # stop server + regenerate models.json
make llm-restart  MODEL=<id>            # stop + start
make llm-health   MODEL=<id>            # curl /v1/models — check server is responding
make llm-status   MODEL=<id>            # full status: path, port, running, disk usage
make llm-logs     MODEL=<id>            # tail live server log
make llm-remove   MODEL=<id>            # delete downloaded weights from disk (prompts)

Deactivating a Model

To stop using a model on this machine without deleting the weights:

make llm-stop MODEL=<id>

This stops the server and regenerates llm-manager/models.json (removing the model from it). If no models remain running, the file is deleted entirely — terte will start pi without any local provider.

To also free disk space: make llm-remove MODEL=<id>

Server URLs (from inside the Docker container)

http://host.docker.internal:<port>/v1

This bypasses the Squid proxy (host-gateway routing, stays on the Mac).

Runtimes

Type	Binary	Install
mlx	/opt/homebrew/opt/mlx-lm/bin/mlx_lm.server	brew install mlx-lm
gguf	/opt/homebrew/bin/llama-server	brew install llama.cpp