ollama

# Ollama — run open-weight LLMs on one box Ollama serves GGUF models from a local daemon at `http://localhost:11434`, exposing both a native HTTP API and an OpenAI-compatible layer. Your job: reach for the right command, the right endpoint, and the right quant **for the hardware in front of you** — and recognize when the model does not fit and the work belongs on a remote GPU instead. This skill owns: install/serve, pull/tag, the local API (native + OpenAI-compat), Modelfiles, quantization choice, and VRAM/RAM sizing on a single machine. ## When to use / when not **Use when** the model runs on *this* machine: pulling/running a model, fixing an OOM, choosing Q4 vs Q8, authoring a Modelfile, or wiring an app to `localhost:11434`. **Go elsewhere when:** - Hosting behind a managed/remote GPU, autoscaling, or serverless inference → `runpod`, `modal`, `replicate`, `together-fireworks`, `fal`. Ollama is local, single-box, no autoscale. - Downloading raw weights, datasets, `hf`/`transformers`, repo management → `huggingface`. - Designing chunking / retrieval / reranking around a model → `rag` or `embeddings-search`. - Orchestrating multi-step calls, routing, pipeline evals → `llm-pipeline` / `agent-eval`. - Writing the prompt/system-message *content* itself → `prompt-engineering`. (Those siblings live in the catalog by id; link them only once their `SKILL.md` exists on disk.) ## Quickstart ```bash ollama serve # start the daemon (a desktop install already runs it) ollama pull qwen3:8b # download a model + tag; :8b is explicit — avoid bare :latest ollama run qwen3:8b # interactive REPL, or: ollama run qwen3:8b "summarize this" ollama ps # what is LOADED in VRAM right now + when it unloads (keep_alive) ollama list # what is on disk (pulled), not what is loaded ollama show qwen3:8b # template, params, context length, quant of a model ollama rm qwen3:8b # free disk; ollama stop qwen3:8b unloads from memory ``` `ps` vs `list` is the OOM-debug split: `list` is disk, `ps` is memory. A model only eats VRAM once a request loads it; it unloads after `keep_alive` (default 5m). ## Pick a model + quant Quantization trades VRAM for quality. The everyday default is **Q4_K_M**: roughly half the memory of fp16 for ~3–5% quality loss. **Q8_0** is near-lossless at ~1 byte/param. **fp16** is the unquantized ceiling at 2 bytes/param. Sizing formula (weights only) — a **rule of thumb**, not a per-model spec sheet: ```text weights_GB ≈ params(B) × bytes_per_param × 1.2 # ×1.2 = runtime overhead bytes_per_param: Q4_K_M ≈ 0.5 Q8_0 ≈ 1.0 fp16 = 2.0 # then ADD the KV cache (see below) — it is NOT in this number. ``` These bytes/param are conservative round-downs of the measured k-quant rates: llama.cpp's quantize benchmark reports Q4_K_M ≈ 4.89 bits/weight (~0.6 byte/param) and Q8_0 ≈ 8.5 bits/weight (~1.06 byte/param) on Llama-3.1-8B ([llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md), accessed 2026-06-02). Rounding to 0.5 / 1.0 keeps the estimate on the safe side; the per-row GB figures in the table below are derived from this formula, not vendor-published numbers — verify with `ollama show`. | VRAM / unified mem | Comfortable choice (Q4_K_M) | Notes | | --- | --- | --- | | 8 GB | 7–8B Q4_K_M (~5–6 GB) | leave headroom for KV cache + the OS | | 12 GB | up to ~14B Q4_K_M (~9–10 GB) | 7–8B at Q8_0 also fits | | 16 GB | 14B Q4_K_M comfortably; 32B is tight | 32B Q4_K_M ≈ 20 GB — won't fit | | 24 GB | 32B Q4_K_M (~20 GB) | 70B does **not** fit at any usable quant | | 48 GB+ / 2×24 GB | 70B Q4_K_M (~40–48 GB) | needs the full budget; long context pushes over | | Mac unified (e.g. 64 GB) | weights share RAM with everything else | budget against total unified memory | **KV cache is the trap.** It grows ~linearly with `num_ctx` and lives in VRAM *on top of* the weights. At long context (e.g. 128K) a 70B can add tens of GB of cache — often more than people budget for. If you are tight: cap `num_ctx`, or shrink the cache with `OLLAMA_KV_CACHE_TYPE=q8_0` (or `q4_0`). See [references/hardware-sizing.md](references/hardware-sizing.md) for the KV math and a per-context table. Ollama runs a llama.cpp-backed engine (GGUF) by default, with a scheduler that reduces OOM crashes and improves multi-GPU placement. On Apple Silicon it can use an **MLX** backend (shipped in Ollama 0.19, per [ollama.com/blog/mlx](https://ollama.com/blog/mlx), 2026-03-30), but **only on Macs with >32 GB of unified memory** — below that gate it stays on the llama.cpp engine. None of this invents memory you don't have: when the box can't hold the model, that's a `runpod`/`modal` job, not a quant downgrade. ## The API Two surfaces, same daemon. Use **native `/api/chat`** when you want Ollama-specific fields (`keep_alive`, `format` as a JSON schema, `think`); use the **OpenAI-compat `/v1`** layer to reuse an existing OpenAI SDK unchanged. Native chat (`/api/chat`), non-streaming: ```bash curl http://localhost:11434/api/chat -d '{ "model": "qwen3:8b", "messages": [{"role": "user", "content": "Name three primes."}], "stream": false, "options": {"temperature": 0.2, "num_ctx": 8192}, "keep_alive": "10m" }' ``` `stream` defaults to **true** (NDJSON, one object per line, final object has `done: true` + timing stats). `options.num_ctx` sets the context window *for this request* — it does not persist; bake it into a Modelfile if you want it permanent. OpenAI-compatible — point any OpenAI SDK at `localhost:11434/v1` with a dummy key: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") # key is ignored resp = client.chat.completions.create( model="qwen3:8b", messages=[{"role": "user", "content": "Name three primes."}], temperature=0.2, ) print(resp.choices[0].message.content) ``` Structured output — pass a JSON schema as `format` (native) so the model is constrained to valid JSON: ```bash curl http://localhost:11434/api/chat -d '{ "model": "qwen3:8b", "messages": [{"role": "user", "content": "Extract name and age from: Ana is 30."}], "stream": false, "format": { "type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}, "required": ["name", "age"] } }' ``` Tool calling (`tools`), multimodal (`images` as base64), embeddings (`/api/embed`), and the full field tables live in [references/api.md](references/api.md). Endpoint map at a glance: `/api/generate`, `/api/chat`, `/api/embed`, `/api/create`, `/api/pull`, `/api/show`, `/api/ps`, `/api/tags`. ## Modelfiles A Modelfile bakes a base model + system prompt + parameters into a new named model. Build with `ollama create`. ```dockerfile FROM qwen3:8b SYSTEM "You are a terse senior code reviewer. Answer in bullet points." PARAMETER num_ctx 16384 PARAMETER temperature 0.2 PARAMETER stop "<|im_end|>" ``` ```bash ollama create reviewer -f Modelfile # now: ollama run reviewer ``` - `FROM` is required — a model tag or a local file (`FROM ./model.gguf` to import a raw GGUF). - `PARAMETER num_ctx` makes the context window permanent (vs the per-request `options.num_ctx`). - `SYSTEM`, `TEMPLATE`, `LICENSE`, `ADAPTER` (LoRA) round out the instruction set. Quantize on create from an fp16/fp32 source: ```bash ollama create reviewer --quantize q4_K_M -f Modelfile # FROM must be an fp16/fp32 model ``` `--quantize` only works when the `FROM` source is full-precision; you cannot re-quantize an already-Q4 model. To go from Hugging Face weights to a GGUF in the first place, that conversion is a `huggingface` job — Ollama imports the result. ## When to leave the box If the comfortable-choice row for your VRAM can't hold the model you actually need (e.g. you need 70B quality on a 12 GB laptop), stop downgrading quant — quality collapses below Q4 and you'll still OOM at real context. Move it to a remote GPU: `runpod` (rent a GPU), `modal` (serverless container + GPU autoscale), or a hosted endpoint (`replicate`, `together-fireworks`, `fal`). Ollama is the right tool *until the weights + KV cache exceed the single box*. ## Anti-patterns | Bad | Good | Why | | --- | --- | --- | | Pull fp16 on a box that only fits Q4 | Pull Q4_K_M (or Q8_0 if it fits) | fp16 is 4× the VRAM of Q4 for ~3–5% quality; you'll OOM for nothing | | `num_ctx: 128000` on a 12 GB GPU | Cap `num_ctx` to what fits; `OLLAMA_KV_CACHE_TYPE=q8_0` | KV cache scales with context and sits on top of weights — long context dwarfs the model | | `/api/generate` for a chat with history | `/api/chat` with a `messages` array | `generate` is single-turn; you'd hand-concatenate history and break the chat template | | `ollama pull mistral:latest`, assume it's small | Pin an explicit tag (`:7b`, a quant tag) and `ollama show` it | `:latest` size/quant drifts release to release; sizing breaks silently | | Treat Ollama as a multi-tenant prod server | Use it local/single-box; scale → `runpod`/`modal` | one daemon, limited parallelism (`OLLAMA_NUM_PARALLEL`); not built for fleet serving | | Hardcode `api.openai.com` when target is local | `base_url="http://localhost:11434/v1"`, dummy key | the OpenAI SDK works unchanged against the compat layer; no remote calls, no key leak | | Downgrade to Q2 to force a 70B onto 12 GB | Pick a model that fits, or move to a remote GPU | sub-Q4 quality drops sharply *and* it still won't fit at real context | | Assume `ollama list` means it's loaded | `ollama ps` for memory, `list` for disk | a pulled model uses 0 VRAM until a request loads it | ## Verify Run `scripts/verify.sh [TARGET]` from your project root (or a dir holding a `Modelfile`). Static by default — it needs neither Ollama installed nor a running daemon. It lints a `Modelfile` (FAIL if no `FROM`; WARN on unknown instructions or a `num_ctx` so high it will OOM consumer GPUs), notes whether app code points at the local `localhost:11434` / `/v1` endpoint vs only-remote hosts, and — only if `ollama` is on PATH — best-effort confirms a model is present (WARN, not FAIL). It exits non-zero **only** on a real FAIL; an empty/clean target passes. ## References - [references/api.md](references/api.md) — full endpoint catalog, request/response field tables, OpenAI-compat path mapping, structured output, tool calling, streaming, embeddings (curl + Python). - [references/hardware-sizing.md](references/hardware-sizing.md) — the full quant ladder, VRAM formula derivation, KV-cache math + per-context table, per-model chart, Apple Silicon unified-memory notes, and the env knobs (`OLLAMA_KV_CACHE_TYPE`, `OLLAMA_FLASH_ATTENTION`, `OLLAMA_NUM_PARALLEL`, `OLLAMA_MAX_LOADED_MODELS`) for fitting tight boxes.