ollama-local-setup

name	ollama-local-setup
description	Stands up and repairs a Windows local Ollama server so models on a custom disk appear on localhost:11434 (/api/tags and /v1/models) and GPU KV-cache is not starved. Use when ollama list is empty, models vanish after reboot, port 11434 conflicts, or throughput collapses. Not for Ollama Cloud GLM planning (ollama). Never treat the tray app as honoring OLLAMA_MODELS without a junction.
version	1.0.1
category	mlops

ollama-local-setup

Production guide for standing up and fixing a local Ollama server on Windows, where models live on a non-default disk and must be visible to clients connecting via http://localhost:11434 (both the native /api/tags and the OpenAI-compatible /v1/models). Also covers performance tuning — diagnosing and fixing GPU starvation caused by oversized context windows (KV cache spilling to CPU RAM).

When to Use

Trigger this skill when any of the following are true:

• ollama list is empty, /api/tags returns {}, or /v1/models returns {"object":"list","data":null}.
• User wants local-only inference (Hermes provider=ollama, base_url=http://localhost:11434/v1, api_key=ollama).
• Models were moved to another drive and the server no longer sees them.
• After a reboot or relaunch the local models "vanished."
• You are about to start an ollama serve / ollama app and port 11434 may already be in use.
• Local inference is very slow (single-digit tok/s) or GPU utilization sits near ~5% even on a powerful GPU (e.g. RTX 3090 Ti). Classic KV-cache spill from an oversized context_length.

Prerequisites

• Windows host (PowerShell primary). Linux/macOS notes provided where relevant.
• Ollama installed (app or CLI). Default install path: $env:LOCALAPPDATA\Programs\Ollama\ollama app.exe.
• A target models directory on a non-default drive, e.g. YOUR_MODELS_DIR, containing blobs/ (multi-GB files) and manifests/ (model tags). Set $env:OLLAMA_MODELS to that path.
• For performance tuning: an NVIDIA GPU with nvidia-smi available.
• For Hermes integration: Hermes config must set provider=ollama, base_url=http://localhost:11434/v1, api_key=ollama, and context_length>=64000 (Hermes refuses to init below 64K).

Procedure

Phase 1 — Fix "empty models" on Windows (three gotchas)

There are three silent failure modes on Windows that cause ollama list to return empty:

1. OLLAMA_MODELS must use BACKSLASHES on Windows (e.g. YOUR_MODELS_DIR via $env:OLLAMA_MODELS). Forward slashes are silently rejected — Ollama falls back to the empty default C:\Users\<user>\.ollama\models, and /api/tags returns {}. This is silent: no error, just empty.
2. The Windows Ollama app (ollama app.exe, the tray app) IGNORES OLLAMA_MODELS entirely. It always serves from its default path. Fix by redirecting the default path itself with a directory junction (see below), not by setting the env var alone.
3. Only ONE server per port. Two ollama serve/app processes on 11434 → /v1/models returns null and clients get 404s. Kill all Ollama processes, then start exactly one.

Step 1 — Set the env var correctly (backslashes)

# Point at YOUR_MODELS_DIR (backslashes on Windows)
$env:OLLAMA_MODELS = "YOUR_MODELS_DIR"
# User scope (re-logon to apply; new shells only)
setx OLLAMA_MODELS $env:OLLAMA_MODELS

# Machine scope (requires admin)
setx OLLAMA_MODELS $env:OLLAMA_MODELS /M

HARD RULE: Always use backslashes on Windows. Forward slashes are the #1 silent failure.

Step 2 — Redirect the default path with a directory junction

The app ignores the env var, so redirect the default path C:\Users\<user>\.ollama\models to your custom directory:

# Only if the default dir is empty or you've moved its contents
cmd /c rmdir "C:\Users\<user>\.ollama\models"

# Create the junction (reboot-safe, filesystem-level)
cmd /c mklink /J "C:\Users\<user>\.ollama\models" $env:OLLAMA_MODELS

A junction is filesystem-level and reboot-safe — it survives restarts and does not depend on an env var being set.

HARD RULE: Do not trust the app to honor OLLAMA_MODELS. Use the junction. Setting the env var alone will not move the app's models.

Step 3 — Kill stale processes, then start exactly ONE server

# Kill all Ollama processes
taskkill /F /IM "ollama app.exe" /T
taskkill /F /IM ollama.exe /T

# Launch the normal app (tray + supervised server)
start "" "$env:LOCALAPPDATA\Programs\Ollama\ollama app.exe"

HARD RULE: Do not run both ollama app.exe and a manual ollama serve. Pick one. Two listeners on 11434 break /v1/models.

Linux/macOS note: OLLAMA_MODELS works normally there and forward slashes are fine. The junction trick is Windows-specific. On Linux a symlink (ln -s) achieves the same redirect.

Step 4 — Verify

See Verification below. If still empty, re-read OLLAMA_MODELS with backslashes and confirm the junction target exists and contains blobs/ + manifests/.

Load references/windows-troubleshooting.md when you need copy-paste command recipes (env set, junction, kill+relaunch, verify) with placeholders for a specific user/machine.

Phase 2 — Performance tuning (GPU starvation from oversized context)

Symptom: a large model (e.g. qwen3.6:27b, 27B) generates ~3 tok/s and nvidia-smi shows ~5% GPU compute util during generation, even on a strong GPU (RTX 3090 Ti, 24 GB). VRAM shows the model loaded but the KV cache is elsewhere.

Root cause: Ollama sizes the KV cache from the effective context length at load = min(client context_length request, OLLAMA_CONTEXT_LENGTH env var). A 27B model at 256K context needs a ~40 GB KV cache. When that exceeds free VRAM, Ollama silently allocates it in CPU RAM and shuffles KV data per token → GPU idle, throughput collapses. The binding cap is the server env var OLLAMA_CONTEXT_LENGTH; lowering the Hermes/client context_length alone does NOT free VRAM unless the env var is also reduced.

Step 1 — Persist tuning in Windows User env

setx OLLAMA_CONTEXT_LENGTH 65536
setx OLLAMA_KV_CACHE_TYPE q4_0
setx OLLAMA_FLASH_ATTENTION true

• OLLAMA_CONTEXT_LENGTH=65536 — 64K satisfies Hermes's context_length>=64000 requirement and is small enough for the KV cache to fit on GPU for most chat.
• OLLAMA_KV_CACHE_TYPE=q4_0 — 4-bit KV cache. qwen3.6:27b ≈ 17.8 GB + q4 KV ≈ 2.6 GB ≈ 20.4 GB of 24 GB, leaving headroom.
• OLLAMA_FLASH_ATTENTION=true — faster attention kernels.

Step 2 — Restart the server

taskkill /F /IM "ollama app.exe" /T
taskkill /F /IM ollama.exe /T
# Then launch one server (app or ollama serve, not both)
start "" "$env:LOCALAPPDATA\Programs\Ollama\ollama app.exe"

Step 3 — Verify performance improvement

In one shell, monitor GPU:

nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1

In another shell, run a 200-token generation. Use think:false for Qwen3 (it defaults to thinking mode, which inflates latency).

Load references/performance-tuning.md when diagnosing GPU starvation or when you need the exact generation-speed probe and before/after numbers (~3→~33 tok/s, ~5%→~90% GPU util on a 3090 Ti).

Pitfalls

• Forward-slash env var is the #1 silent failure. Always set backslashes on Windows. A forward-slash YOUR_MODELS_DIR is silently rejected; Ollama falls back to the empty default with no error.
• Don't trust the app to honor OLLAMA_MODELS. Use the junction. Setting the env var alone will not move the app's models.
• Don't run both ollama app.exe and a manual ollama serve. Pick one. Two listeners on 11434 break /v1/models (returns null, clients get 404s).
• Oversized context_length starves the GPU (performance, not visibility). A 27B model at 256K context allocates a ~40 GB KV cache that won't fit VRAM, so Ollama spills it to CPU RAM → ~5% GPU util, ~3 tok/s. Fix: cap OLLAMA_CONTEXT_LENGTH (e.g. 65536) and set OLLAMA_KV_CACHE_TYPE=q4_0. Reducing the client context_length alone does not help — the server env var is the binding cap.
• Verify with ollama list and raw curl, not fragile JSON-parsing one-liners. Prefer the official CLI and a direct curl http://localhost:11434/api/tags. A mis-parsed probe can report "empty" when the server is actually healthy (false alarm). See scripts/verify-ollama.ps1.
• Confirm models are actually on disk at the junction target before blaming the server: blobs/ should contain multi-GB files and manifests/ should contain model tags.
• Qwen3 defaults to thinking mode, which inflates latency during speed probes. Always pass think:false when measuring generation throughput.

Verification

Run scripts/verify-ollama.ps1 (PowerShell, no external deps) for a deterministic probe that checks /api/tags, /v1/models, OLLAMA_MODELS, and the default-vs-custom path; reports a clear PASS/FAIL.

Or manually:

# CLI check
ollama list

# Native API
curl -s http://localhost:11434/api/tags

# OpenAI-compatible API
curl -s http://localhost:11434/v1/models

Expected: all configured models appear under both endpoints, including the one the client needs (e.g. qwen3.6:27b for Hermes).

For Hermes integration, ensure config sets:

• provider=ollama
• base_url=http://localhost:11434/v1
• api_key=ollama
• context_length>=64000 (Hermes refuses to init below 64K)

For performance verification, watch nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1 in one shell while running a 200-token generation in another. Expect GPU util to jump from ~5% to ~90% and throughput from ~3 tok/s to ~33 tok/s on a 3090 Ti with the tuning recipe applied.

Support Files

• references/windows-troubleshooting.md — Load when you need copy-paste command recipes (env set, junction, kill+relaunch, verify) with placeholders for a specific user/machine.
• references/performance-tuning.md — Load when diagnosing GPU starvation or when you need the exact generation-speed probe and before/after numbers.
• scripts/verify-ollama.ps1 — Load and run for a deterministic end-to-end probe (checks /api/tags, /v1/models, OLLAMA_MODELS, and default-vs-custom path; reports PASS/FAIL).