local-model-fit

name	local-model-fit
description	Compute the VRAM/RAM budget and pick a model size and quantization before downloading anything. Use when choosing local models, planning GPU hardware, or hitting out-of-memory errors.
version	1.0.0
author	Zorost Intelligence
license	MIT
metadata	{"zorost":{"tags":["local-inference","gpu","vram","quantization","ollama","llamacpp","build"],"related_skills":["eval-first-development","agent-loop-safety"],"program_weeks":[8,9]}}

Local Model Fit

1 · Purpose

Turn "will this model run on my machine?" from a download-and-pray experiment into arithmetic you do before touching the network.

2 · When to use

• Choosing any local model + quantization for a laptop, workstation, or server.
• Planning GPU purchases or cloud GPU instances.
• Debugging CUDA out of memory / Metal allocation failures.

3 · Inputs

• Your usable GPU VRAM (or unified memory on Apple Silicon, budget ~70% of total RAM for the GPU).
• The model's parameter count (from its card) and the context length you need.
• The quantization you are considering (FP16, Q8, Q6_K, Q5_K_M, Q4_K_M…).

4 · Procedure

1. Compute the weights budget: params (B) × bytes-per-param. FP16 ≈ 2 bytes, Q8 ≈ 1, Q6_K ≈ 0.80, Q5_K_M ≈ 0.68, Q4_K_M ≈ 0.60. A 14B at Q4_K_M ≈ 8.4 GB.
2. Compute the KV-cache budget: roughly 1 to 2 GB per 8k context for a 7 to 14B model, scaling with layers and heads, when precision matters, serve once and read the actual allocation from the engine's log.
3. Add runtime overhead: ~1 to 2 GB for the engine, compute buffers, and the OS sharing the GPU.
4. Total = weights + KV cache + overhead. Require total ≤ 90% of usable VRAM. If over, drop one lever: smaller quant, shorter context, or smaller model, in that preference order for quality preservation.
5. Pick the model class by task before picking the quant: tool-calling agents need tool-tuned models (e.g. Hermes-class); embeddings need embedding models; chat quality tracks size within a family.
6. Then pick the quant: Q4_K_M is the default sweet spot; Q5_K_M/Q6_K when quality measurably matters and VRAM allows; Q8 near-lossless; below Q4 only when nothing else fits.
7. Download, load, and read the engine's actual memory report. Compare against your arithmetic. Investigate any gap over 15%, it means you miscounted the cache.
8. Benchmark tokens/sec on a representative prompt. A model that fits but crawls below your interactivity floor (often ~10 tok/s for chat) does not "fit".

5 · Anti-rationalization

Excuse	Answer
"The page says it runs on 16 GB."	Marketing pages quote weights only, at minimum context, before the OS takes its share. Do the arithmetic.
"I'll use the smallest quant to be safe."	Below Q4 the quality drop is real and task-visible. Spend VRAM on quality when you have it.
"Context length is free to crank up."	The KV cache grows linearly with context. 128k context on a 14B can cost more VRAM than the weights.
"It OOM'd once; just retry."	OOM is arithmetic announcing itself. Recompute, do not retry.

6 · Red flags

• The plan names a model and quant but no context length.
• Total memory "should be fine" with no written number.
• The OS and other apps share the GPU but got 0 GB in the budget.
• Quality was never measured after the quant choice (see eval-first-development).

7 · Verify

• A written budget exists: weights + KV cache + overhead, total, and usable VRAM.
• The engine's measured allocation matches the budget within 15%.
• tokens/sec on a representative prompt is recorded.
• The chosen configuration clears the task eval (quality was not traded away blind).

8 · ZoroLogistics example

Week 8 to 9: the triage brain on a 16 GB MacBook. Usable ≈ 11 GB. Hermes-4-14B at Q4_K_M: weights 8.4 GB + KV cache at 8k context ~1.5 GB + overhead ~1 GB ≈ 10.9 GB, fits at the edge of the 90% rule. The alternative, 14B at Q5_K_M (9.5 GB), does not fit with headroom; the 8B at Q6_K does and benchmarks 2× faster. The eval decides between them, fit is the gate, accuracy is the vote.

---

© 2026 Zorost Intelligence LLC · zorost.com