open-weights

# Open weights — pick the model, then clear the license This is the CHOICE layer. Given a task, a box, and a shipping constraint, it tells you *which* open model to reach for and *whether the license lets you ship it*. It does not download, run, serve, or fine-tune anything — it feeds the skills that do (see [Related skills](#related-skills)). ## The one rule (read before you name a model) **Model names and their licenses change monthly. Never state a license from memory — open the model card and confirm it, every time.** This file was authored against cards live in mid-2026; by the time you read it, versions have shipped and terms have moved. So: - Treat every specific version/license line below as **[verify the card]**, not as settled fact. - State the license **class** (OSI-open vs custom-community vs restricted), then send the reader to the actual `LICENSE` / terms page for the exact model + size they intend to ship. - The *decision framework* and the *class taxonomy* are the durable parts. The version list is the perishable part. Weight your trust accordingly. Real, verified-mid-2026 examples of *why* this rule exists — all confirmed against the source, and all things that were different at my Jan-2026 cutoff: - **Gemma 4** moved to **Apache-2.0**, while Gemma 1–3 stay on the custom Gemma Terms of Use (confirmed on [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms), accessed 2026-07-18). Same brand, opposite license class depending on version. - **Codestral 2** was relicensed **Apache-2.0** (per Mistral, Apr 2026), while the original Codestral stays **non-production (MNPL)**. Same name, opposite shippability depending on release. - **DeepSeek-R1** is **MIT**, but the original **DeepSeek-V3** weights ship under a **custom DeepSeek License Agreement with OpenRAIL-style use restrictions** (confirmed on the [V3 LICENSE-MODEL](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/LICENSE-MODEL)), not MIT. Same family, different license per model. If three of the load-bearing facts moved in one release cycle, yours have too. Check the card. ## 1. How to choose an open model Five gates, in this order. The license gate can veto everything above it, so never fall in love with a model before you clear it. 1. **Task fit** — match the model's design to the job: - General chat / instruct → Llama, Qwen, Mistral, Gemma instruct variants. - Coding → Qwen-Coder, Codestral (license!), DeepSeek-Coder, gpt-oss. - Reasoning / math → DeepSeek-R1-style reasoning models, gpt-oss, Qwen reasoning variants. - Multilingual → Qwen and Gemma tend to lead; check the card's language list. - Multimodal (vision) → Llama vision, Qwen-VL, Gemma vision, PaliGemma, Phi multimodal. - Embeddings / retrieval → a dedicated embedding model, **not** a chat LLM → `embeddings-search`. - Small / edge → 0.5B–4B (Qwen small, Gemma small, Phi-mini, gpt-oss-20b class). 2. **Size vs your hardware class** — can it even fit? See [§4](#4-size--vram). A 70B at fp16 is ~140 GB; if you have one 24 GB GPU, that model is off the table unless you quantize *and* accept the quality/throughput cost. Pick a size your box can hold before you compare quality. 3. **License / shippability** — **the gate that matters most** (see [§3](#3-licenses)). Decide up front: is this a hobby/internal use, or a commercial product you distribute? "Open weights" does **not** mean "open source" and does **not** guarantee commercial rights. Verify the card. 4. **Hardware target** — consumer GPU (8–24 GB), data-center GPU (A100/H100 80 GB), Apple unified memory, or CPU/edge. This narrows both size and quant format (GGUF for CPU/Mac/llama.cpp; AWQ/GPTQ for GPU serving). 5. **Community support** — a proxy for "will this actually work Monday morning": are there ready-made GGUF/AWQ quants on the Hub, is it supported by your runtime (llama.cpp/Ollama/vLLM), are there fine-tunes and recent downloads? A model with no quants and no runtime support is a research artifact, not a shippable choice. Rule of thumb: **smallest model that passes your eval wins.** Don't reach for 70B when a well-chosen 8B clears the bar — it's cheaper to run, faster, and fits more hardware. Prove it with `agent-eval`. ## 2. Model families Families and *typical* size ladders below. Specific version claims are marked — **[verify]** at author time; the family and its rough sizing are the durable part. Full per-family notes, strengths, and license pointers in [references/model-families.md](references/model-families.md). | Family | Maker | Typical open sizes | Notes / current line [verify] | |---|---|---|---| | **Llama** | Meta | ~1B–~400B (dense + MoE) | Llama 4 (Scout/Maverick, MoE) current line; custom **Community License**, gated | | **Qwen** | Alibaba | 0.5B–235B+ (dense + MoE), strong Coder/VL | mostly Apache-2.0 but **per-size variance** — check each card | | **Mistral / Mixtral** | Mistral AI | 7B dense, 8x7B/8x22B MoE, "Small/Large" | open ones Apache-2.0; **Codestral original = MNPL non-production** | | **Gemma** | Google | ~1B–~27B, plus PaliGemma/CodeGemma/ShieldGemma | Gemma 1–3 = **custom Gemma Terms**; Gemma 4 = Apache-2.0 [verify] | | **DeepSeek** | DeepSeek | V3-class MoE (~600B+), R1 reasoning, distills | **per-model license split** — R1 MIT, original V3 custom; verify | | **Phi** | Microsoft | ~3B–15B ("mini", reasoning, multimodal) | **MIT** across the Phi-4 family [verify] | | **gpt-oss** | OpenAI | 20B and 120B (MoE, open *weights*) | **Apache-2.0** [verify]; OpenAI's first open-weight LLMs since GPT-2 | | Others | various | — | SmolLM, OLMo (fully-open incl. data), Falcon, Yi, Command — check card | ## 3. Licenses **This is the load-bearing section. Get it wrong and you ship something you have no right to ship.** Every model falls into one of three classes. Identify the class, then open the card. ### The three classes | Class | Examples (class, not a promise) | What it means for shipping | |---|---|---| | **OSI-open** (Apache-2.0, MIT) | Qwen (most), Mistral open, Phi, gpt-oss, DeepSeek-R1, Gemma **4** | Commercial use, modify, redistribute — permissive. Still read the card for attribution/notice. | | **Custom / community** | **Llama** (Meta Community), **Gemma 1–3** (Gemma Terms) | Broad free use **but with conditions**: acceptable-use policy, attribution, sometimes a scale cap. NOT OSI-open. | | **Non-commercial / restricted** | original **Codestral** (MNPL), any "research-only", some RAIL, some "≥$X revenue → buy a license" | You **cannot** ship it in a commercial product without a separate license. Fatal if missed. | "Open weights" describes availability of the weights file — it says nothing about your legal rights. A model can be a free download and still be non-commercial. **The download button is not a license.** ### Per-family license notes (state the class, then verify the exact card) - **Llama = Meta Llama Community License — NOT OSI-open.** Carries an Acceptable Use Policy, a **"Built with Llama"** attribution requirement on derivatives/products, naming rules, and a **>700M-monthly-active-users clause** (above that you must request a separate license from Meta, granted at Meta's discretion). Gated on the Hub (accept terms first). Verify the *current* Llama version's terms — Meta has revised them across releases. - **Gemma = custom Gemma Terms of Use + Prohibited Use Policy** for Gemma 1–3 (not Apache), with a duty to pass the restrictions **downstream** to every user and to ship the terms/notice file. Gemma **4** reportedly moved to **Apache-2.0** — a textbook reason to check the version's card. - **Qwen = mostly Apache-2.0, but with per-size variance.** Historically some sizes (often the very largest or a special tier) carried a separate Qwen license instead of Apache. **Check each size's card** — do not assume "Qwen = Apache" for the whole family. - **Mistral / Mixtral open models = Apache-2.0**, but the **original Codestral is MNPL (non-production)** — free for research/eval, **not** for commercial deployment. Codestral **2** was relicensed Apache-2.0; verify which Codestral release you actually have. Note some newer Mistral models use a "Modified MIT" with a revenue-threshold commercial clause — verify. - **DeepSeek = per-model split.** Code repos are MIT; **DeepSeek-R1 weights are MIT**; but the original **DeepSeek-V3 weights** ship under a **custom DeepSeek License Agreement with OpenRAIL- style use-based restrictions**. "DeepSeek = MIT" is too simple — verify the specific model. - **Phi = MIT** across the Phi-4 family [verify] — genuinely permissive, commercial-friendly. - **gpt-oss (OpenAI open-weight models, 20B/120B) = Apache-2.0** [verify] — permissive. ### Commercial-use checklist (run before you ship) - [ ] Opened the **exact model + size** card and read its `LICENSE` / terms — not a blog, not this file. - [ ] Confirmed the **class**: OSI-open, custom-community, or restricted/non-commercial. - [ ] If custom/community: identified the **conditions** (attribution string, acceptable-use policy, MAU/revenue caps, downstream pass-through duty) and can meet them. - [ ] If restricted: confirmed you are **not** deploying commercially, or obtained a separate license. - [ ] Checked whether the model is **gated** (accept terms on the Hub before download). - [ ] Checked the **base model** license — a fine-tune inherits the base's obligations (a Llama fine-tune still owes "Built with Llama"; a distill can inherit the teacher's terms). Deeper class breakdown, gated-model mechanics, and the fine-tune-inheritance trap: [references/licenses.md](references/licenses.md). ## 4. Size ↔ VRAM A **selection-time** budget: will this size class even fit your hardware? (Operational per-request KV-cache math lives in `ollama`/`vllm` — this is the "before I pick it" estimate.) ```text weights_GB ≈ params(B) × bytes_per_param # then add KV cache + runtime overhead on top fp16/bf16 → 2.0 (≈ 2 GB per 1B params) 8-bit → ~1.0 (≈ 1 GB per 1B params) 4-bit → ~0.6 (≈ 0.6 GB per 1B + cache + overhead; conservative ~0.5–0.6/param) ``` What that means per hardware class (weights only — leave headroom for context/KV cache): | Hardware | fp16 ceiling | Practical pick (4-bit) | |---|---|---| | 8 GB consumer GPU | ~3B | 7–8B at 4-bit (~5 GB) | | 16 GB | ~7B | 13–14B at 4-bit | | 24 GB (e.g. 3090/4090) | ~10B | ~32B at 4-bit (~20 GB) | | 80 GB (A100/H100) | ~34B | 70B at 4-bit (~40 GB), or a big-MoE like gpt-oss-120b | | Apple unified (e.g. 64 GB) | shares with the OS | budget against **total** unified memory | | multi-GPU / 2×80 GB+ | 70B fp16 (~140 GB) via tensor-parallel | full-precision large models | **A 70B at fp16 is ~140 GB — it does not fit one GPU.** You either quantize (4-bit ≈ ~40 GB, fits one 80 GB card) or split across GPUs with **tensor parallelism** (that's a `vllm` job). Longer context grows the KV cache *on top of* weights, so keep a margin. Full derivation + KV math: [references/sizing-and-quant.md](references/sizing-and-quant.md). ## 5. Quant / formats Quantization shrinks weights (fewer bits/param) to fit smaller hardware, trading a little quality. Which **format** you pick is driven by your runtime: | Format | Where it runs | Use it for | |---|---|---| | **GGUF** | llama.cpp, **Ollama**, LM Studio | CPU, Apple Silicon, single-box local; the everyday local format | | **AWQ** | vLLM, TGI, transformers | GPU serving — activation-aware 4-bit, strong quality/latency | | **GPTQ** | vLLM, TGI, transformers | GPU serving — older, widely available 4-bit | | bitsandbytes (NF4) | transformers | quick load-time 4/8-bit for experiments/fine-tuning | | EXL2 | ExLlamaV2 | flexible bit-rates on consumer GPUs | **Decoding a GGUF quant tag like `Q4_K_M`:** `Q4` = ~4-bit weights; `_K` = a k-quant (mixed precision — keeps the more sensitive tensors at higher bit-depth); `_M` = the medium size/quality tier (`_S` smaller/lower, `_L` larger/higher). **`Q4_K_M` is the standard default** — roughly half the memory of fp16 for a few percent quality loss. `Q8_0` is near-lossless (~1 byte/param); below `Q4`, quality drops off fast. Don't go sub-Q4 to force a too-big model onto a too-small box — pick a smaller model instead. More in [references/sizing-and-quant.md](references/sizing-and-quant.md). ## 6. Where to get it + how to run it (routing) You've chosen a model and cleared its license. Now hand off — this skill stops here. - **Get the weights / host on HF** → `huggingface` (Hub download/upload, Inference Providers, endpoints). Also where you convert to GGUF. - **Run it locally on one box** (laptop/desktop, GGUF via Ollama) → `ollama`. - **Serve it at throughput / with tensor-parallel** (production, batching, big models) → `vllm`. - **Fine-tune it** (LoRA/QLoRA, SFT, preference tuning) → `finetuning`. - **Prove the small model is enough** before committing → `agent-eval`. The Hub filters (task + license + size + recent downloads) and the Ollama library are where you actually find candidates — that mechanics lives in `huggingface` / `ollama`, not here. ## Guardrails / gotchas - **Never assert a license from memory.** State the class, open the card. Licenses change per version, per size, and per release month. - **"Open weights" ≠ "open source" ≠ "commercial-use OK".** Three different things. - **A fine-tune inherits its base license.** Fine-tuning Llama does not launder away "Built with Llama" or the acceptable-use policy; a distill can inherit the teacher model's terms. - **Gated ≠ restricted.** Gemma/Llama are gated (accept terms to download) yet usable commercially under their conditions; a "research-only" card is restricted regardless of gating. - **Don't sub-Q4 a too-big model onto a too-small GPU.** Quality collapses and it still may OOM at real context. Choose a size that fits. - **Newest ≠ best for you.** The smallest model that passes your eval wins on cost, speed, and fit. - **MoE total vs active params.** A "120B" MoE may activate only ~5B/token (fast) but you still need VRAM for *all* experts — size against total params, not active. ## Related skills - **`huggingface`** — get/host: Hub download/upload, Inference Providers, endpoints, GGUF conversion. This skill tells you *which* repo to pull; `huggingface` pulls it. - **`ollama`** — run one model locally (GGUF, single box, VRAM sizing at request time). This skill says which model + quant *class* fits; `ollama` runs it and does the KV-cache math. - **`vllm`** — serve at throughput, batching, tensor-parallel for models too big for one GPU. - **`finetuning`** — adapt a chosen base with LoRA/QLoRA/SFT. Choose+clear the base here first. - Boundary: this skill is **selection + licensing knowledge only**. It never re-teaches calling, running, serving, or training — it feeds those four. ## Checklist - [ ] Named the **task** and matched a family designed for it (chat/code/reasoning/vision/embeddings/edge). - [ ] Picked the **smallest size** likely to pass the eval, and confirmed it **fits the hardware class**. - [ ] Identified the **license class** (OSI-open / custom-community / restricted). - [ ] Opened the **exact model + size card** and confirmed the license — not from memory, not from a blog. - [ ] For commercial use: met the conditions (attribution, acceptable-use, MAU/revenue caps) or confirmed permissive. - [ ] Checked the **base model** license if it's a fine-tune/distill.