| name | huggingface-local-models |
| description | Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving. |
| risk | unknown |
| source | https://github.com/huggingface/skills/tree/main/skills/huggingface-local-models |
| source_repo | huggingface/skills |
| source_type | official |
| date_added | 2026-07-01T00:00:00.000Z |
| license | Apache-2.0 |
| license_source | https://github.com/huggingface/skills/blob/main/LICENSE |
Hugging Face Local Models
When to Use
Use this skill when you need use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving.
Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.
Default Workflow
- Search the Hub with
apps=llama.cpp.
- Open
https://huggingface.co/<repo>?local-app=llama.cpp.
- Prefer the exact HF local-app snippet and quant recommendation when it is visible.
- Confirm exact
.gguf filenames with https://huggingface.co/api/models/<repo>/tree/main?recursive=true.
- Launch with
llama-cli -hf <repo>:<QUANT> or llama-server -hf <repo>:<QUANT>.
- Fall back to
--hf-repo plus --hf-file when the repo uses custom file naming.
- Convert from Transformers weights only if the repo does not already expose GGUF files.
Quick Start
Install llama.cpp
brew install llama.cpp
winget install llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make
Authenticate for gated repos
hf auth login
Search the Hub
https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
Run directly from the Hub
llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
Run an exact GGUF file
llama-server \
--hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \
--hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
-c 4096
Convert only when no GGUF is available
hf download <repo-without-gguf> --local-dir ./model-src
python convert_hf_to_gguf.py ./model-src \
--outfile model-f16.gguf \
--outtype f16
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M
