huggingface

# Hugging Face: Hub, routed/hosted inference, and transformers Hugging Face is three surfaces, and you should always know which one you are on: 1. **The Hub** — versioned git repos for models, datasets, and Spaces. You search it, you `hf download` / `hf upload`, you read and write model cards. 2. **Inference** — three ways to actually run a model: the **Inference Providers** router (serverless, you own nothing), a **dedicated Inference Endpoint** (you own a deployment that autoscales), or **local `transformers`** (you own the machine). 3. **The catalog** — 1M+ open models you choose from by task, license, and size. The whole skill is choosing the right surface for the job and proving it works: a 200 router response, a live endpoint URL, a pushed repo commit. If the model is open and the workflow lives on huggingface.co, you are in the right place. Operating the GPU box yourself is [`../ollama/SKILL.md`](../ollama/SKILL.md) (your machine) or [`../runpod/SKILL.md`](../runpod/SKILL.md) (a rented box); training weights is [`../finetuning/SKILL.md`](../finetuning/SKILL.md). ## Decision: how should I run this model? Pick the row before you write a line of code. The cheapest mistake is standing up infra you did not need. | Situation | Use | Why | |---|---|---| | Try a model now, low/dev volume, own no infra | **Inference Providers router** (`InferenceClient`) | Fastest path; monthly credits cover dev. | | CPU task: embeddings, text-ranking, text-classification, small BERT/GPT-2 | `provider="hf-inference"` | That is exactly its remaining niche as of July 2025. | | Big LLM (8B, 70B, 405B) through HF | router with a **partner provider** (Together/Fireworks/Cerebras/DeepInfra…) | `hf-inference` does not serve big LLMs — it will 404 or stall. | | Steady prod traffic, need fixed latency/SLA | **dedicated Inference Endpoint** + scale-to-zero | Predictable, autoscaling, billed per minute. | | Interactive demo or shareable GPU app | **Space** (Gradio + ZeroGPU) | Free-ish, public URL, GPU only while a call runs. | | One-off GPU job (eval, batch convert) | `hf jobs run` | No standing infra; PRO feature. | | Offline, data-private, or already on a GPU box | local `transformers` `pipeline()` | No network, no per-call cost. | ## Auth & install ```bash pip install "huggingface_hub[inference]" # 1.17.0; needs Python >=3.10 pip install transformers # 5.x line, PyTorch-first, optional/local hf auth login # stores a token; or export HF_TOKEN=... ``` - **The CLI is `hf` now**, shaped `hf <resource> <action>` (`hf auth login`, `hf download`, `hf upload`, `hf repo create`, `hf jobs run`). `huggingface-cli` still runs but prints a deprecation warning — do not write it into new scripts. - **Never hardcode a `hf_...` token in code** — tokens leak the moment the file hits git. Read from the environment instead: ```python import os from huggingface_hub import InferenceClient client = InferenceClient(api_key=os.environ["HF_TOKEN"]) # never api_key="hf_xxx" ``` - Token scopes: **read** to pull public/gated repos and run inference, **write** to push, **fine-grained** to scope to specific repos/orgs — why: a leaked read token cannot overwrite your models. ## Inference Providers — the default path One router reaches 200+ models across partner providers plus `hf-inference`; HF passes provider cost through with **no markup**. Two equivalent entry points: ```python # Native client — task methods, NOT the removed .post() from huggingface_hub import InferenceClient client = InferenceClient(api_key=os.environ["HF_TOKEN"]) out = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "One sentence on diffusion models."}], provider="together", # name a partner; or omit for auto-routing ) print(out.choices[0].message.content) ``` ```python # OpenAI-compatible — same router, drop-in for existing OpenAI code from openai import OpenAI client = OpenAI( base_url="https://router.huggingface.co/v1", # this exact host, nothing else api_key=os.environ["HF_TOKEN"], ) ``` - **`InferenceClient.post()` was removed** (dropped in hub v0.31.0). Use the task methods: `chat.completions.create()`, `text_generation()`, `feature_extraction()` (embeddings), `text_to_image()`, `automatic_speech_recognition()`. - **Credits are real and small**: Free $0.10/mo, PRO $2.00/mo, Team/Enterprise $2.00 per seat (shared). Past that you are pay-as-you-go and must buy credits. Budget accordingly — why: a chat loop on a 70B model burns the free tier in minutes. - A **Custom Provider Key** bypasses HF billing entirely (the provider bills you; HF credits do not apply). For org billing, pass `bill_to="org-name"` (header `X-HF-Bill-To`). - Full recipes (embeddings, image, ASR, streaming, rate-limit handling, the provider list) live in `references/inference-providers.md`. ## Hub ops ```bash hf download meta-llama/Llama-3.1-8B-Instruct --include "*.safetensors" hf repo create my-org/my-model --repo-type model hf upload my-org/my-model ./out --commit-message "v1 weights" ``` ```python from huggingface_hub import snapshot_download path = snapshot_download("BAAI/bge-small-en-v1.5") # full repo, cached, resumable ``` - **Gated models** (Llama, Gemma, many others) need you to accept terms on the model page first, then a token with read access — otherwise the download 403s. - A model card is a `README.md` with YAML front-matter (`license`, `pipeline_tag`, `tags`, `base_model`). Ship one on every upload — why: an uncarded repo is unsearchable and unusable by anyone but you. Command map and `hf jobs run` details in `references/hub-and-cli.md`. ## Choosing a model Filter the Hub by **task + license + size + recent downloads**, then read the card before you commit. Match the model to your constraint; do not grab whatever is trending. - Check the **license**: Apache-2.0/MIT are permissive; Llama/Gemma carry commercial terms and are gated; "non-commercial"/"research-only" cards mean you cannot ship them. - Check **size vs target**: a 70B will not fit a single A10G; an embedding model belongs on CPU. - Check **context length** and **intended use** in the card — the headline number is not always the usable one. ## Dedicated Inference Endpoints — when to graduate Move off the router when you need fixed latency/SLA, or the router's PAYG cost stops being predictable. An Endpoint is your own autoscaling deployment. - Pricing: CPU from ~$0.032/core/hr, GPU from ~$0.50/hr (A10G ~$1.00/hr, H100 ~$6.40–8.00/hr), **billed per minute** even though shown hourly. - Enable **scale-to-zero** for bursty traffic — it parks at $0 when idle and cold-starts on the next request. A bursty 100–1000 req/day workload typically lands at **$20–60/mo**. - Deploy from the UI or with `huggingface_hub` (`create_inference_endpoint(...)`). Config and a cost worksheet are in `references/endpoints-and-spaces.md`. ## Spaces + ZeroGPU A Space hosts a demo app with a public URL. **ZeroGPU** grabs an H200 MIG slice (~70GB) only while a decorated function runs, then releases it. ```python import spaces @spaces.GPU # GPU acquired for this call only def generate(prompt: str) -> str: ... ``` - **ZeroGPU is Gradio-SDK only** — Streamlit/Docker/static Spaces cannot use it. PRO ($9/mo) gives 8x daily quota, queue priority, and up to 10 owned ZeroGPU Spaces. Details in `references/endpoints-and-spaces.md`. ## Local transformers ```python from transformers import pipeline pipe = pipeline("text-generation", model="meta-llama/Llama-3.1-8B-Instruct", device_map="auto", torch_dtype="auto") print(pipe("Hello", max_new_tokens=64)[0]["generated_text"]) ``` - `pipeline("task", model=...)` for quick use; `AutoModelForCausalLM.from_pretrained(...)` when you need control over generation/quantization. Set `device_map`/`torch_dtype` explicitly. - Use local only when you are **offline, data-private, or already on a GPU**. Otherwise the router is far less ops than babysitting CUDA and weights. ## Anti-patterns | Anti-pattern | Why it bites | Do instead | |---|---|---| | `InferenceClient.post(...)` | Removed in hub v0.31.0; raises | Task methods: `chat.completions.create()`, `feature_extraction()` | | `provider="hf-inference"` for a 70B/405B LLM | CPU niche; 404s or stalls | Route to a partner provider (Together/Fireworks/Cerebras) | | `api_key="hf_abc123..."` in code | Token leaks in git history | Read `os.environ["HF_TOKEN"]` | | Spin up a dedicated Endpoint just to try a model | Burns money idle | Use the router first; graduate only on real traffic | | Assuming router calls are free/unlimited | Free tier is $0.10/mo | Budget credits; expect PAYG | | ZeroGPU under Streamlit/Docker SDK | Unsupported, silently no GPU | Use the Gradio SDK | | `huggingface-cli ...` in new scripts | Deprecated, warns | Use `hf ...` | | OpenAI base URL other than `https://router.huggingface.co/v1` | Won't reach the HF router | Use that exact host | ## verify.sh `scripts/verify.sh [TARGET]` is a static, read-only linter (no network, no token). It flags the hard violations above — `.post(`, hardcoded `hf_` tokens, big-LLM-to-`hf-inference`, wrong router host — and warns on legacy `huggingface-cli`. It exits 0 on a clean or empty target.