serve-on-jobs

name	serve-on-jobs
description	Serve any Hugging Face Hub model as a temporary OpenAI-compatible endpoint using HF Jobs + vLLM, so you can evaluate/benchmark it, iterate prompts, or hit it from an agent — without a permanent deployment. Use when a model is NOT on Inference Providers (e.g. a research model like datalab-to/lift) and you just need an endpoint for an afternoon. Always cancel the job when done so billing stops.

Serve a model on HF Jobs (jobs-serving) for eval

HF Jobs can run vLLM (or any HTTP server) as a temporary inference server with an exposed port: you get an OpenAI-compatible endpoint, pay per minute, and the endpoint disappears when you cancel the job. Ideal when the endpoint is a means (an eval run, a labelling session, prompt iteration), not the product. Docs: https://huggingface.co/docs/hub/jobs-serving

1. Start the server

hf jobs run --detach --expose 8000 --flavor <FLAVOR> -s HF_TOKEN \
  vllm/vllm-openai \
  vllm serve <MODEL_ID> --max-model-len <N>

• Returns a job id and an exposed URL https://<job_id>--8000.hf.jobs. The OpenAI base URL is that + /v1.
• -s HF_TOKEN forwards your token (required for gated/private models; faster downloads for all).
• Jobs runs the command literally — spell out the full vllm serve ... (it does NOT pass args to an entrypoint).
• Flavor by model size: ≤~8B → a10g-large/l4x1 (24 GB); ~9–30B → a100-large (80 GB); bigger → check hf jobs hardware. When unsure, a100-large is safe.
• --max-model-len for vision/document models: image tokens are large — set 16384–32768, or you get truncation / [OCR ERROR]-style failures. Don't leave the 8192 default when sending images.

2. Wait until ready

Takes a few minutes (image pull + model download + load). Ready when the logs show Application startup complete:

hf jobs logs -f davanstrien/<job_id>
# or poll:  hf jobs logs davanstrien/<job_id> --tail 15 | grep -q "Application startup complete"

Gotcha: hf jobs logs no-ops while the job is still SCHEDULING/queuing. Poll hf jobs inspect <id> until it reaches RUNNING, then tail logs for the startup line.

3. Call it (OpenAI-compatible)

The HF token rides in the Authorization header, so it IS the api_key:

import os
from openai import OpenAI
client = OpenAI(base_url="https://<job_id>--8000.hf.jobs/v1", api_key=os.environ["HF_TOKEN"])
r = client.chat.completions.create(
    model="<MODEL_ID>",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "..."},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},  # for VLMs
    ]}],
    extra_body={"guided_json": <json_schema>},  # vLLM schema-constrained decoding (optional)
)

• GGUF instead? ghcr.io/ggml-org/llama.cpp:server-cuda -- /app/llama-server -hf <repo> --host 0.0.0.0 --port 8080.
• Any OpenAI-compatible server works (SGLang, …) — just listen on 0.0.0.0 (the jobs proxy can't reach 127.0.0.1).

4. ALWAYS cancel when done — billing stops with the job

hf jobs cancel davanstrien/<job_id>

Default timeout is 30 min (--timeout for longer). Best practice: a watcher that waits for Application startup complete → runs your eval → cancels, so the GPU is never billed idle:

JOB=davanstrien/<job_id>
until hf jobs logs $JOB --tail 15 2>/dev/null | grep -q "Application startup complete"; do sleep 20; done
HF_TOKEN=$(hf auth token) uv run your_eval.py   # hits the endpoint
hf jobs cancel $JOB

In this repo

glam_bench/models.py has a commented lift-9B entry showing this exact path: serve datalab-to/lift with the command above, set its base_url to the exposed URL + /v1, and the kind:"openai" adapter (with guided:True) benchmarks it. See the run-benchmark skill.