| name | bump-vllm-pins |
| description | Internal, dev-only: smoke-test and bump the pinned vLLM recipes in this repo. Use periodically (~quarterly) or after a vLLM release to confirm each pinned vLLM script still runs on Jobs and to propose pin / image-tag bumps to a newer known-good. Tests on l4x1 via `hf jobs uv run`; reports a diff of proposed bumps; never auto-merges. |
bump-vllm-pins
Pins give stability; this prevents staleness. Internal/dev-only, not shipped.
vLLM facts to apply (verified 2026-06)
- Structured outputs: use
StructuredOutputsParams (structured_outputs=), floor vllm>=0.11.0. GuidedDecodingParams / guided_decoding= was removed in 0.12.0 → ImportError on ≥0.12.0 (incl. 0.22.x). Param kinds carry over: choice=, json=, regex=, grammar=.
- FlashInfer / nvcc at init: recent FlashInfer JIT-compiles kernels needing the CUDA toolkit. Prefer running on the
vllm/vllm-openai image (ships the toolkit) over the VLLM_USE_FLASHINFER_SAMPLER=0 env opt-out. (The official vllm-project/vllm-skills docker skill likewise recommends this image.)
- Driver / flavor: CUDA-13 builds need a newer driver.
a100-large's driver caps at CUDA 12.9 (found version 12090) → CUDA-13 wheels/images fail there; l4x1 works. Prefer a CUDA-12.x image for broad flavor compat, or test on l4x1.
- Pinning: pin the image tag (
vllm/vllm-openai:<ver>, not :latest) for the strongest reproducibility (freezes vLLM + toolkit together); also pin vllm== in the PEP 723 header for the local path. vLLM releases ~monthly.
Procedure (per pinned script)
- [ ] 1. Smoke-test the current pin on l4x1 (image, small --max-samples, public dataset)
- [ ] 2. If green, try the latest vLLM image tag / pin; re-test
- [ ] 3. Record pass/fail (+ toks/s); propose the bump or hold
- [ ] 4. Report a diff of pin / image-tag changes for review — do NOT auto-merge
- Discover pinned scripts:
grep -rnE "vllm" */*.py and the documented --image vllm/vllm-openai:<tag> in docstrings.
- Smoke-test (template):
hf jobs uv run -d --flavor l4x1 --secrets HF_TOKEN \
--image vllm/vllm-openai:<tag> --python /usr/bin/python3 \
-e PYTHONPATH=/usr/local/lib/python3.12/dist-packages \
<script.py> <public-input-dataset> <out> --max-samples 5
Poll hf jobs inspect <id> --format json. (--python /usr/bin/python3 -e PYTHONPATH=/usr/local/lib/python3.12/dist-packages make uv use the image's preinstalled Python 3.12 + vLLM instead of fetching its own — required for the vllm/vllm-openai image; verified to run clean on l4x1.)
- Green = runs to completion + writes output. Flag failures by class:
ImportError (API drift → re-harden to the new API), nvcc/FlashInfer (toolkit/image), or "driver too old" (CUDA-13-vs-flavor).
- Propose bumps (image tag +
vllm==) as a reviewable diff; maintainer merges.
Capture-then-pin
Pin to a tested version, never a guessed one. Workflow: run loose (or on :latest) → read the working versions from the green run (vLLM logs its version at engine init; or uv pip freeze / uv lock --script <file> inside the job) → pin to those. Lead with the image tag (vllm/vllm-openai:<that-version>) — it freezes vLLM + torch + CUDA toolkit + FlashInfer together — and keep a vllm==/>= floor in the PEP 723 header for the local path.
Output
A short report: per script — current pin, tested tag, pass/fail, proposed bump. No file changes without review.
