메뉴
Run vLLM Buildkite CI-equivalent tests locally on NVIDIA GPUs using the current shell environment. Use when the user asks to run CI tests locally, reproduce CI failures, run a specific test file or test area, or match Buildkite test behavior.
기본적으로 소스를 먼저 확인하는 Prompt가 선택됩니다. 직접 명령으로 전환하거나 로컬 사본을 다운로드할 수도 있습니다.
설치 여부를 결정하기 전에 SKILL.md와 SkillsMP에 표시된 보조 파일을 읽어 보세요.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
직접 명령은 검토 Prompt를 거치지 않습니다. 실행하기 전에 소스를 확인하세요.
npx skills add https://github.com/cohere-ai/vllm-skills --skill local-test-runner명령은 한 줄로 유지됩니다. 복사하기 전에 가로로 스크롤해 전체 내용을 확인하세요.
로컬 사본을 원하시나요? SkillsMP에서 현재 제공할 수 있는 파일을 다운로드하세요.
SOC 직업 분류 기준
SKILL.md 표시 중
| name | local-test-runner |
| description | Run vLLM Buildkite CI-equivalent tests locally on NVIDIA GPUs using the current shell environment. Use when the user asks to run CI tests locally, reproduce CI failures, run a specific test file or test area, or match Buildkite test behavior. |
Run vLLM tests locally — either a specific test file or a full Buildkite test area — directly in the current shell with an editable vLLM install.
| Buildkite concept | Local equivalent |
|---|---|
test_areas/*.yaml steps | pytest commands from those YAML files, run in current env |
CI working dir /vllm-workspace/tests | <repo_root>/tests |
Buildkite sharding (--num-shards/--shard-id) | Omit locally (run full suite) |
$$BUILDKITE_PARALLEL_JOB vars in YAML | Remove or ignore |
In containerized environments, uv may fail with Cross-device link (os error 18) when its cache and workspace are on different filesystems. If this happens, set the cache to a local directory before any uv commands:
export UV_CACHE_DIR=<repo_root>/.cache/uv
Check that vLLM is fully installed. An import vllm check alone is
insufficient: when Python is run from the repo root, the vllm/ source
directory shadows the package, so imports can succeed even when
uv pip install -e . was never run. In that state there is no
console script in , and entrypoint tests that spawn
via fail with a confusing
(an empty PATH
entry causes the exec loop to land on the source directory). All three
of these checks must pass:
vllm.venv/bin/vllm serveRemoteOpenAIServerPermissionError: [Errno 13] Permission denied: 'vllm'.venv/bin/python -c "import vllm; print('vllm version:', vllm.__version__)"
test -x .venv/bin/vllm && echo "vllm CLI OK"
uv pip show vllm > /dev/null 2>&1 && echo "pip metadata OK"
If any of the three checks fails, use the install-vllm skill to create
the env and install vLLM — do not try to patch around a partial
install. Then install test dependencies:
uv pip install -r requirements/test.in
uv pip install -r requirements/dev.txt
Always prompt the user for which precompiled wheel to use, unless they have already specified one. The precompiled wheel determines which C++/CUDA binaries are loaded. Using the wrong wheel can cause silent correctness regressions (e.g. a model producing 100% WER instead of ~12%).
.venv/bin/python -c "import vllm; print(vllm.__version__)"
git rev-parse --abbrev-ref HEAD # e.g. "v0.19.0-branch" → release 0.19.0
Prompt the user with the currently installed wheel version, the current branch name, the inferred matching release version (if any), and these options:
install-vllm skill, Option B)VLLM_USE_PRECOMPILED=1 (via install-vllm skill, Option A)Wait for the user's answer before proceeding. Do not assume a default and do not run tests until the wheel choice is confirmed.
Run:
nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu --format=csv,noheader
Map common strings: H100 → h100, H200 → h200, A100 → a100, B200 → b200, GB200 → gb200.
Show utilization and suggest which GPU indices appear free. If all are idle, default to all and skip CUDA_VISIBLE_DEVICES. If some are busy, suggest restricting to idle indices.
Two modes — pick based on user input:
A) Specific test file or command (user provides a path or pytest command):
tests/v1/core/test_scheduler.py or pytest -v -s basic_correctness/test_basic_correctness.py -k test_add_requestsB) Buildkite test area (user names an area):
test_area: one of attention, basic_correctness, benchmarks, compile, cuda, distributed, e2e_integration, engine, entrypoints, expert_parallelism, kernels, lm_eval, lora, misc, model_executor, model_runner_v2, models_basic, models_distributed, models_language, models_multimodal, plugins, pytorch, quantization, ray_compat, samplers, spec_decode, weight_loading.buildkite/test_areas/<test_area>.yaml and extract the commands: list. If there are multiple steps (different device requirements or GPU counts), show them and let the user pick.$$BUILDKITE_PARALLEL_JOB, $$BUILDKITE_PARALLEL_JOB_COUNT) and --num-shards / --shard-id flags.export lines (especially VLLM_WORKER_MULTIPROC_METHOD=spawn).uv pip install or pip install lines from the YAML before the test commands.Optional parameters for both modes:
cuda_visible_devices: comma-separated GPU indices (defaults to all)num_gpus: for test areas, read from YAML num_devices field (default 1)In Buildkite CI, HF_TOKEN is pre-set on agents (via secrets/hooks or /etc/environment) and passed through to containers with -e HF_TOKEN. There is no YAML-level declaration — scripts simply assume it exists in the host environment.
Skip this step for unit tests that don't download models (e.g. scheduler, core, engine unit tests).
Only needed when tests load models from Hugging Face (model tests, lm_eval, benchmarks, etc.):
HF_TOKEN is set without printing it: test -n "${HF_TOKEN:-}" && echo "HF_TOKEN is set".~/.cache/huggingface/token). If it exists and is non-empty, export it: export HF_TOKEN=$(cat ~/.cache/huggingface/token)huggingface-cli login and then retry.Always use .venv/bin/python -m pytest — source .venv/bin/activate does not persist across agent shell calls.
Important: Always add .venv/bin to PATH. Some tests (e.g. entrypoint tests using RemoteOpenAIServer) spawn vllm serve as a subprocess and expect the vllm CLI to be on PATH. Without this, those tests fail with PermissionError: [Errno 13] Permission denied: 'vllm'.
cd <repo_root>
export PATH="<repo_root>/.venv/bin:$PATH"
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export CUDA_VISIBLE_DEVICES=<cuda_visible_devices> # omit if using all GPUs
# Only if needed:
export HF_HOME=~/.cache/huggingface
export HF_TOKEN="<hf_token>"
.venv/bin/python -m pytest <test_path_or_commands> -v \
2>&1 | tee .cache/test-logs/<test_name>-$(date +%Y%m%d-%H%M%S).log
Useful pytest flags for local runs:
-x — stop on first failure (add when debugging a specific failure)-k <expression> — run only tests matching a name pattern-s — disable output capture (show prints)--timeout=<seconds> — per-test timeoutSave test output under <repo_root>/.cache/test-logs/. Create the directory if it doesn't exist:
mkdir -p <repo_root>/.cache/test-logs
Log file naming: <test-name>-<YYYYMMDD-HHMMSS>.log, e.g. test_scheduler-20260421-175200.log.
Use tee to both display output and write to the log file:
.venv/bin/python -m pytest tests/v1/core/test_scheduler.py -v \
2>&1 | tee .cache/test-logs/test_scheduler-$(date +%Y%m%d-%H%M%S).log
For test area runs, name the log after the area: basic_correctness-20260421-175200.log.
After the run, report the log file path to the user.
Tests can take minutes to hours. Run them as background commands (block_until_ms: 0) and monitor by polling the terminal output file. Check periodically with exponential backoff until the exit code footer appears.
commands: use paths relative to /vllm-workspace/tests. Locally these are relative to <repo_root>/tests.working_dir: "/vllm-workspace/" — locally that maps to <repo_root>/.num_devices.mkdir -p .cache/test-logs.venv/bin/python -m pytest tests/v1/core/test_scheduler.py -v 2>&1 | tee .cache/test-logs/test_scheduler-<timestamp>.log.buildkite/test_areas/basic_correctness.yaml, show extracted commands