メニュー
Launch and configure vLLM-Omni API servers for production model serving. Use when starting a model server, configuring stage pipelines, setting up GPU memory, enabling optimizations, or deploying models behind a load balancer.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
| name | vllm-omni-serving |
| description | Launch and configure vLLM-Omni API servers for production model serving. Use when starting a model server, configuring stage pipelines, setting up GPU memory, enabling optimizations, or deploying models behind a load balancer. |
vLLM-Omni serves models via an OpenAI-compatible HTTP server. It supports autoregressive models (text, omni), diffusion models (image, video), and TTS models (audio) through a unified vllm serve command with the --omni flag.
vllm serve <model-name> --omni --port 8091
Examples by modality:
# Image generation
vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091
# Omni-modality (text + image + audio)
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
# TTS
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
# Video generation
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091
| Argument | Description | Example |
|---|---|---|
--omni | Enable omni-modality pipeline (required) | --omni |
--port | HTTP port | --port 8091 |
--host | Bind address | --host 0.0.0.0 |
--gpu-memory-utilization | Fraction of GPU memory to use | --gpu-memory-utilization 0.85 |
--tensor-parallel-size | Number of GPUs for tensor parallelism | --tensor-parallel-size 2 |
--pipeline-parallel-size | Pipeline parallelism stages | --pipeline-parallel-size 2 |
--max-model-len | Maximum sequence length | --max-model-len 4096 |
--dtype | Model dtype | --dtype float16 |
vLLM-Omni uses stage configs to define multi-stage pipelines. Each model has default stage configs, but you can customize them:
vllm serve Qwen/Qwen2.5-Omni-7B --omni \
--stage-configs-path ./my-stage-config.yaml
Stage config structure:
stages:
- name: "encoder"
stage_type: "ar"
stage_args:
runtime:
max_batch_size: 4
- name: "diffusion"
stage_type: "diffusion"
stage_args:
runtime:
max_batch_size: 1
The max_batch_size for diffusion stages defaults to 1. Increase it only for models that support batched diffusion.
Calculate memory needs based on model size and desired throughput:
# Conservative (80% GPU memory)
vllm serve <model> --omni --gpu-memory-utilization 0.8
# Aggressive (95% for maximum throughput)
vllm serve <model> --omni --gpu-memory-utilization 0.95
Split model across multiple GPUs:
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--tensor-parallel-size 4 --port 8091
For very large models:
vllm serve <model> --omni \
--tensor-parallel-size 2 \
--pipeline-parallel-size 2
--host 0.0.0.0 for external access--gpu-memory-utilization based on model size--max-model-len--disable-log-requests for reduced I/O overhead/healthnvidia-smi dmonRun separate server instances on different ports:
# Terminal 1: Image generation
vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091
# Terminal 2: Text/Omni
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8092
Use a reverse proxy to route by path or model name.
Server fails to start: Check GPU memory availability with nvidia-smi. Reduce --gpu-memory-utilization or choose a smaller model.
Slow first request: Model weights are loaded lazily. The first request triggers full model initialization. Subsequent requests are fast.
Connection refused: Verify --host and --port settings. Default host is 127.0.0.1 (localhost only).
--dtype ignored with default stage configs: When using default stage configs (no --stage-configs-path), the --dtype arg was silently dropped from diffusion stage engine args. Fixed in #2530 — dtype now correctly propagates from CLI.
--stage-init-timeout not respected: User-configured stage init timeout was being overridden. Default is now 300s (server-side). Pass --stage-init-timeout <seconds> to customize. Fixed in #2519.
OOM errors produce no response: Diffusion pipeline OOM and execution errors now return structured HTTP error responses (e.g., 507) with request_id, stage_id, and error_type fields instead of hanging. Uses OmniRequestError dataclass for end-to-end propagation. Fixed in #2638.
DiffusionEngine.close() hangs or leaks resources: Fixed in #3494. Close now properly waits for worker thread and completes pending futures with errors.
HunyuanImage3 deploy config fails at startup: Fixed in #3537. Pipeline name changed from hunyuan_image3 to hunyuan_image_3_moe; inter-stage connectors default to rdma_connector.
Multimodal cache miss across AR replicas: Fixed in #3605. Multimodal UUIDs are now scoped per stage-0 replica to prevent the sender from skipping tensor transfers.
Review PRs on vllm-project/vllm-omni by routing to the right domain skills, checking critical evidence, and focusing comments on blocking issues. Use when reviewing pull requests or local branches, triaging review depth, running detailed or default review, or checking tests, benchmarks, and breaking changes in vllm-omni.
Two report kinds; **default output is always HTML** unless the user explicitly asks for Markdown (.md). **Release** — `scripts/compose_full_report.py` (**测试结论**, Buildkite metrics, **Test Result** = Common stack + optional `--log-dir-h*` nightly-style summaries + H100/CI block, **Issue tracking** = GitHub `ci-failure` + *local test* in:title, Open bugs); use `--format markdown` only when the user wants .md or `patch_report_*.py`. **Nightly** — `scripts/nightly_local_log_report.py` from local `nightly_jobs` (fetch: vllm-omni-nightly-local) plus optional latest Buildkite scheduled nightly when token is set; use `--markdown-report` / `--to-stdout markdown` only when the user asks for Markdown. Use when generating Buildkite release summaries, parsing local nightly_jobs with CI cross-check, or opening https://buildkite.com/vllm/vllm-omni/builds?branch=main for CI documentation.
Use before submitting a PR to vllm-project/vllm-omni — self-check the branch against project conventions, catch dead code, verify accuracy/performance claims, and confirm merge readiness. Use when the user says "pre-check", "self review", "pre-submit check", or "check my PR before I open it."
Generate audio and speech with vLLM-Omni using Qwen3-TTS, Fish Speech S2 Pro, CosyVoice3, MiMo-Audio, and Stable-Audio models. Use when synthesizing speech from text, generating audio effects or music, configuring TTS parameters, cloning voices, adding new TTS models, or working with text-to-speech models.
Contribute to vLLM-Omni by adding new model support, fixing bugs, or improving features. Use when integrating a new model into vllm-omni, setting up a development environment, writing tests, or submitting pull requests to the vllm-omni project.
Scale vLLM-Omni across multiple GPUs and nodes using tensor parallelism, pipeline parallelism, OmniConnector disaggregation, connector backends, and Ray. Use when setting up multi-GPU inference, distributing model execution across machines, deploying disaggregated execution, developing OmniConnector backends, or scaling inference horizontally.