vllm-omni

name	vllm-omni
description	vLLM-Omni — serving de modèles omni-modalité (texte, image, vidéo, audio) sur GPU/NPU CUDA/ROCm via API OpenAI unifiée, étend vLLM aux Diffusion Transformers (DiT). NON supporté sur Apple Silicon (utiliser mflux/mlx-video/mlx-audio). Charge on-demand sur fichiers vllm-omni.
paths	["**/*vllm*omni*","**/stage-config*"]

vLLM-Omni — Omni-Modality Model Serving (rule on-demand)

Provenance : transféré du skill intellisoins-mlx:vllm-omni (SKILL.md) le 2026-05-24. Détail catalogue modèles + API → ~/.claude/rules/vllm-omni/models-and-apis.md. ⚠️ GPU/NPU Linux uniquement — pas Apple Silicon. Pour génération image/vidéo/audio sur Mac, voir mflux / mlx-video / mlx-audio. Distinct de vmlx, vllm-mlx, vllm-metal (eux sur Apple Silicon).

Framework for efficient serving of omni-modality models: text, image, video, and audio generation via a unified OpenAI-compatible API. Extends vLLM beyond autoregressive LLMs to support Diffusion Transformers (DiT) and non-autoregressive architectures.

Upstream: https://github.com/vllm-project/vllm-omni (official) Fork: https://github.com/Blaizzy/vllm-omni (Prince Canuma / Blaizzy) Docs: https://vllm-omni.readthedocs.io/ License: Apache 2.0 Version: v0.18.0 (stable, 2026-03-28) — rebased to upstream vLLM v0.18.0 Latest prerelease: v0.20.0rc1 (2026-05-01) — rebased to vLLM v0.20.0

Platform Requirements

Requirement	Value
OS	Linux
Python	3.12
GPU	NVIDIA CUDA or AMD ROCm
NPU	Supported (subset of models)
Apple Silicon	Not supported (use vllm-mlx or vllm-metal skills instead)

Installation

uv venv --python 3.12 --seed
source .venv/bin/activate

# CUDA
uv pip install vllm==0.18.0 --torch-backend=auto

# ROCm
uv pip install vllm==0.18.0 --extra-index-url https://wheels.vllm.ai/rocm/0.18.0/rocm700

git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
uv pip install -e .

Supported Models Overview

Omni-Modality (text + image + audio in/out)

Model	HuggingFace	Notes
Qwen3-Omni-30B-A3B	Qwen/Qwen3-Omni-30B-A3B-Instruct	MoE, latest
Qwen2.5-Omni-7B/3B	Qwen/Qwen2.5-Omni-7B	Multi-modal I/O
MammothModa2	bytedance-research/MammothModa2-Preview	Omni-modal (v0.18.0+)
Ming-flash-omni-2.0	inclusionAI/Ming-flash-omni-2.0	Omni-modal flash (v0.20.0rc1+)
Dynin-omni	snu-aidas/Dynin-Omni	Omni-modal (v0.20.0rc1+)

Image Generation (Diffusion)

Model	HuggingFace	Type
Z-Image Turbo	Tongyi-MAI/Z-Image-Turbo	Fast DiT
Qwen-Image	Qwen/Qwen-Image	T2I
Qwen-Image-2512	Qwen/Qwen-Image-2512	T2I high-res
BAGEL-7B-MoT	ByteDance-Seed/BAGEL-7B-MoT	DiT-only MoT
Ovis-Image	OvisAI/Ovis-Image	T2I
LongCat-Image	meituan-longcat/LongCat-Image	T2I
Stable Diffusion 3.5	stabilityai/stable-diffusion-3.5-medium	DiT
FLUX.2-klein 4B/9B	black-forest-labs/FLUX.2-klein-4B	DiT
FLUX.1-dev	black-forest-labs/FLUX.1-dev	DiT
FLUX.2-dev	black-forest-labs/FLUX.2-dev	DiT (v0.18.0+)
FLUX.1-Kontext-dev	black-forest-labs/FLUX.1-Kontext-dev	DiT context-aware (v0.18.0+)
Hunyuan Image3 AR	tencent/HunyuanImage-3.0	Tencent Hunyuan AR (v0.18.0+, closest officiel)

Image Editing

Model	HuggingFace
Qwen-Image-Edit	Qwen/Qwen-Image-Edit
Qwen-Image-Edit-2509	Qwen/Qwen-Image-Edit-2509
Qwen-Image-Layered	Qwen/Qwen-Image-Layered
LongCat-Image-Edit	meituan-longcat/LongCat-Image-Edit

Video Generation

Model	HuggingFace
Wan2.2-T2V-A14B	Wan-AI/Wan2.2-T2V-A14B-Diffusers
Wan2.2-TI2V-5B	Wan-AI/Wan2.2-TI2V-5B-Diffusers
Wan2.2-I2V-A14B	Wan-AI/Wan2.2-I2V-A14B-Diffusers
LTX-2	Lightricks/LTX-Video (v0.18.0+, repo officiel Lightricks LTX)
LTX-2.3	Lightricks/LTX-2.3 (v0.20.0rc1+)
HunyuanVideo-1.5	tencent/HunyuanVideo-1.5 (v0.18.0+)
FastGen Wan 2.1	(v0.20.0rc1+ — HF ID NON VERIFIE)

Audio

Model	HuggingFace
Stable-Audio-Open	stabilityai/stable-audio-open-1.0
Qwen3-TTS 1.7B/0.6B	Qwen/Qwen3-TTS-12Hz-1.7B-Base
Voxtral TTS	mistralai/Voxtral-4B-TTS-2603 (v0.18.0+)
MiMo-Audio	XiaomiMiMo/MiMo-Audio-7B-Base (v0.18.0+)
Fish Speech S2 Pro	AEmotionStudio/fish-speech-s2-pro (v0.18.0+, 3rd party fork)
MiMo-V2.5-ASR	XiaomiMiMo/MiMo-V2.5-ASR (v0.20.0rc1+)
MOSS-TTS-Nano	OpenMOSS-Team/MOSS-TTS-Nano-100M (v0.20.0rc1+)
VoxCPM2 native AR TTS	openbmb/VoxCPM2 (v0.20.0rc1+)

For complete model catalog with NPU support: ~/.claude/rules/vllm-omni/models-and-apis.md

Quick Start — Offline Inference

from vllm_omni.entrypoints.omni import Omni

omni = Omni(model="Tongyi-MAI/Z-Image-Turbo")
outputs = omni.generate("a cup of coffee on the table")
outputs[0].request_output[0].images[0].save("coffee.png")

Quick Start — Online Serving

Launch Server

vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8000

The --omni flag activates the omni-modality pipeline (required for all non-LLM models).

Image Generation API (OpenAI DALL-E compatible)

curl -X POST http://localhost:8000/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "a dragon over Green Mountains",
    "size": "1024x1024",
    "num_inference_steps": 50,
    "guidance_scale": 4.0,
    "seed": 42
  }' | jq -r '.data[0].b64_json' | base64 -d > dragon.png

Image Edit API

vllm serve Qwen/Qwen-Image-Edit-2511 --omni --port 8000

curl -X POST http://localhost:8000/v1/images/edits \
  -F "image=@input.png" \
  -F "prompt='make the bear wear sportswear'" \
  -F "size=1024x1024"

Chat Completions API (Omni models)

vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8000

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "describe this scene"}],
    "extra_body": {"height": 1024, "width": 1024}
  }'

Architecture — OmniConnector

vLLM-Omni introduces a heterogeneous pipeline abstraction via OmniConnector:

• Stage-based execution: Models decomposed into stages with pipelined overlapping
• Dynamic resource allocation: GPU memory managed across stages
• KV cache management: Inherited from vLLM for autoregressive components
• Distributed inference: Tensor, pipeline, data, and expert parallelism

Configure stage behavior via stage_configs_path YAML:

omni = Omni(model="...", stage_configs_path="./stage-config.yaml")

What's New v0.18.0 (2026-03-28)

• Rebased to upstream vLLM v0.18.0 — major entrypoint refactor (PD disaggregation scaffolding, coordinator support, multimodal output decoupling), scheduler/executor refactoring + runtime cleanups.
• Audio/Speech/Omni production hardening: Qwen3-TTS, Qwen3-Omni, MiMo-Audio, Fish Speech S2 Pro, Voxtral TTS — lower latency, better concurrency, robust streaming.
• Diffusion optim: cache-dit / TeaCache integration, TP/SP/HSDP support, faster startup.
• Unified quantization framework: INT8, FP8, GGUF coverage across diffusion + image workloads.
• RL E2E: Qwen-Image end-to-end RL with verl + Flow-GRPO training, collective RPC support.
• New models: Helios, Helios-Mid/Distilled, MammothModa2, Fun CosyVoice3-0.5B-2512, FLUX.2-dev, FLUX.1-Kontext-dev, Hunyuan Image3 AR, DreamID-Omni, LTX-2, HunyuanVideo-1.5.

Coming in v0.20.0rc1 (2026-05-01, prerelease)

• Rebased to upstream vLLM v0.20.0 + sleep-mode support + coordinator reliability fixes.
• TTS hardening: lower VRAM, CUDA graph reuse, voice-cloning fixes, deterministic Fish Speech, universal TTS benchmark.
• Diffusion: Z-Image image-to-image, FLUX.1/FLUX.2 TeaCache + CFG-parallel paths, VAE tiling, MP4 latency optimization.
• BAGEL: LoRA support, think mode, fused projections, layerwise offload, diffusion metrics.
• HW broadening: CUDA + ROCm + MUSA flash attention + NPU graph/fused-op + AMD CI + XPU torch inductor + OmniGen2 FP8 + HunyuanImage3 NPU quantization.
• New models: MagiHuman, Dynin-omni, InternVLA-A1, Ming-flash-omni-2.0, MiMo-V2.5-ASR, MOSS-TTS-Nano, VoxCPM2 native AR TTS, LTX-2.3, FastGen Wan 2.1.

API Extension Parameters

Beyond standard OpenAI parameters, vLLM-Omni adds:

Parameter	Type	Description
negative_prompt	string	What to avoid in generation
num_inference_steps	int	Diffusion steps (more = higher quality, slower)
guidance_scale	float	CFG scale (0.0-20.0 typical)
true_cfg_scale	float	Model-specific true CFG
seed	int	Reproducibility

Relation to Other Skills/Rules

Cible	Target	Use Case
vllm-omni (this)	GPU/NPU (CUDA, ROCm)	Omni-modal: images, video, audio, diffusion
vllm-mlx	Apple Silicon MLX	LLM/VLM serving on Mac
vllm-metal	Apple Silicon Metal	vLLM with Metal GPU plugin
vmlx	Apple Silicon MLX	Moteur d'inférence MLX local complet
mflux	Apple Silicon MLX	Image generation (FLUX, Z-Image) locally
mlx-video	Apple Silicon MLX	Video generation (LTX, Wan) locally

Key distinction: vLLM-Omni requires CUDA/ROCm GPU. For Apple Silicon image/video/audio generation, use mflux, mlx-video, or mlx-audio instead.

Additional Resources

~/.claude/rules/vllm-omni/models-and-apis.md — Complete model catalog (GPU + NPU), API endpoint reference for image generation, image editing, chat completions, extension parameters, Python SDK examples, error handling, and stage configuration.