vllm-multimodal

name	vllm-multimodal
description	Vision, audio, and video model inference
triggers	["When user wants to use vision-language models (VLM)","When user needs audio processing (speech recognition)","When user wants video understanding capabilities","When user needs to handle multimodal inputs"]

vllm-multimodal

Overview

vLLM supports multimodal models for vision, audio, and video understanding. This skill covers using VLMs (Vision Language Models), audio models, and video processing.

Prerequisites

• vLLM 0.5.0+ installed
• GPU with sufficient memory (VLMs typically need more VRAM)
• Input media files (images, audio, video)

Main Workflow

Step 1: Vision Model Inference

Supported Models: LLaVA, Qwen-VL, InternVL, Phi-3-Vision

from vllm import LLM, SamplingParams
from PIL import Image

# Load vision model
llm = LLM(model="llava-hf/llava-1.5-7b-hf")

# Load image
image = Image.open("image.jpg")

# Prepare prompt with image placeholder
prompt = "USER: <image>\nWhat is shown in this image?\nASSISTANT:"

# Generate
sampling_params = SamplingParams(max_tokens=200)
output = llm.generate(
    {"prompt": prompt, "multi_modal_data": {"image": image}},
    sampling_params
)
print(output[0].outputs[0].text)

Step 2: Multi-Image Inference

from vllm import LLM
from PIL import Image

llm = LLM(model="Qwen/Qwen-VL-Chat")

# Load multiple images
images = [Image.open(f"image_{idx}.jpg") for idx in range(3)]

prompt = "USER: <image><image><image>\nCompare these three images.\nASSISTANT:"

output = llm.generate(
    {"prompt": prompt, "multi_modal_data": {"image": images}},
        SamplingParams(max_tokens=300)
)

Step 3: Audio Model Inference

Supported Models: Whisper, Qwen-Audio

from vllm import LLM, SamplingParams

# Load audio model
llm = LLM(model="openai/whisper-large-v3")

# Prepare audio file
audio_path = "audio.wav"
prompt = {"prompt": "Transcribe this audio:", "multi_modal_data": {"audio": audio_path}}

output = llm.generate(prompt, SamplingParams(max_tokens=200))
transcription = output[0].outputs[0].text
print(transcription)

Step 4: Video Model Inference

from vllm import LLM, SamplingParams
import cv2

# Load video model (e.g., Video-LLaVA)
llm = LLM(model="LanguageBind/Video-LLaVA-7B-hf")

# Extract frames from video
cap = cv2.VideoCapture("video.mp4")
frames = []
frame_count = 0
while cap.isOpened() and frame_count < 8:  # Sample 8 frames
    ret, frame = cap.read()
    if not ret:
        break
    frames.append(frame)
    frame_count += 1
cap.release()

prompt = "USER: <video>\nDescribe what happens in this video.\nASSISTANT:"

output = llm.generate(
    {"prompt": prompt, "multi_modal_data": {"video": frames}},
    SamplingParams(max_tokens=300)
)

Common Patterns

Pattern 1: Vision Q&A

from vllm import LLM, SamplingParams
from PIL import Image

llm = LLM(model="llava-hf/llava-1.5-7b-hf")

def answer_question(image_path, question):
    image = Image.open(image_path)
    prompt = f"USER: <image>\n{question}\nASSISTANT:"

    output = llm.generate(
        {"prompt": prompt, "multi_modal_data": {"image": image}},
        SamplingParams(temperature=0.2, max_tokens=150)
    )
    return output[0].outputs[0].text.strip()

# Usage
answer = answer_question("chart.png", "What is the trend shown in this chart?")

Pattern 2: OCR (Text Extraction)

from vllm import LLM, SamplingParams
from PIL import Image

llm = LLM(model="Qwen/Qwen2-VL-7B-Instruct")

image = Image.open("document.jpg")
prompt = "Extract all text from this image. Return only the text content."

output = llm.generate(
    {
        "prompt": f"USER: <image>\n{prompt}\nASSISTANT:",
        "multi_modal_data": {"image": image}
    },
    SamplingParams(temperature=0.0, max_tokens=500)
)
text = output[0].outputs[0].text

Pattern 3: Audio Transcription with Timestamps

from vllm import LLM, SamplingParams

llm = LLM(model="openai/whisper-large-v3")

# Use prompt to request timestamps
prompt = {
    "prompt": "Transcribe with timestamps:",
    "multi_modal_data": {"audio": "meeting.wav"}
}

output = llm.generate(prompt, SamplingParams(max_tokens=1000))
transcript = output[0].outputs[0].text

Pattern 4: Batch Image Processing

from vllm import LLM, SamplingParams
from PIL import Image
import os

llm = LLM(model="llava-hf/llava-1.5-7b-hf")

# Process directory of images
image_dir = "images/"
results = []

for filename in os.listdir(image_dir):
    if filename.endswith((".jpg", ".png")):
        image = Image.open(os.path.join(image_dir, filename))
        prompt = "USER: <image>\nDescribe this image in one sentence.\nASSISTANT:"

        output = llm.generate(
            {"prompt": prompt, "multi_modal_data": {"image": image}},
            SamplingParams(max_tokens=50)
        )
        results.append({
            "file": filename,
            "description": output[0].outputs[0].text
        })

Troubleshooting

Problem: Image format not supported

Solution:

from PIL import Image

# Convert to supported format
image = Image.open("image.webp")
image = image.convert("RGB")  # Ensure RGB format

# Save as supported format if needed
image.save("image_converted.jpg", "JPEG")

Problem: Out of memory with large images

Solution:

from PIL import Image

# Resize image before processing
image = Image.open("large_image.jpg")
image = image.resize((512, 512))  # Reduce resolution

# Or use quantization
llm = LLM(
    model="llava-hf/llava-1.5-7b-hf",
    quantization="awq"
)

Problem: Video processing too slow

Solution:

import cv2

# Sample fewer frames
cap = cv2.VideoCapture("video.mp4")
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

# Sample 8 evenly distributed frames
frame_indices = [int(idx * total_frames / 8) for idx in range(8)]
frames = []
for idx in frame_indices:
    cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
    ret, frame = cap.read()
    if ret:
        frames.append(frame)
cap.release()

Problem: Audio file format not recognized

Solution:

# Convert audio to supported format
from pydub import AudioSegment

audio = AudioSegment.from_file("audio.m4a")
audio = audio.set_frame_rate(16000).set_channels(1)
audio.export("audio_converted.wav", format="wav")

References

• Vision Models - Vision model details and capabilities
• Audio Models - Audio model reference
• Video Models - Video processing guide
• Image Formats - Supported formats and preprocessing