vllm-serving

name	vllm-serving
description	Deploy API service, production environment configuration
triggers	["When user wants to deploy vLLM as a service","When user needs production deployment configuration","When user wants to configure model serving parameters","When user needs monitoring and logging setup"]

vllm-serving

Overview

Deploying vLLM in production requires careful configuration of model parameters, resource allocation, and monitoring. This skill covers production deployment patterns.

Prerequisites

• vLLM installed
• GPU server with adequate resources
• Basic understanding of the target model's requirements

Main Workflow

Step 1: Basic Deployment

# Simple deployment
vllm serve meta-llama/Llama-2-7b-chat-hf

# With custom configuration
vllm serve meta-llama/Llama-2-7b-chat-hf \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9

Step 2: Production Configuration

# Production-ready deployment
vllm serve meta-llama/Llama-2-7b-chat-hf \
  --host 0.0.0.0 \
  --port 8000 \
  --api-key ${API_KEY} \
  --max-model-len 4096 \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.85 \
  --dtype bfloat16 \
  --tensor-parallel-size 2 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4096

Step 3: Model Configuration Parameters

# Common model configurations
vllm serve meta-llama/Llama-2-7b-chat-hf \
  --max-model-len 4096 \              # Maximum context length
  --max-num-seqs 256 \                 # Maximum concurrent sequences
  --gpu-memory-utilization 0.9 \       # GPU memory fraction
  --dtype bfloat16 \                   # Data type (float16, bfloat16, float32)
  --quantization awq \                 # Quantization method
  --tokenizer-mode auto                # Tokenizer mode

Step 4: Docker Deployment

# Pull and run official image
docker run --runtime nvidia --gpus all \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN=${HF_TOKEN} \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-2-7b-chat-hf \
  --tensor-parallel-size 2

# With custom Dockerfile
```dockerfile
FROM vllm/vllm-openai:latest

COPY config.yaml /config.yaml
ENV VLLM_CONFIG=/config.yaml

ENTRYPOINT ["python", "-m", "vllm.entrypoints.openai.api_server"]

## Common Patterns

### Pattern 1: Multi-GPU Deployment

```bash
# Tensor parallelism across 4 GPUs
vllm serve meta-llama/Llama-2-70b-chat-hf \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --max-model-len 4096

Pattern 2: Quantized Model Serving

# Serve AWQ quantized model (4-bit)
vllm serve TheBloke/Llama-2-7B-AWQ \
  --quantization awq \
  --max-model-len 4096

# Serve GPTQ quantized model
vllm serve TheBloke/Llama-2-7B-GPTQ \
  --quantization gptq

Pattern 3: Systemd Service

# /etc/systemd/system/vllm.service
[Unit]
Description=vLLM API Server
After=network.target

[Service]
Type=simple
User=vllm
Environment="HF_TOKEN=your_token"
Environment="CUDA_VISIBLE_DEVICES=0,1"
ExecStart=/usr/local/bin/vllm serve meta-llama/Llama-2-7b-chat-hf \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

# Enable and start service
sudo systemctl enable vllm
sudo systemctl start vllm
sudo systemctl status vllm

Pattern 4: Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - "--model"
          - "meta-llama/Llama-2-7b-chat-hf"
          - "--tensor-parallel-size"
          - "2"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 2
        env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token

Troubleshooting

Problem: Port already in use

Solution:

# Find and kill process using port 8000
lsof -ti:8000 | xargs kill -9

# Or use different port
vllm serve model --port 8001

Problem: Model loading timeout

Solution:

# Increase download timeout
export HF_HUB_DOWNLOAD_TIMEOUT=300

# Use local model cache
vllm serve /path/to/local/model

Problem: High latency under load

Solution:

# Increase max_num_seqs for better throughput
vllm serve model --max-num-seqs 512

# Enable chunked prefill
vllm serve model --enable-chunked-prefill

# Adjust scheduling delay
vllm serve model --scheduler-delay-factor 0.0

References

• Model Configurations - Common model configuration examples
• Production Checklist - Deployment verification checklist