vllm-distributed

name	vllm-distributed
description	Distributed inference, tensor/pipeline parallelism
triggers	["When user wants to run models larger than single GPU memory","When user needs multi-GPU inference","When user wants to set up Ray cluster","When user needs tensor or pipeline parallelism"]

vllm-distributed

Overview

vLLM supports distributed inference across multiple GPUs through tensor parallelism and pipeline parallelism. This skill covers multi-GPU and multi-node configurations.

Prerequisites

• Multiple GPUs available on single node or cluster
• NCCL (for CUDA) or Gloo (for CPU) configured
• Sufficient interconnect bandwidth (NVLink, InfiniBand)

Main Workflow

Step 1: Tensor Parallelism (Single Node Multi-GPU)

Tensor parallelism splits model layers across GPUs.

from vllm import LLM

# Use 4 GPUs for tensor parallelism
llm = LLM(
    model="meta-llama/Llama-2-70b-chat-hf",
    tensor_parallel_size=4
)

output = llm.generate("Hello world")

# Via command line
vllm serve meta-llama/Llama-2-70b-chat-hf \
  --tensor-parallel-size 4

Step 2: Pipeline Parallelism (Multi-Stage)

Pipeline parallelism splits model into stages.

from vllm import LLM

# Combine tensor and pipeline parallelism
llm = LLM(
    model="meta-llama/Llama-2-70b-chat-hf",
    tensor_parallel_size=2,
    pipeline_parallel_size=2  # Total 4 GPUs (2x2)
)

Step 3: Ray Cluster Setup

Start Ray Head:

# On head node
ray start --head --port=6379

Connect Workers:

# On worker nodes
ray start --address="head-node-ip:6379"

Verify Cluster:

ray status

Step 4: Multi-Node Inference

from vllm import LLM
import ray

# Connect to Ray cluster (optional, vLLM auto-connects)
ray.init(address="auto")

# Tensor parallelism across 8 GPUs on multiple nodes
llm = LLM(
    model="meta-llama/Llama-2-70b-chat-hf",
    tensor_parallel_size=8
)

Common Patterns

Pattern 1: Large Model on 4 GPUs

from vllm import LLM

# Llama-2-70B on 4x A100 (40GB)
llm = LLM(
    model="meta-llama/Llama-2-70b-chat-hf",
    tensor_parallel_size=4,
    dtype="bfloat16",
    gpu_memory_utilization=0.90
)

Pattern 2: 8-GPU Configuration

from vllm import LLM

# Mixtral 8x22B on 8x A100 (80GB)
llm = LLM(
    model="mistralai/Mixtral-8x22B-Instruct-v0.1",
    tensor_parallel_size=8,
    max_model_len=8192,
    dtype="bfloat16"
)

Pattern 3: Select Specific GPUs

# Use specific GPUs
export CUDA_VISIBLE_DEVICES=0,1,2,3

vllm serve model --tensor-parallel-size 4

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3"

from vllm import LLM
llm = LLM(model="model", tensor_parallel_size=4)

Pattern 4: Ray Cluster with Docker

# Head node
docker run --gpus all --network host \
  -e RAY_ROLE=head \
  vllm/vllm-openai:latest \
  ray start --head --port=6379

# Worker nodes
docker run --gpus all --network host \
  -e RAY_ROLE=worker \
  vllm/vllm-openai:latest \
  ray start --address="head-ip:6379"

Pattern 5: Pipeline Parallel for Very Large Models

from vllm import LLM

# 405B model on 8x H100
# Use pipeline parallelism for very large models
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-405B-Instruct",
    tensor_parallel_size=4,
    pipeline_parallel_size=2,  # 8 GPUs total
    dtype="bfloat16",
    max_model_len=4096
)

Troubleshooting

Problem: NCCL communication error

Solution:

# Check NCCL is working
python -c "import torch; print(torch.cuda.is_available())"

# Set NCCL debugging
export NCCL_DEBUG=INFO
export NCCL_SOCKET_IFNAME=eth0  # Specify network interface

# For InfiniBand
export NCCL_IB_DISABLE=0
export NCCL_NET_GDR_LEVEL=5

Problem: Ray cluster connection failed

Solution:

# Check Ray status
ray status

# Restart Ray
ray stop
ray start --head --port=6379

# Verify Python versions match across nodes
python --version

Problem: OOM despite multiple GPUs

Solution:

# May need more GPUs or quantization
llm = LLM(
    model="model",
    tensor_parallel_size=8,  # Increase if available
    quantization="awq",      # Or use quantization
    max_model_len=4096       # Reduce context
)

Problem: Slow distributed inference

Solution:

# Check interconnect
nvidia-smi topo -m

# Use NVLink if available
# Otherwise ensure high-speed network (InfiniBand, 100Gbps+)

# Disable if slow interconnect
export NCCL_P2P_DISABLE=1

References

• Tensor & Pipeline Parallelism - Detailed parallelism guide
• Ray Deployment - Ray cluster setup reference