vllm-hardware

name	vllm-hardware
description	Multi-hardware backend configuration (CUDA, ROCm, NPU, XPU, CPU)
triggers	["When user wants to run vLLM on specific hardware","When user needs to configure AMD ROCm","When user wants Intel XPU/NPU support","When user needs CPU-only inference"]

vllm-hardware

Overview

vLLM supports multiple hardware backends beyond NVIDIA CUDA, including AMD ROCm, Intel XPU/NPU, and CPU. This skill covers configuration for each backend.

Prerequisites

• Hardware-specific drivers installed
• vLLM built or installed for target hardware
• Sufficient system resources

Main Workflow

Step 1: NVIDIA CUDA (Default)

# Standard installation (CUDA 12.1)
pip install vllm

# For CUDA 11.8
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu118

from vllm import LLM

# CUDA is default
llm = LLM(model="meta-llama/Llama-2-7b-chat-hf")

Step 2: AMD ROCm

Installation:

# Install ROCm first (Ubuntu example)
sudo apt install rocm-dev rocm-libs

# Install vLLM for ROCm
pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.1

Configuration:

# Verify ROCm
rocm-smi

# Set environment
export HIP_VISIBLE_DEVICES=0,1

# Run vLLM
vllm serve meta-llama/Llama-2-7b-chat-hf

Step 3: Intel XPU

Installation:

# Install Intel PyTorch extension
pip install intel-extension-for-pytorch

# Install vLLM with XPU support
export VLLM_TARGET_DEVICE=xpu
pip install vllm

Configuration:

from vllm import LLM

# vLLM automatically detects XPU when available
llm = LLM(
    model="meta-llama/Llama-2-7b-chat-hf",
    device="xpu"
)

Step 4: Intel CPU

Installation:

# CPU-only installation
export VLLM_TARGET_DEVICE=cpu
pip install vllm --extra-index-url https://download.pytorch.org/whl/cpu

Optimization:

# Set number of threads
export OMP_NUM_THREADS=8

# Enable OpenVINO (optional)
pip install openvino
export VLLM_OPENVINO_ENABLE=1

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-2-7b-chat-hf",
    device="cpu"
)

Step 5: Ascend NPU

Installation:

# Install CANN toolkit first
# Then install vLLM
export VLLM_TARGET_DEVICE=npu
pip install vllm

Configuration:

# Set device
export ASCEND_RT_VISIBLE_DEVICES=0

# Run
vllm serve meta-llama/Llama-2-7b-chat-hf

Common Patterns

Pattern 1: Multi-GPU Selection

# CUDA: Select specific GPUs
export CUDA_VISIBLE_DEVICES=0,2,3

# ROCm: Select specific GPUs
export HIP_VISIBLE_DEVICES=0,1

# Then run
vllm serve model --tensor-parallel-size 3

Pattern 2: Mixed Precision by Hardware

from vllm import LLM

# NVIDIA A100/H100 - use bfloat16
llm = LLM(model="model", dtype="bfloat16")

# Older GPUs - use float16
llm = LLM(model="model", dtype="float16")

# CPU - use float32
llm = LLM(model="model", dtype="float32")

Pattern 3: Hardware Capability Detection

import torch

def get_optimal_config():
    if torch.cuda.is_available():
        capability = torch.cuda.get_device_capability()
        if capability[0] >= 8:  # Ampere+
            return {"dtype": "bfloat16", "kv_cache_dtype": "fp8"}
        else:
            return {"dtype": "float16"}
    elif torch.xpu.is_available():
        return {"dtype": "bfloat16", "device": "xpu"}
    else:
        return {"dtype": "float32", "device": "cpu"}

config = get_optimal_config()
llm = LLM(model="model", **config)

Troubleshooting

Problem: ROCm device not found

Solution:

# Check ROCm installation
rocm-smi

# Verify PyTorch ROCm support
python -c "import torch; print(torch.cuda.is_available())"

# Reinstall with correct version
pip uninstall torch vllm -y
pip install torch --index-url https://download.pytorch.org/whl/rocm6.1
pip install vllm

Problem: XPU initialization error

Solution:

# Check Intel GPU
sycl-ls

# Set environment
export SYCL_CACHE_PERSISTENT=1
export VLLM_TARGET_DEVICE=xpu

# Verify IPEX
python -c "import intel_extension_for_pytorch as ipex; print(ipex.__version__)"

Problem: CPU inference too slow

Solution:

# Use smaller model or quantization
llm = LLM(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    device="cpu",
    quantization="gptq"  # If available
)

# Or use model optimized for CPU
# Like Phi-2, Gemma 2B

Problem: NPU out of memory

Solution:

llm = LLM(
    model="model",
    device="npu",
    max_model_len=2048,  # Reduce context
    gpu_memory_utilization=0.8
)

References

• CUDA Optimization - NVIDIA-specific optimizations
• ROCm Guide - AMD ROCm detailed setup
• CPU Inference - CPU optimization techniques