vllm-perf

name	vllm-perf
description	Performance optimization, profiling, benchmarking
triggers	["When user needs to optimize throughput","When user wants to reduce latency","When user needs to benchmark vLLM","When user wants to profile performance bottlenecks"]

vllm-perf

Overview

vLLM performance optimization involves tuning batch processing, memory usage, and scheduling parameters. This skill covers benchmarking, profiling, and optimization techniques.

Prerequisites

• vLLM installed and running
• Understanding of target workload characteristics
• Basic knowledge of GPU profiling

Main Workflow

Step 1: Benchmark Serving

# Basic benchmark
python -m vllm.entrypoints.benchmark_serving \
  --host localhost \
  --port 8000 \
  --dataset-name random \
  --num-prompts 100 \
  --max-concurrency 10

# With specific model
python -m vllm.entrypoints.benchmark_serving \
  --host localhost \
  --port 8000 \
  --model meta-llama/Llama-2-7b-chat-hf \
  --num-prompts 1000 \
  --request-rate 10

Step 2: Tune max_num_seqs

from vllm import LLM, SamplingParams

# Increase for higher throughput (more concurrent sequences)
# Decrease for lower latency (less queuing)

llm = LLM(
    model="meta-llama/Llama-2-7b-chat-hf",
    max_num_seqs=256,  # Default is 256, try 128 or 512
)

Step 3: Enable Chunked Prefill

# For long context workloads
vllm serve model --enable-chunked-prefill

from vllm import LLM

llm = LLM(
    model="model",
    enable_chunked_prefill=True,
    max_num_batched_tokens=2048
)

Step 4: Memory Optimization

from vllm import LLM

# Adjust GPU memory utilization
llm = LLM(
    model="meta-llama/Llama-2-7b-chat-hf",
    gpu_memory_utilization=0.95,  # Use more GPU memory
    max_model_len=4096,
    max_num_seqs=512
)

Common Patterns

Pattern 1: Throughput Optimization

# Maximize throughput for batch processing
vllm serve model \
  --max-num-seqs 512 \              # More concurrent sequences
  --gpu-memory-utilization 0.95 \   # Use more GPU memory
  --enable-chunked-prefill \        # Better for long contexts
  --max-num-batched-tokens 4096     # Larger batches

Pattern 2: Latency Optimization

# Minimize latency for real-time use
vllm serve model \
  --max-num-seqs 64 \               # Fewer concurrent, faster scheduling
  --scheduler-delay-factor 0.0 \    # No batching delay
  --gpu-memory-utilization 0.85     # Leave headroom

Pattern 3: Profiling with PyTorch Profiler

import torch
from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b-chat-hf")

# Enable profiler
with torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU,
        torch.profiler.ProfilerActivity.CUDA,
    ],
    schedule=torch.profiler.schedule(wait=1, warmup=1, active=3),
    on_trace_ready=torch.profiler.tensorboard_trace_handler("./log"),
    record_shapes=True,
    profile_memory=True,
) as prof:
    for _ in range(5):
        llm.generate("Hello world", SamplingParams(max_tokens=50))
        prof.step()

# Print summary
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))

Pattern 4: Custom Benchmark Script

import time
import statistics
from vllm import LLM, SamplingParams

def benchmark_throughput(model, prompts, max_tokens=100):
    llm = LLM(model=model)
    sampling_params = SamplingParams(max_tokens=max_tokens)

    # Warm up
    llm.generate(prompts[:5], sampling_params)

    # Benchmark
    start = time.time()
    outputs = llm.generate(prompts, sampling_params)
    end = time.time()

    total_tokens = sum(
        len(o.outputs[0].token_ids) for o in outputs
    )
    elapsed = end - start

    print(f"Total prompts: {len(prompts)}")
    print(f"Total tokens: {total_tokens}")
    print(f"Elapsed: {elapsed:.2f}s")
    print(f"Throughput: {len(prompts)/elapsed:.2f} prompts/s")
    print(f"Token throughput: {total_tokens/elapsed:.2f} tokens/s")

# Usage
prompts = ["Tell me a story"] * 100
benchmark_throughput("meta-llama/Llama-2-7b-chat-hf", prompts)

Pattern 5: Latency Distribution Analysis

import time
import numpy as np
from vllm import LLM, SamplingParams

def measure_latency(model, prompt, num_requests=100):
    llm = LLM(model=model)
    sampling_params = SamplingParams(max_tokens=50)

    # Warm up
    llm.generate(prompt, sampling_params)

    # Measure
    latencies = []
    for _ in range(num_requests):
        start = time.time()
        llm.generate(prompt, sampling_params)
        latencies.append((time.time() - start) * 1000)  # ms

    print(f"P50: {np.percentile(latencies, 50):.2f}ms")
    print(f"P90: {np.percentile(latencies, 90):.2f}ms")
    print(f"P99: {np.percentile(latencies, 99):.2f}ms")
    print(f"Mean: {np.mean(latencies):.2f}ms")
    print(f"Std: {np.std(latencies):.2f}ms")

# Usage
measure_latency("meta-llama/Llama-2-7b-chat-hf", "Hello")

Troubleshooting

Problem: Low GPU utilization

Solution:

# Increase batch size
vllm serve model --max-num-seqs 512

# Or reduce scheduling delay
vllm serve model --scheduler-delay-factor 0.0

Problem: High TTFT (Time To First Token)

Solution:

# Enable chunked prefill
vllm serve model --enable-chunked-prefill

# Or reduce max_num_batched_tokens for faster first response
vllm serve model --max-num-batched-tokens 1024

Problem: Memory fragmentation

Solution:

# Set environment variable
import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

# Or restart with fresh memory pool

References

• Benchmarking Guide - Detailed benchmark methodology
• Memory Optimization - Memory tuning techniques