vllm

name	vllm
description	Serves LLMs with vLLM PagedAttention and continuous batching: OpenAI-compatible /v1/chat/completions, GPTQ/AWQ/FP8 quantization, tensor parallelism, and offline LLM.generate. Use when deploying high-throughput GPU inference endpoints or fitting 30B-70B models. Not for TRL training, llama.cpp CPU/edge, or using vLLM only as an inspect-ai/lighteval backend.
version	1.0.1
author	Orchestra Research
license	MIT
dependencies	["vllm","torch","transformers"]
platforms	["linux","macos"]
metadata	{"hermes":{"tags":["vLLM","Inference Serving","PagedAttention","Continuous Batching","High Throughput","Production","OpenAI API","Quantization","Tensor Parallelism"]}}

vLLM — High-Performance LLM Serving

Overview

vLLM achieves up to 24x higher throughput than standard HuggingFace transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests). It provides an OpenAI-compatible API server, offline batch inference, quantization support (GPTQ/AWQ/FP8), and tensor parallelism for multi-GPU deployments.

When to Use

Use vLLM when:

• Deploying production LLM APIs targeting 100+ req/sec
• Serving OpenAI-compatible endpoints (/v1/chat/completions, /v1/completions)
• Fitting large models (30B–70B) into limited GPU memory via quantization
• Building multi-user applications (chatbots, assistants, RAG backends)
• Running offline batch inference over large datasets efficiently
• You need low latency with high throughput on NVIDIA GPUs

Use alternatives instead when:

• llama.cpp: CPU/edge inference, single-user scenarios
• HuggingFace transformers: Research, prototyping, one-off generation
• TensorRT-LLM: NVIDIA-only, need absolute maximum performance
• Text-Generation-Inference: Already invested in HuggingFace ecosystem

Prerequisites

Hardware Requirements

Model Size	Minimum GPU	Notes
7B–13B	1x A10 (24GB) or A100 (40GB)	Single GPU sufficient
30B–40B	2x A100 (40GB)	Use --tensor-parallel-size 2
70B+	4x A100 (40GB) or 2x A100 (80GB)	Use AWQ/GPTQ quantization

Supported platforms: NVIDIA (primary), AMD ROCm, Intel GPUs, TPUs.

Software Installation

pip install vllm

Dependencies: vllm, torch, transformers.

Procedure

Workflow 1: Basic Offline Inference

1. Import and initialize the LLM engine:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=, max_tokens=)