Use vLLM for high-throughput production serving of self-hosted models with DSPy via dspy.LM with openai/ prefix and api_base. Use when you want production LLM serving, tensor parallelism, multi-GPU inference, batch processing, or high-concurrency self-hosted models. Also used for vllm, vLLM, production serving, high throughput LLM, tensor parallelism, self-hosted production, PagedAttention, local production server, GPU serving, batch inference, vllm serve, pip install vllm, multi-GPU LLM, speculative decoding, continuous batching, deploy local model, NVIDIA GPU serving, openai compatible server, AWQ quantization vllm, GPTQ vllm, dspy.LM api_base vllm, openai/ provider prefix vllm, connect DSPy to vLLM.
Use vLLM for high-throughput production serving of self-hosted models with DSPy via dspy.LM with openai/ prefix and api_base. Use when you want production LLM serving, tensor parallelism, multi-GPU inference, batch processing, or high-concurrency self-hosted models. Also used for vllm, vLLM, production serving, high throughput LLM, tensor parallelism, self-hosted production, PagedAttention, local production server, GPU serving, batch inference, vllm serve, pip install vllm, multi-GPU LLM, speculative decoding, continuous batching, deploy local model, NVIDIA GPU serving, openai compatible server, AWQ quantization vllm, GPTQ vllm, dspy.LM api_base vllm, openai/ provider prefix vllm, connect DSPy to vLLM.
vLLM — High-Throughput Production Serving for DSPy
Guide the user through serving self-hosted models with vLLM for production DSPy deployments. High concurrency, multi-GPU, OpenAI-compatible API.
Step 1: Understand the setup
Before generating vLLM configuration, clarify:
What GPU hardware? — Model (A100, H100, RTX 4090), count, and VRAM per GPU. This determines tensor parallelism and quantization needs.
Which model? — Model name and size (7B, 13B, 70B). Determines VRAM requirements and whether quantization is needed.
Workload type? — Production serving (concurrent users), batch processing (offline), or optimization (running MIPROv2/BootstrapFewShot)?
Already using Ollama locally? — If yes, help them add vLLM for production while keeping Ollama for dev.
What is vLLM
vLLM is a high-throughput inference engine (74k+ GitHub stars) for LLMs. Key features:
PagedAttention — 4x memory efficiency vs naive attention, serves more concurrent users
Continuous batching — processes requests as they arrive, no waiting for batch to fill
Tensor parallelism — split models across multiple GPUs
OpenAI-compatible API — drop-in replacement, DSPy connects via openai/ provider
Speculative decoding — use a small draft model to speed up large model generation
When to use vLLM
Scenario
Use vLLM?
Alternative
Production API (10+ concurrent users)
Yes
—
Multi-GPU serving
Yes
—
Batch processing (1000s of inputs)
Yes
—
Local development on macOS
No
Ollama (/dspy-ollama)
Apple Silicon (M1/M2/M3)
No
Ollama (/dspy-ollama)
Quick prototyping
No
Ollama (/dspy-ollama)
Cloud API (no self-hosting)
No
OpenAI/Anthropic (/dspy-lm)
vLLM requires NVIDIA GPUs (CUDA). It does not support Apple Silicon or AMD GPUs (ROCm support is experimental).
Setup
Install
pip install vllm
Requires: Python 3.9+, NVIDIA GPU with CUDA 12.1+, Linux (recommended) or WSL2.
Start a vLLM server
# Basic — serve a model with OpenAI-compatible API
vllm serve meta-llama/Llama-3.1-8B-Instruct
# With common options
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--max-model-len 8192 \
--gpu-memory-utilization 0.9 \
--dtype auto
The server exposes /v1/chat/completions and /v1/completions endpoints.
Connect DSPy to vLLM
import dspy
lm = dspy.LM(
"openai/meta-llama/Llama-3.1-8B-Instruct",
api_base="http://localhost:8000/v1",
api_key="none", # required but any value works
temperature=0.7,
max_tokens=1000,
)
dspy.configure(lm=lm)
# Now all DSPy modules use your vLLM-served model
classify = dspy.ChainOfThought("text -> category, reasoning")
result = classify(text="Server is down, customers can't log in!")
print(result.category)
Note:dspy.HFClientVLLM is deprecated. Use dspy.LM("openai/...") with api_base instead.
Claude uses the deprecated dspy.HFClientVLLM class. This was removed in DSPy 2.5+. Always use dspy.LM("openai/model-name", api_base="http://localhost:8000/v1", api_key="none") instead.
Claude omits api_key when connecting to vLLM. LiteLLM (which DSPy uses under the hood) requires the api_key parameter even though vLLM does not authenticate. Set api_key="none" — any non-empty string works.
Claude recommends vLLM for macOS or Apple Silicon users. vLLM requires NVIDIA GPUs with CUDA. If the user mentions macOS, M1/M2/M3/M4, or no NVIDIA GPU, route to /dspy-ollama instead.
Claude forgets --enable-prefix-caching for DSPy workloads. DSPy optimized programs prepend the same few-shot demos to every request. Without prefix caching, vLLM recomputes the KV cache for those shared tokens on every call. Always recommend it for DSPy serving.
Claude sets --max-model-len too high for available VRAM. This causes OOM on startup. Calculate available VRAM minus ~20% for KV cache overhead. For a 70B FP16 model on 2x A100-80GB, cap at ~8192 tokens. Suggest --gpu-memory-utilization 0.9 as the default and tell users to lower --max-model-len if they hit OOM.
Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do
