vllm-api

name	vllm-api
description	Use OpenAI-compatible API, integrate with clients
triggers	["When user wants to call vLLM via API","When user needs OpenAI-compatible endpoints","When user wants to integrate vLLM with existing applications","When user encounters API authentication or error handling issues"]

vllm-api

Overview

vLLM provides an OpenAI-compatible API server, enabling seamless integration with existing OpenAI clients and tools. This skill covers endpoint usage, client integration, and error handling.

Prerequisites

• vLLM installed (pip install vllm)
• A running vLLM server or the ability to start one
• API client (Python openai library, curl, or HTTP client)

Main Workflow

Step 1: Start the API Server

# Basic server
vllm serve meta-llama/Llama-2-7b-chat-hf

# With custom settings
vllm serve meta-llama/Llama-2-7b-chat-hf \
  --host 0.0.0.0 \
  --port 8000 \
  --api-key your-api-key

Step 2: Chat Completions (Primary Endpoint)

Python Client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"  # Required but ignored if not configured
)

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is machine learning?"}
    ],
    temperature=0.7,
    max_tokens=150
)

print(response.choices[0].message.content)

cURL:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-7b-chat-hf",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ],
    "max_tokens": 50
  }'

Step 3: Text Completions (Legacy)

response = client.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    prompt="Once upon a time",
    max_tokens=100,
    temperature=0.8
)

print(response.choices[0].text)

Step 4: Streaming Responses

stream = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True,
    max_tokens=500
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Step 5: Authentication

With API Key:

# Start server with API key
vllm serve model --api-key sk-secret-key

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-secret-key"
)

Without Authentication (Development Only):

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"  # Required by client but ignored by vLLM
)

Common Patterns

Pattern 1: Batch Requests

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

async def process_batch(prompts):
    tasks = [
        client.chat.completions.create(
            model="meta-llama/Llama-2-7b-chat-hf",
            messages=[{"role": "user", "content": p}],
            max_tokens=100
        )
        for p in prompts
    ]
    responses = await asyncio.gather(*tasks)
    return [r.choices[0].message.content for r in responses]

# Usage
prompts = ["What is AI?", "Explain Python", "Define cloud computing"]
results = asyncio.run(process_batch(prompts))

Pattern 2: Error Handling

from openai import OpenAI, APIError, RateLimitError

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

try:
    response = client.chat.completions.create(
        model="meta-llama/Llama-2-7b-chat-hf",
        messages=[{"role": "user", "content": "Hello"}]
    )
except RateLimitError:
    print("Rate limit exceeded. Retry after delay.")
except APIError as e:
    print(f"API error: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")

Pattern 3: Multi-turn Conversation

messages = [
    {"role": "system", "content": "You are a coding assistant."}
]

while True:
    user_input = input("User: ")
    if user_input.lower() == "exit":
        break

    messages.append({"role": "user", "content": user_input})

    response = client.chat.completions.create(
        model="meta-llama/Llama-2-7b-chat-hf",
        messages=messages,
        max_tokens=200
    )

    assistant_reply = response.choices[0].message.content
    print(f"Assistant: {assistant_reply}")

    messages.append({"role": "assistant", "content": assistant_reply})

Troubleshooting

Problem: Connection refused

Solution:

# Check if server is running
curl http://localhost:8000/health

# Verify correct port
vllm serve model --port 8000
# Client must use same port: base_url="http://localhost:8000/v1"

Problem: Model not found

Solution:

# Verify model name matches exactly
curl http://localhost:8000/v1/models

# Use the exact model name shown

Problem: 422 Unprocessable Entity

Solution:

Check request format. Common issues:

• Missing required fields (messages, model)
• Invalid parameter values
• Malformed JSON

# Correct format
{
    "model": "model-name",
    "messages": [{"role": "user", "content": "Hello"}]
}

Problem: Slow response time

Solution:

# Increase max_num_seqs for better throughput
vllm serve model --max_num_seqs 256

# Enable chunked prefill (v0.5.0+)
vllm serve model --enable-chunked-prefill

References

• Complete Endpoints - Full API endpoint documentation
• Error Codes - Error code reference and solutions