serving-llms-vllm

vLLM: high-throughput LLM serving, OpenAI API, quantization.

Dépôt: NousResearch/hermes-agent
Dernière activité de la source: 24 juillet 2026 à 15:29
Langue détectée de SKILL.md: anglais
Étoiles: 227 004
Forks: 44 361

Options d'installation

Le prompt qui vérifie d'abord la source est sélectionné par défaut. Vous pouvez passer à une commande directe ou télécharger une copie locale.

Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.

Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.

npx skills add https://github.com/NousResearch/hermes-agent --skill serving-llms-vllm

La commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.

Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.

Vérifiez les fichiers source

Lisez SKILL.md et les fichiers associés affichés par SkillsMP avant de décider de l'installer.

Explorateur de fichiers

5 fichiers

SKILL.md

Instructions source · Aperçu en lecture seule

Retour à la recherche Retour à l’installation

document-to-action-items

Extract cited obligations, deadlines, tasks from documents.

2026-08-07

hermes-agent-skill-authoring

Author in-repo SKILL.md files: frontmatter and structure.

2026-08-07

polymarket

Query Polymarket: markets, prices, orderbooks, history.

2026-08-06

Tout voir dans ce dépôt

pip install vllm

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=0.7, max_tokens=256)

outputs = llm.generate(["Explain quantum computing"], sampling)
print(outputs[0].outputs[0].text)

vllm serve meta-llama/Meta-Llama-3-8B-Instruct

# Query with OpenAI SDK
python -c "
from openai import OpenAI
client = OpenAI(base_url='http://localhost:8000/v1', api_key='EMPTY')
print(client.chat.completions.create(
    model='meta-llama/Meta-Llama-3-8B-Instruct',
    messages=[{'role': 'user', 'content': 'Hello!'}]
).choices[0].message.content)
"

Deployment Progress:
- [ ] Step 1: Configure server settings
- [ ] Step 2: Test with limited traffic
- [ ] Step 3: Enable monitoring
- [ ] Step 4: Deploy to production
- [ ] Step 5: Verify performance metrics

# For 7B-13B models on single GPU
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --port 8000

# For 30B-70B models with tensor parallelism
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --quantization awq \
  --port 8000

# For production with caching (Prometheus metrics are exposed
# automatically at /metrics on the API port)
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --port 8000 \
  --host 0.0.0.0

# Install load testing tool
pip install locust

# Create test_load.py with sample requests
# Run: locust -f test_load.py --host http://localhost:8000

curl http://localhost:8000/metrics | grep vllm

# Run vLLM in Docker
docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching

Batch Processing:
- [ ] Step 1: Prepare input data
- [ ] Step 2: Configure LLM engine
- [ ] Step 3: Run batch inference
- [ ] Step 4: Process results

# Load prompts from file
prompts = []
with open("prompts.txt") as f:
    prompts = [line.strip() for line in f]

print(f"Loaded {len(prompts)} prompts")

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=2,  # Use 2 GPUs
    gpu_memory_utilization=0.9,
    max_model_len=4096
)

sampling = SamplingParams(
    temperature=0.7,
    top_p=0.95,
    max_tokens=512,
    stop=["</s>", "\n\n"]
)

# Process all prompts in one call
outputs = llm.generate(prompts, sampling)

# vLLM handles batching internally
# No need to manually chunk prompts

# Extract generated text
results = []
for output in outputs:
    prompt = output.prompt
    generated = output.outputs[0].text
    results.append({
        "prompt": prompt,
        "generated": generated,
        "tokens": len(output.outputs[0].token_ids)
    })

# Save to file
import json
with open("results.jsonl", "w") as f:
    for result in results:
        f.write(json.dumps(result) + "\n")

print(f"Processed {len(results)} prompts")

Quantization Setup:
- [ ] Step 1: Choose quantization method
- [ ] Step 2: Find or create quantized model
- [ ] Step 3: Launch with quantization flag
- [ ] Step 4: Verify accuracy

# Search for AWQ models
# Example: TheBloke/Llama-2-70B-AWQ

# Using pre-quantized model
vllm serve TheBloke/Llama-2-70B-AWQ \
  --quantization awq \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.95

# Results: 70B model in ~40GB VRAM

# Compare quantized vs non-quantized responses
# Verify task-specific performance unchanged

vllm serve MODEL \
  --gpu-memory-utilization 0.7 \
  --max-model-len 4096

vllm serve MODEL --quantization awq

vllm serve MODEL --enable-prefix-caching

vllm serve MODEL --enable-chunked-prefill

vllm serve MODEL --trust-remote-code

vllm serve MODEL --max-num-seqs 512

vllm serve MODEL --tensor-parallel-size 4  # Not 3

vllm serve MODEL \
  --speculative-config '{"model": "DRAFT_MODEL", "num_speculative_tokens": 5, "method": "draft_model"}'

name	serving-llms-vllm
description	vLLM: high-throughput LLM serving, OpenAI API, quantization.
version	1.0.1
author	Orchestra Research
license	MIT
dependencies	["vllm","torch","transformers"]
platforms	["linux","macos"]
metadata	{"hermes":{"tags":["vLLM","Inference Serving","PagedAttention","Continuous Batching","High Throughput","Production","OpenAI API","Quantization","Tensor Parallelism"]}}

name	serving-llms-vllm
description	vLLM: high-throughput LLM serving, OpenAI API, quantization.
version	1.0.1
author	Orchestra Research
license	MIT
dependencies	["vllm","torch","transformers"]
platforms	["linux","macos"]
metadata	{"hermes":{"tags":["vLLM","Inference Serving","PagedAttention","Continuous Batching","High Throughput","Production","OpenAI API","Quantization","Tensor Parallelism"]}}

serving-llms-vllm

Options d'installation

Vérifiez les fichiers source

vLLM - High-Performance LLM Serving

When to use

vLLM - High-Performance LLM Serving

When to use

Quick start

Common workflows

Workflow 1: Production API deployment

Workflow 2: Offline batch inference

Workflow 3: Quantized model serving

When to use vs alternatives

Common issues

Advanced topics

Hardware requirements

Resources

serving-llms-vllm

Informations de source

Options d'installation

Vérifiez les fichiers source

Plus depuis ce dépôt

Plus depuis ce dépôt

vLLM - High-Performance LLM Serving

When to use

vLLM - High-Performance LLM Serving

When to use

Quick start

Common workflows

Workflow 1: Production API deployment

Workflow 2: Offline batch inference

Workflow 3: Quantized model serving

When to use vs alternatives

Common issues

Advanced topics

Hardware requirements

Resources