vllm-llm

name	vllm-llm
description	Text model inference (Llama, Qwen, Mistral, GPT, etc.)
triggers	["When user wants to run text LLM inference","When user needs to configure inference parameters (temperature, top_p, etc.)","When user wants to use chat vs text completion","When user needs to handle long text contexts"]

vllm-llm

Overview

vLLM supports 130+ text-based language models. This skill covers model selection, inference parameter tuning, and common text generation tasks.

Prerequisites

• vLLM installed
• Sufficient GPU memory for target model
• HuggingFace token (for gated models)

Main Workflow

Step 1: Select and Load Model

from vllm import LLM, SamplingParams

# Popular open models
model_id = "meta-llama/Llama-2-7b-chat-hf"  # Llama 2
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"  # Llama 3
model_id = "Qwen/Qwen2-7B-Instruct"  # Qwen2
model_id = "mistralai/Mistral-7B-Instruct-v0.2"  # Mistral

llm = LLM(model=model_id)

Step 2: Configure Sampling Parameters

# Balanced generation
sampling_params = SamplingParams(
    temperature=0.7,      # Randomness (0.0 = deterministic, 2.0 = very random)
    top_p=0.95,           # Nucleus sampling
    top_k=40,             # Top-k sampling
    max_tokens=256,       # Maximum output length
    repetition_penalty=1.1  # Penalize repeated tokens
)

# Deterministic (coding, math)
code_params = SamplingParams(
    temperature=0.0,      # Greedy decoding
    max_tokens=512
)

# Creative (storytelling)
creative_params = SamplingParams(
    temperature=1.0,
    top_p=0.9,
    max_tokens=1024
)

Step 3: Chat Completion Format

# Apply chat template
messages = [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "Write a Python function to calculate factorial."}
]

# Convert to model's chat format
prompt = llm.get_tokenizer().apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

output = llm.generate(prompt, sampling_params)
print(output[0].outputs[0].text)

Step 4: Text Completion Format

# Direct text completion (base models)
prompt = """Translate the following to French:
Hello, how are you?
Translation:"""

output = llm.generate(prompt, sampling_params)
print(output[0].outputs[0].text)

Common Patterns

Pattern 1: Classification Task

# Zero-shot classification
prompt = """Classify the sentiment of this text as positive, negative, or neutral:
Text: "The movie was absolutely fantastic!"
Sentiment:"""

classification_params = SamplingParams(
    temperature=0.0,  # Deterministic
    max_tokens=10,
    stop=["\n"]       # Stop at newline
)

output = llm.generate(prompt, classification_params)
sentiment = output[0].outputs[0].text.strip()

Pattern 2: Structured Output (JSON)

import json

prompt = """Extract information from this text and return as JSON:
Text: "John Doe is 30 years old and works as an engineer in New York."
JSON:{"name": "John Doe", "age": 30, "job": "engineer", "location": "New York"}"""

json_params = SamplingParams(
    temperature=0.0,
    max_tokens=200,
    stop=["}\n"]
)

output = llm.generate(prompt, json_params)
try:
    result = json.loads(output[0].outputs[0].text)
except json.JSONDecodeError:
    result = None

Pattern 3: Summarization

long_text = """[Your long article here...]"""

prompt = f"Summarize the following text in 3 sentences:\n\n{long_text}\n\nSummary:"

summary_params = SamplingParams(
    temperature=0.3,
    max_tokens=200
)

output = llm.generate(prompt, summary_params)
summary = output[0].outputs[0].text

Pattern 4: Long Context Handling

# Models with large context windows
long_context_models = [
    "Qwen/Qwen2-7B-Instruct",      # 32K context
    "mistralai/Mistral-7B-Instruct-v0.2",  # 32K context
    "gradientai/Llama-3-8B-Instruct-Gradient-1048k"  # 1M context
]

# For very long contexts, use RoPE scaling
llm = LLM(
    model="model-name",
    max_model_len=32768,  # Set desired context length
    rope_scaling={
        "type": "dynamic",
        "factor": 4.0
    }
)

Pattern 5: Multi-turn Conversation

conversation = []

while True:
    user_input = input("User: ")
    if user_input.lower() == "exit":
        break

    conversation.append({"role": "user", "content": user_input})

    prompt = llm.get_tokenizer().apply_chat_template(
        conversation,
        tokenize=False,
        add_generation_prompt=True
    )

    output = llm.generate(prompt, SamplingParams(max_tokens=200))
    response = output[0].outputs[0].text

    print(f"Assistant: {response}")
    conversation.append({"role": "assistant", "content": response})

Troubleshooting

Problem: Outputs truncated mid-sentence

Solution:

# Increase max_tokens
sampling_params = SamplingParams(max_tokens=1024)  # Increase limit

# Or use length-based stopping
sampling_params = SamplingParams(
    max_tokens=512,
    stop=["\n\n", "###"]  # Stop at paragraph breaks
)

Problem: Repetitive outputs

Solution:

sampling_params = SamplingParams(
    temperature=0.8,          # Increase randomness
    repetition_penalty=1.2,   # Increase penalty
    top_p=0.95,               # Nucleus sampling
    frequency_penalty=0.5     # Penalize frequent tokens
)

Problem: Context too long error

Solution:

# Check model's max context
max_len = llm.llm_engine.model_config.max_model_len
print(f"Max context: {max_len}")

# Truncate input if needed
tokenizer = llm.get_tokenizer()
tokens = tokenizer.encode(long_text, max_length=max_len-100, truncation=True)
truncated = tokenizer.decode(tokens)

References

• Supported Models - Complete list of supported text models
• Inference Parameters - Detailed sampling parameters guide