vllm

name	vllm
description	Deploy and serve LLMs with vLLM behind an OpenAI-compatible endpoint, with tool calling enabled for agent workloads.
short_description	Serve models locally with vLLM.
short_description_zh	用 vLLM 部署本地模型服务。
version	2
updated	2026-08-23T15:42:44.000Z

vLLM Serving

vLLM serves open-weight LLMs on local GPUs with high-throughput inference behind an OpenAI-compatible API, ready for chat and agent workloads.

Before you start

If the user's message only invokes this skill (e.g. "use vllm skill") without a concrete request, ask the user what they want. Do not run any command until the goal is clear.

Ask the user which model to serve; if they have no preference, recommend the small default Qwen/Qwen3.5-0.8B. Also ask what context length the workload needs.

vLLM needs an NVIDIA or AMD GPU. Engine choice follows the user's preference: Ollama also runs on GPUs and is the simpler default — pick vLLM for high-throughput serving, and Ollama on macOS or CPU-only machines, which vLLM does not serve. Confirm the hardware first:

nvidia-smi          # NVIDIA: GPU model and free VRAM (AMD ROCm: rocm-smi)
python3 --version   # a recent Python is required

The model must fit the available VRAM — model size and context length drive the serve flags below.

Suggested workflow

1. Ask the user which model to serve; with no preference, recommend Qwen/Qwen3.5-0.8B.
2. Pick the engine the user prefers: vLLM for high-throughput GPU serving; Ollama is the simple default and the choice on macOS or CPU-only machines.
3. Serve on a free port, with the tool-calling flags whenever agents will call it (see below).
4. Verify with curl http://localhost:8000/v1/models.
5. Register the endpoint: penguin config model add ... --client-type openai-chat --base-url http://localhost:8000/v1 — a served model is not visible to Penguin until added.
6. Confirm the new entry with penguin config model list.

Install

Use a fresh virtual environment (or uv):

python3 -m venv .venv && source .venv/bin/activate
pip install vllm

Serve

vllm serve Qwen/Qwen3.5-0.8B --port 8000

This exposes an OpenAI-compatible API at http://localhost:8000/v1. Key flags:

• --served-model-name <name> — the model id clients request (defaults to the model path).
• --api-key <key> — require this bearer token on every request.
• --max-model-len <n> — context window; agent sessions need a large one.
• --gpu-memory-utilization <0..1> — fraction of VRAM to claim (default 0.9).
• --tensor-parallel-size <n> — shard across n GPUs.
• --dtype <auto|bfloat16|float16> and --quantization <awq|gptq|fp8> — precision and quantized weights.

If the port is taken, pick a free one — never kill a process already listening on it.

Tool calling — required for agents

Agent harnesses (PenguinHarness included) send tools with their requests. vLLM must opt in at startup:

vllm serve Qwen/Qwen3.5-0.8B --enable-auto-tool-choice --tool-call-parser hermes

Choose the parser for the model family — e.g. hermes for Qwen models, llama3_json for Llama models. Without these flags, requests that set tool_choice fail with 400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set.

Verify

curl http://localhost:8000/v1/models

Register with PenguinHarness

Model configuration is the penguin CLI's job — penguin config model add registers an endpoint and penguin config model list shows what has been registered. A served model is not visible to Penguin until you add it:

penguin config model add --provider custom --client-type openai-chat \
  --base-url http://localhost:8000/v1 --model-id <served-model-name> --api-key <key>
penguin config model list   # the new entry should now be listed

Troubleshooting

• Out of memory at startup: lower --gpu-memory-utilization or --max-model-len, or serve a quantized model.
• Long prompts truncated or context-length errors: raise --max-model-len (bounded by VRAM).
• 400 on tool calls: restart the server with the tool-calling flags above.