| name | vllm-benchmark |
| description | Run prefill + decode throughput benchmarks against a vLLM server and generate a markdown report with plots. |
| disable-model-invocation | true |
vLLM Benchmark
Runs both prefill and decode throughput sweeps in a single invocation. Time-based measurement: sends streaming requests at max concurrency, measures throughput over a fixed window, then cancels and moves on.
Outputs per model:
benchmark_{model}.md — markdown report with server config, prefill table, decode tablesbenchmark_{model}.json — raw dataprefill_{model}.png — prefill throughput plotdecode_{model}.png — decode throughput plot
Step 1: Launch the vLLM server
Start the server as a background task before running benchmarks.
Single-node (TP <= 8):
uv run python -m vllm.entrypoints.openai.api_server \
--model <model> \
--dtype bfloat16 \
--tensor-parallel-size <tp> \
--gpu-memory-utilization 0.95 \
--enable-chunked-prefill \
--no-enable-prefix-caching \
--max-num-seqs <max_concurrency> \
--max-model-len 65536 \
--port 8000 \
--host 0.0.0.0
Multi-node (requires Ray cluster):
VLLM_HOST_IP=<ray_head_ip> uv run python -m vllm.entrypoints.openai.api_server \
--model <model> \
--dtype bfloat16 \
--tensor-parallel-size <tp> \
--pipeline-parallel-size <pp> \
--gpu-memory-utilization 0.95 \
--enable-chunked-prefill \
--no-enable-prefix-caching \
--max-num-seqs <max_concurrency> \
--max-model-len 65536 \
--port 8000 \
--host 0.0.0.0
Key notes:
- Multi-node only:
VLLM_HOST_IP must be set to the Ray head node's internal/cluster IP. Find it with ray status or hostname -I.
--no-enable-prefix-caching — must be disabled for benchmarking since all requests use the same prompt, which would inflate prefill numbers ~40x if cached.--enable-chunked-prefill — allows interleaving prefill and decode for better throughput.--max-num-seqs — must be >= the highest concurrency level being benchmarked, otherwise the server caps the batch size and queues the rest, bottlenecking the results.- Wait for the server to be healthy before benchmarking:
curl http://localhost:8000/health
- Read the
KV cache size: X tokens line from server logs — you need this for --kv-cache-tokens.
Step 2: Run the benchmark
Collect these parameters from the user (defaults shown):
--base-url (default: http://localhost:8000)--model (required — the model name served by vLLM)--kv-cache-tokens (required — total KV cache capacity in tokens, from server logs)--tp (default: 1 — tensor parallel size, for report metadata)--kv-cache-dtype (default: auto — for report metadata)--ratios (default: 0.25,0.5 — input token fractions for decode sweep)--concurrency-levels (default: 32,64,128,256,512,1024,2048)--max-model-len (default: 65536)--warmup (default: 10.0 — seconds to warm up before measuring)--duration (default: 15.0 — seconds to measure throughput)--auto-warmup (flag — compute decode warmup from prefill results so measurement starts after all slots finish prefilling)--warmup-margin (default: 1.5 — multiplier on auto-warmup estimate)--output-dir (default: ./bench-results)
uv run python .claude/skills/vllm-benchmark-skill/scripts/bench_sweep.py \
--base-url <url> \
--model <model> \
--kv-cache-tokens <tokens> \
--tp <tp_size> \
--auto-warmup \
--output-dir <dir>
Important: only run ONE benchmark at a time against a server. Running multiple benchmarks concurrently produces garbage results since they compete for GPU resources.
After the script completes, show the user the generated markdown report at <output-dir>/benchmark_{model}.md.
