benchmark-vllm

name	benchmark-vllm
description	Measures vLLM serving performance on the zentorch/ZenDNN CPU backend on AMD EPYC. Brings up a multi-instance stack -- N cpuset-pinned `vllm serve` containers, one core slice each, behind an NGINX round-robin load balancer -- and drives it with GuideLLM through the redhat-et/vllm-cpu-perf-eval ansible automation in external-endpoint mode. Sizes the instance layout from the machine's real topology (SMT siblings kept inside one instance), derives the server context window from the selected workloads, and sweeps a list of request rates against one warm stack, so throughput numbers (high concurrency, 32/64+) and latency numbers (low concurrency, 1/2/4/8) come from an identical stack and stay comparable. Can also run a native (non-zentorch) A/B leg from the same image with zentorch uninstalled, so a zentorch-vs-native delta is not confounded by a different vLLM commit, torch build or compiler flags. Runs the perf-eval workload set (chat, rag, code, summarization, reasoning, variable-length and context-scaling profiles) against any Hugging Face model the image can serve, and produces per-run GuideLLM results, a resident-memory trace and a per-instance provenance banner (pip list, exported env, exact serve command). Use when asked to benchmark vLLM throughput, latency or serving performance on CPU with zentorch, or to size a multi-instance CPU serving run.
version	1.0.0

Benchmark vLLM (zentorch / ZenDNN CPU backend)

An interactive orchestrator around one stack shape:

GuideLLM (container, own cores) -> NGINX round-robin (own cores)
                                     -> N x `vllm serve` containers, one cpuset each

There is no second mode. Peak-throughput ("offline") numbers are the same stack measured at high concurrency (--rates 32,64 and up); latency numbers are the same stack at low concurrency (). Nothing else changes, so the two profiles are directly comparable and a single warm stack can serve both.