| name | benchmark-vllm |
| description | Measures vLLM serving performance on the zentorch/ZenDNN CPU backend on AMD EPYC. Brings up a multi-instance stack -- N cpuset-pinned `vllm serve` containers, one core slice each, behind an NGINX round-robin load balancer -- and drives it with GuideLLM through the redhat-et/vllm-cpu-perf-eval ansible automation in external-endpoint mode. Sizes the instance layout from the machine's real topology (SMT siblings kept inside one instance), derives the server context window from the selected workloads, and sweeps a list of request rates against one warm stack, so throughput numbers (high concurrency, 32/64+) and latency numbers (low concurrency, 1/2/4/8) come from an identical stack and stay comparable. Can also run a native (non-zentorch) A/B leg from the same image with zentorch uninstalled, so a zentorch-vs-native delta is not confounded by a different vLLM commit, torch build or compiler flags. Runs the perf-eval workload set (chat, rag, code, summarization, reasoning, variable-length and context-scaling profiles) against any Hugging Face model the image can serve, and produces per-run GuideLLM results, a resident-memory trace and a per-instance provenance banner (pip list, exported env, exact serve command). Use when asked to benchmark vLLM throughput, latency or serving performance on CPU with zentorch, or to size a multi-instance CPU serving run. |
| version | 1.0.0 |
Benchmark vLLM (zentorch / ZenDNN CPU backend)
An interactive orchestrator around one stack shape:
GuideLLM (container, own cores) -> NGINX round-robin (own cores)
-> N x `vllm serve` containers, one cpuset each
There is no second mode. Peak-throughput ("offline") numbers are the same stack
measured at high concurrency (--rates 32,64 and up); latency numbers are the
same stack at low concurrency (--rates 1,2,4,8). Nothing else changes, so the
two profiles are directly comparable and a single warm stack can serve both.
The run itself is executed by redhat-et/vllm-cpu-perf-eval
(ansible + GuideLLM) in VLLM_ENDPOINT_MODE=external: our compose stack is the
endpoint, ansible only drives load and collects metrics. The harness that wires
the two together is vendored under harness/ from
amd/skills PR #81 and kept close to
upstream so re-syncs stay cheap.
Sections
Setup (one time)
scripts/setup-harness.sh
Clones vllm-cpu-perf-eval into harness/vllm-cpu-perf-eval/ (gitignored,
deliberately unpinned, same as upstream) and applies
harness/vllm-cpu-perf-eval.patch with git apply --3way (rootless guidellm
user fix, /tmp -> BENCH_TMPDIR redirect, local-model bind mount). Safe to
re-run. It also reports missing ansible collections
(containers.podman, ansible.posix, community.general) -- install any it
flags before the first run.
Pre-warm the HF cache once per model so the first instance does not download
under load:
HF_HOME=$HOME/.cache/hf-shared/huggingface hf download meta-llama/Llama-3.1-8B-Instruct
Golden rule: ask at every stage
Drive the user through the stages one at a time, offering a sensible default
at each. Do NOT run anything until the Stage 5 confirmation. bench.sh --dry-run prints the resolved configuration and every ansible command without
executing -- show that before the real run.
Fail fast
If any step fails, STOP immediately -- do not continue to the next rate,
workload or A/B leg. A "failure" includes: a non-zero exit from any script, an
image pull error, an instance that does not become healthy within the timeout, a
vllm serve/GuideLLM crash or traceback, an empty/zero result, or a container
OOM. When something fails: surface the exact error and the relevant log
(results/sweep_<label>.log, the run's guidellm.log, podman ps), tear down
anything still up (harness/stop.sh), and report the likely cause and fix.
Never silently push past an error or fabricate results.
Stage 1: profile, rates, workloads, duration
Ask for the profile first; it sets the rates and nothing else:
| Profile | --rates | Measures | Read from GuideLLM |
|---|
| throughput ("offline") | 32,64 (and up: 96, 128) | saturation: tok/s across the whole stack | Server Throughput Statistics |
| latency | 1,2,4,8 | per-request behaviour: TTFT / ITL / TPOT | Request Latency Statistics |
Rates are absolute at the load balancer, not per instance: --rates 64 with
--n 4 is ~16 concurrent requests per instance. Ask the user to confirm the
list; do not assume.
Then ask the workloads (-w, repeatable, default chat). All of them run
against one warm stack -- one GuideLLM run each, one results dir each -- so
adding a workload costs a load run, not another ~15 min warm-up.
scripts/bench.sh --list-workloads
| Workload | isl / osl | Shape |
|---|
chat | 512 / 512 | the primary working point |
chat_lite | 128 / 128 | short turns |
rag | 7680 / 512 | long retrieved context, short answer |
code | 1024 / 1024 | code generation, near the practical CPU OSL ceiling |
summarization | 2048 / 256 | long in, short out |
reasoning | 256 / 2048 | short in, long out |
chat_var, code_var, summarization_var | variable | realistic traffic (normal-distributed isl/osl) |
context_scaling_1k/4k/8k | 1k/4k/8k / 256 | TTFT vs context length |
embedding | 512 / 1 | embedding models only |
Mixing workloads is fine and cheap, but the context window is sized for the
largest of them (see below), so -w chat -w rag runs chat on an
8320-token window rather than 2048.
Finally the duration: --max-seconds, seconds per rate, default 300.
Offer 300; shorter runs (60-120) are for smoke tests only.
Stage 2: hardware check + core bands
scripts/check_hardware.sh
scripts/check_hardware.sh --mode multi -n <N> -c <CPI> \
--exclude 1-15,16-31 --slices
Ask for N instances and cores per instance (CPI); suggest the detected
recommendation (typically one instance per NUMA node, or per CCD group).
Enforced:
N * CPI <= physical cores remaining after the nginx and guidellm bands.
If it does not fit, do not run -- show the recommended N/CPI and
re-prompt.- Warn when
CPI does not divide the NUMA-node core count (instances would
straddle NUMA nodes).
Confirm the 3-band layout with the user -- changing a band moves the vLLM
instances:
nginx 1-15 (--nginx-cores) GuideLLM 16-31 (--guidellm-cpus) vLLM 32+
bench.sh excludes both bands from the pool, then takes the per-instance
cpusets from check_hardware.sh --slices, which walks physical cores and
keeps SMT siblings inside one instance. Announce the resolved sets (e.g.
cpusets: 32-47,160-175;48-63,176-191) and get a yes.
Stage 3: image + backend (optional native A/B)
Always ask for the image. Default: the newest amdih/zendnn_zentorch tag
(DEFAULT_IMAGE in scripts/common.sh, currently
vllm_v0.24.0_zentorch_v2.11.0.3_ubuntu22.04_2026_ww28); browse tags at
https://hub.docker.com/r/amdih/zendnn_zentorch/tags .
Prefer an image already present on the host (podman images) over pulling a new
tag, and note that comparing two separately-built images conflates zentorch with
the vLLM commit, torch build and compiler flags baked into each, so change one
thing at a time.
Then the backend, and ask whether an A/B against native is wanted:
--zentorch (default) -- zentorch backend, the ZENDNNL_* and inductor knobs
from scripts/incontainer_env.sh and harness/generate-config.sh.--native -- the same image with zentorch pip-uninstalled and committed as
<image>_native (built by harness/start.sh on first use, cached after
that), zentorch env knobs off. Deriving the native image rather than asking
for a second one is the point: two separately-built images also differ in
vLLM commit, torch build and compiler flags, and those differences show up in
the numbers as if they were zentorch's doing.
Ask it plainly, e.g. "zentorch only, or also a native (non-zentorch) run on the
same image so the two can be compared?" If A/B is wanted, that is two
bench.sh invocations, run back to back on the same host, identical apart
from --native and --run-tag:
scripts/bench.sh ... --run-tag _zentorch
scripts/bench.sh ... --native --run-tag _native
Run zentorch first, and treat the pair as one job: confirm both commands at
Stage 5, and if the first leg fails, stop and report rather than running the
second (see Fail fast). Comparing legs from different hosts, stack
shapes or images is not a zentorch-vs-native result. The first --native run
pays a one-off image build (a few minutes) on top of the usual warm-up.
Stage 4: model + memory
- model (
-m): HF repo id, or a path under --models-dir (bind-mounted).
Default meta-llama/Llama-3.1-8B-Instruct. Gated models need HF_TOKEN
exported. The test_name tag is derived from the basename; pass the upstream
"path | tag" form to choose it yourself.
--kv-cache-space N (GiB, per instance) -- VLLM_CPU_KVCACHE_SPACE. It
is the direct memory knob and is per instance, so N instances multiply it.
Raise for long-context workloads (rag, context_scaling_8k), lower it first
when a container gets OOM-killed.--dtype / --block-size -- only when the user asks; defaults come from
the image/vLLM. Both are passed straight to vllm serve.--max-model-len -- normally leave it alone (derived, see below). Set
below what the workloads need, bench.sh warns and the run fills with HTTP
400s.
Stage 5: confirm + run
Show the assembled command, get a yes, dry-run it, then run for real:
scripts/bench.sh --model meta-llama/Llama-3.1-8B-Instruct \
--n 4 --cpi 32 --rates 32,64 --max-seconds 300 \
-w chat -w rag --image <IMG> --kv-cache-space 40 [--dry-run]
scripts/bench.sh --model meta-llama/Llama-3.1-8B-Instruct \
--n 4 --cpi 32 --rates 1,2,4,8 --max-seconds 300 -w chat --image <IMG>
scripts/bench.sh ... --native --run-tag _native
bench.sh --help lists every flag. Use --run-tag so repeat sweeps do not
clobber each other's results/ files.
Call chain:
scripts/bench.sh flags -> env, cpusets, max-model-len, model tag
scripts/run_combo.sh mem poller + results/ + tmp/ + sweep log
harness/run_sweep.sh preflight, test_name, per-workload ansible loop
harness/start.sh generate-config.sh -> compose up -> health wait
ansible (perf-eval) GuideLLM against the NGINX endpoint
harness/stop.sh compose down
What bench.sh adds on top of the harness
Three adjustments the upstream harness does not make for a zentorch-on-EPYC
run:
--max-model-len is derived from the workloads, not hardcoded to 4096.
scripts/workload_info.py reads the vendored test-workloads.yml and
computes max(isl_max|isl + osl_max|osl + 128, declared --max-model-len)
across every -w, so a workload whose isl+osl exceeds the server's context
window (rag needs 8320, context_scaling_8k 16384) is not sent to an
undersized server.- Instance cpusets come from
check_hardware.sh --slices (physical cores,
SMT siblings kept together) instead of contiguous core arithmetic, which
straddles sibling threads on SMT-enabled EPYC. If slicing fails, bench.sh
warns and falls back to upstream's arithmetic.
- The
test_name tag is derived and length-checked here, so a long HF repo
id cannot fail ansible's ^[A-Za-z0-9-]{1,30}$ validator at preflight.
Plus: a provenance banner per instance (scripts/incontainer_serve.sh prints
pip list, the exported env and the exact vllm serve command into the
instance log), and several workloads per warm stack.
Results
Upstream layout, unchanged. Per GuideLLM run, inside the clone:
harness/vllm-cpu-perf-eval/results/llm/<model>/<workload>-<ts>-<test_name>/external-endpoint/
Alongside, from the wrapper (in the directory you ran bench.sh from):
results/sweep_<label>.log full sweep transcript
results/mem_<label>.csv podman stats samples + a final PEAK line
harness/sweep-logs/ per-run harness logs
Scoring rule: the authoritative numbers are in each run's guidellm.log under
Server Throughput Statistics (throughput profile) and Request Latency
Statistics (latency profile). Parse with:
scripts/parse_guidellm_log.py <path>/guidellm.log
<label> defaults to n<N>-<backend>-<tag> (e.g. n4-zentorch-llama-3-1-8b,
n4-native-llama-3-1-8b),
plus --run-tag.
Verify
scripts/check_hardware.sh alone prints topology + a recommended layout.scripts/bench.sh --list-workloads prints the workload table with the derived
context window per workload.scripts/bench.sh ... --dry-run prints the resolved banner and every ansible
invocation (with test_name, base_workload, guidellm_cpus) and exits
without starting a container.
Gotchas
podman info can hang on some hosts (>25 s), and harness/check-host.sh
calls it, so a run appears to stall before anything starts. Work around with
SKIP_HOST_CHECK=1 scripts/bench.sh ... after confirming podman itself works
(podman ps).- The vendored clone is unpinned (same as upstream). If upstream drifts, the
--3way apply can conflict; setup-harness.sh says so and leaves the tree for
manual resolution in harness/vllm-cpu-perf-eval/. Re-generate
harness/vllm-cpu-perf-eval.patch from the resolved diff if the fix should
stick.
- Missing ansible collections are the most common first-run failure. Install
whatever
setup-harness.sh marks MISS.
sudo is not required. The harness auto-detects it (sudo -n true) and
passes -e ansible_become=false when unavailable; --no-become forces it.- Rates are absolute, at the load balancer.
GUIDELLM_MAX_CONCURRENCY is set
to at least the largest rate, otherwise the concurrent profile silently clamps.
- Container OOM while the host has free RAM is the per-instance
mem_limit
cgroup cap. Lower --kv-cache-space first (per instance, so N instances
multiply it); run_combo.sh already passes --no-mem-limit by default, which
drops only that cap and keeps cpuset/shm/caps.
- Rootless podman DNS: instances get static IPs on a dedicated /24
(
10.201.0.0/24) because aardvark-dns is unreachable on the bridge; NGINX
reaches them by IP.
- HF offline:
run_combo.sh exports HF_HUB_OFFLINE=1 and a placeholder
HF_TOKEN. Pre-warm the cache (see Setup) or the run fails resolving the repo.
- Do not compare across stack shapes. N, CPI, the core bands and
--max-model-len all move the numbers; change one thing at a time. For a
zentorch-vs-native A/B this means the two legs differ only by --native, on
the same host, back to back.
- The
_native image is cached once built (podman images | grep _native).
If the base image is rebuilt under the same tag, delete the stale
or the A/B compares against an old build.
Files
.claude/skills/benchmark-vllm/
SKILL.md # this file
reference.md # upstream harness notes (from amd/skills PR #81)
benchmark-vllm-flow.mmd # raw Mermaid source for the flow diagram
scripts/
bench.sh # THE entry point: flags -> harness env
setup-harness.sh # one-time: clone + patch vllm-cpu-perf-eval
check_hardware.sh # lscpu topology, N/CPI validation, --slices
workload_info.py # workload table -> derived --max-model-len
common.sh # DEFAULT_IMAGE + model-name helper
run_combo.sh # mem poller + results/ + tmp/ + sweep log
mem_poll.sh # podman stats sampler (PEAK line)
incontainer_serve.sh # (in container) entrypoint + provenance banner
incontainer_env.sh # (in container) LD_PRELOAD + tuning env
parse_guidellm_log.py # guidellm.log -> throughput / latency table
extract_perf.py, detect.py # upstream helpers
Dockerfile.buildB # upstream alternate image build
harness/ # vendored from amd/skills PR #81
run_sweep.sh # preflight, test_name, per-workload ansible loop
start.sh / stop.sh # compose up (+ native image build) / down
generate-config.sh # compose + nginx.conf + .env
check-host.sh # host preflight (see the podman info gotcha)
vllm-cpu-perf-eval.patch # patch applied to the clone
vllm-cpu-perf-eval/ # the clone itself (gitignored, unpinned)
