model-config-recommend

name

model-config-recommend

description

Recommend how to configure vLLM-XPU for a Hugging Face decoder-only LLM on Intel Arc B-series GPUs: choose quantization, KV dtype, DP/TP layout, max concurrency, and max context using roofline math against published hardware specs. Use when the user asks "How should I configure vLLM?", requests the best vLLM configuration, or asks how to use one or multiple Arc cards. Must run recommend.py; predictions are physics-bounded ranges, not measured throughput. Use after xpu-discover and before vllm-xpu-run.

model-config-recommend

Status: experimental. Emits physics-bounded predictions, not measurements. Scoped to vLLM-XPU; for SGLang use model-can-it-fit + sglang-xpu-run + sglang-xpu-bench.

When to use

The user has chosen an HF decoder-only LLM and wants to know which vLLM-XPU config (quant, KV dtype, DP/TP, max concurrency, max context) to start with on Intel Arc B-series. Broad wording such as "How should I configure vLLM for this model on my Arc cards?" still activates this skill because it requires selecting layout and deployment parameters rather than merely launching a server. Skip when:

• Exact throughput numbers required -> use vllm-xpu-bench.
• VLM or diffusion model -> use model-can-it-fit.
• Hardware not in data/hardware.json (other Intel families).
• MoE expert parallelism or speculative-decoding speedup — both workload-specific; the skill flags them as bench-only.

Mandatory output contract (read first)

Every recommendation answer MUST do all three, even when the model fits on one GPU — there is no "it's small, skip this" exception:

1. Run recommend.py this session against the user's actual model, device, context, and concurrency. Never answer from memory or from the "Worked example" numbers below.
2. State the layout as the literal dp=N, tp=M token (e.g. dp=1, tp=1). Prose like "no TP needed" does not count.
3. Reproduce the full docker run ... <model> ... launch block verbatim in a fenced code block. Never a flag table, a bare vllm serve line, or a "block above" pointer instead.

"Reporting the recommendation" below has the detail.

Three tiers

Tier	Script	What it does
1	recommend.py	Pulls config.json, applies roofline math against the spec table, emits candidates + launch line. Stdlib only, no GPU. ~2s.
2	calibrate.py	Runs a short BF16 bench on a reference model (Qwen/Qwen2.5-1.5B-Instruct by default), measures actual MFU/BWE, caches per (image, device). Requires Docker.
3	verify.py	Launches the recommended config on the target model, prints predicted-vs-measured with IN BAND / OUT OF BAND flags.

# Tier 1
python3 scripts/recommend.py \
    --model Qwen/Qwen2.5-7B-Instruct \
    --device arc-pro-b70 \
    --num-devices 2 --discover-host --runtime vllm-xpu \
    --ctx 4096 --concurrency 4 \
    --gpu-memory-utilization 0.85

# Tier 2 (one-time per image)
python3 scripts/calibrate.py --image vllm/vllm-openai-xpu:latest --device arc-pro-b70
# Then re-run Tier 1 with --use-calibration vllm/vllm-openai-xpu:latest

# Tier 3 (verify a candidate)
python3 scripts/verify.py --image vllm/vllm-openai-xpu:latest --device arc-pro-b70 \
    --model Qwen/Qwen2.5-1.5B-Instruct --quant fp8 --kv-dtype fp8 \
    --ctx 4096 --concurrency 4

For a plain roofline readout without a launch recommendation:

python3 scripts/llm_roofline.py --model <id> --device <id> \
    --quant fp8 --kv-dtype fp8 --ctx 8192 --concurrency 8

Use --discover-host whenever the recommendation is for the current machine. It runs xpu-smi discovery first and fails if --num-devices exceeds the Intel XPUs visible on the host. Omit it only for what-if planning for a different host.

What recommend.py outputs per candidate

• Layout dp=N, tp=M. TP shards one replica across GPUs (use when weights/KV don't fit on one GPU); DP runs independent replicas (use when the model fits per GPU). Recommender prefers DP when the model fits.
• Fit breakdown weights / KV / activations / framework per GPU. Fit is checked against device_vram × --gpu-memory-utilization, matching the vLLM launch flag emitted for the top candidate.
• Capacity ceilings max concurrency at target context, max context at target concurrency (memory-only).
• Predicted TTFT band (compute roofline, prefill).
• Predicted decode band step latency at target concurrency, aggregate decode tok/s.
• Caveats for the (quant, runtime) pair.
• Available checkpoints in each quant from a Hub search.
• Launch line for the top candidate.

Reporting the recommendation

The launch block recommend.py prints begins with docker run --rm -d --name vllm-xpu ... and ends with the model id plus serve flags. Copy that whole block into a fenced ```bash block; the recommender's stdout is not shown to the user, so a pointer like "the block above" leaves them with nothing to run. A flag table is fine in addition, never instead.

This skill only plans — it never launches anything itself. Present the block as a command for the user to review and run, and have them confirm the image tag and device mask before running it on their host.

Pinning the model revision

--revision REF takes a commit SHA, tag, or branch. It sets the revision the config is read from and the revision the emitted launch line pins, so the plan and the deployment describe the same artifact.

Left unset, vLLM resolves the repo's default branch when the server starts: a later push by the publisher silently changes what is served, and the run stops being reproducible. recommend.py prints a # NOTE saying so. A commit SHA is the only value that cannot be moved.

For models whose config.json declares auto_map, the pin stops being hygiene and becomes the mitigation. recommend.py therefore requires --revision alongside --trust-remote-code for any Hub model and exits 2 without it: granting arbitrary Python execution against a moving branch means the engine runs the publisher's latest push, not the code anyone reviewed. When both are given, the launch line carries --revision and --code-revision — vLLM resolves repo-local modeling code separately from the weights, so pinning one does not pin the other. A local --model /path/config.json has no branch to move and needs no pin.

Quants and compute tiers on Battlemage

--quantization drives both kernel selection and which compute tier the matmuls land on. The Xe2 XMX engines run INT8 at 367 TOPS vs FP16/BF16 at 45.88 TFLOPS (Arc Pro B70 spec). The roofline models W4A8 as INT8-tier when the W4A8 kernel is active; FP8/BF16 stay on the FP16 tier.

--quantization	Bytes/param	XMX tier	CLI pairing
(omit; BF16/FP16)	2.0	FP16/BF16 (45.88 TF on B70)	--dtype bfloat16, --kv-cache-dtype auto
fp8	1.0	FP16/BF16 (no native FP8 DPAS — kernel dequantizes FP8->BF16)	--kv-cache-dtype fp8, --attention-backend TRITON_ATTN
awq / gptq	0.55	INT8-modeled when W4A8 kernel is active	--kv-cache-dtype fp8, --attention-backend TRITON_ATTN
inc	0.55	INT8-modeled when W4A8 kernel is active	INC-quantized pipelines only; most AutoRound checkpoints auto-detect via gptq/awq instead.
mxfp4	0.55	INT8-modeled W4A8-style path	Microscaling FP4 (GPT-OSS family).
AutoRound checkpoint	0.55	INT8-modeled when W4A8 kernel is active	Omit --quantization — vLLM auto-detects from quantization_config.quant_method=auto-round and routes through gptq/awq loader.

W4A8 decode win is workload-dependent on Battlemage

The roofline predicts INT4 quants win decode by ~4× over BF16 from the weight-streaming reduction. On current vLLM-XPU images, how close the measurement comes depends on:

• W4A8 kernel coverage on Battlemage's INT8 XMX tier is still maturing; some matmul shapes route through the BF16 path.
• Online dequant overhead the static formula treats as free.
• For MoE with low active-parameter ratios, routing cost can dominate weight-streaming savings.

Treat any INT4 candidate as "verify with bench" rather than "preferred." The fit/headroom argument (smaller weights -> larger context, larger concurrency) holds independently of throughput.

Roofline model

weights     = params × bytes_per_param / tp
kv_cache    = 2 × num_layers × num_kv_heads × head_dim × kv_dtype_bytes × ctx × concurrency / tp
activations ≈ 2 × concurrency × ctx × hidden × dtype_bytes + 512 MiB
framework   = vllm-xpu overhead (~2.0 GB)

prefill_tok_s = peak_compute × MFU / (2 × params)         # compute-bound
TTFT          = prompt_tokens / prefill_tok_s

single_stream_decode_tok_s = (mem_bw / weights_bytes) × BWE   # memory-bound
TPOT_at_concurrency        = step time including KV traffic at (ctx, concurrency)

MFU (compute efficiency) and BWE (bandwidth efficiency) live in data/hardware.json as ranges per runtime — see "Worked example" below for why bands are wide. Roofline reference: Williams, Waterman, Patterson, Communications of the ACM 52(4), 2008.

Hardware coverage (data/hardware.json)

Card	PCI device ID	Die	VRAM	Bandwidth	FP16/BF16	INT8
Arc B580	0xe20b	BMG-G21	12 GB	456 GB/s	26.9 TF	233 TOPS
Arc Pro B50	0xe220 ¹	BMG-G21 (LP)	16 GB	224 GB/s	21.30 TF	170 TOPS
Arc Pro B60	0xe211 ¹	BMG-G21	24 GB	456 GB/s	24.56 TF	197 TOPS
Arc Pro B65	0xe221 ¹	BMG-G31 (cut)	32 GB	608 GB/s	24.56 TF	197 TOPS
Arc Pro B70	0xe223	BMG-G31 (full)	32 GB	608 GB/s	45.88 TF	367 TOPS

¹ Provisional — derived from the xe kernel driver's INTEL_BMG_IDS list; not yet verified by lspci on physical hardware.

Each row in the JSON cites Intel's SKU spec page directly. The PCI device ID is what xpu-smi discovery and lspci -d 8086: show in brackets (e.g. Intel(R) Graphics [0xe223]).

Worked example

This shows the shape of a recommend.py run, not real numbers — the values are deliberately <placeholders> so they can't be pasted into an answer. It is not a substitute for running the script: checkpoints, spec rows, and fit math all change with the inputs. If your answer has a number you didn't get from a recommend.py run this session, it is wrong.

Qwen/Qwen2.5-7B-Instruct, 8K context, concurrency 8, Arc Pro B70 produces a block shaped like this:

--- Candidate 1: --quant <q>, --kv-cache-dtype <kv>, dp=<N>, tp=<M> ---
  Fit:       <W> GB weights + <K> GB KV + <A> GB act + <F> GB framework
             = <T> GB / <U> GB usable per GPU (headroom <H> GB)
  Capacity:  up to <C> concurrent at <ctx> tok per replica; ...
  Predicted: TTFT <lo>–<hi> ms; decode step <lo>–<hi> ms at concurrency <c>;
             aggregate decode <lo>–<hi> tok/s across DP=<N>
  ... (more candidates) ...

=== Launch line for top candidate ===
docker run --rm -d --name vllm-xpu \
    ...
    vllm/vllm-openai-xpu:latest \
    <checkpoint> \
        ...

Bands are wide because community-typical MFU/BWE ranges cover both maturing and well-tuned kernels. Run calibrate.py to tighten to your image, then verify.py to collapse to a measurement.

What this skill does NOT predict

• Expert Parallelism (EP) for MoE — workload-specific. Bench --enable-expert-parallel against --tensor-parallel-size N.
• Speculative decoding speedup (n-gram, EAGLE, EAGLE3, MTP) — depends on draft accuracy on the target distribution.
• Tuning knobs --block-size, --max-num-batched-tokens, --max-num-seqs — emitted as starting points, sweep with vllm-xpu-bench.
• Tokens-per-second point estimates. Bands only.

Common errors

• HTTP 401 fetching config.json -> typo, gated repo, or rate limit. Set HF_TOKEN or pass --config-path <local file>.
• "config.json missing required keys" -> not a decoder-only LLM. Use model-can-it-fit for VLMs.
• Empty candidates list -> (model, device, ctx, concurrency) doesn't fit. Reduce concurrency / context, or use a more compressed quant.
• GGUF repo with no config.json -> use a safetensors checkpoint for vLLM-XPU, or a llama.cpp SYCL workflow for GGUF.

What this skill does NOT cover

• Running the model — use vllm-xpu-run with the emitted line.
• Benchmarking — use vllm-xpu-bench.
• Fitting non-LLMs — use model-can-it-fit.

References

• Roofline model (Williams, Waterman, Patterson, 2008): https://dl.acm.org/doi/10.1145/1498765.1498785
• vLLM Arc Pro B-series serving recipe (2025-11-11): https://blog.vllm.ai/2025/11/11/intel-arc-pro-b.html
• Intel Arc B-series SKU spec pages — one URL per row in data/hardware.json.