| name | sglang-xpu-run |
| description | Serve a Hugging Face safetensors model on an Intel GPU using SGLang's XPU backend with the OpenAI-compatible API. Covers pulling the pre-built `intel/sglang-dev:latest` image, fixing the render-group and UMD/kernel compatibility issues that affect non-root sglang images, the SYCL_UR / Level Zero env vars needed on Battlemage, the `--device xpu --attention-backend intel_xpu` flag set, multimodal serving, and how to validate output content (not just HTTP 200). Use when the user needs SGLang's RadixAttention prefix caching or grammar-constrained output; for broad-coverage serving on Intel today prefer vllm-xpu-run, and for benchmarking a running server use sglang-xpu-bench. |
sglang-xpu-run
SGLang's XPU backend is functional. Use the pre-built
intel/sglang-dev:latest image. Verify GPU detection before
serving; SGLang silently falls back to CPU and you'll only notice when
throughput is 50× lower than expected.
If you don't specifically need RadixAttention prefix caching or
grammar-constrained output, prefer vllm-xpu-run — its Intel coverage
is broader today.
Step 0 — discover GPUs and host RAM before anything else
Run xpu-discover first, or at minimum:
xpu-smi discovery
GPU_COUNT=$(xpu-smi discovery 2>/dev/null | grep -cE "^\| +[0-9]|Device [0-9]+:")
[ "${GPU_COUNT:-0}" -gt 0 ] || GPU_COUNT=1
RAM_GB=$(awk '/MemTotal/{print int($2/1024/1024)}' /proc/meminfo)
RENDER_GID=$(getent group render 2>/dev/null | cut -d: -f3)
RENDER_GID=${RENDER_GID:-$(stat -c '%g' /dev/dri/renderD128 2>/dev/null)}
echo "GPUs: $GPU_COUNT RAM: ${RAM_GB} GB render GID: $RENDER_GID"
Use GPU_COUNT to set ZE_AFFINITY_MASK and --tp; use RAM_GB to
set MAX_JOBS for the build; use RENDER_GID in every docker run
command below.
CUDA → XPU cheat sheet
| CUDA | Intel |
|---|
lmsysorg/sglang:latest | intel/sglang-dev:latest |
--gpus all | --privileged --device /dev/dri -v /dev/dri/by-path:/dev/dri/by-path --group-add <render-gid> |
--device cuda (implicit) | --device xpu + --attention-backend intel_xpu |
--tp 2 | --tp 2 + ZE_AFFINITY_MASK=0,1 |
--quantization awq | silently broken on XPU (HTTP 200, garbage content) |
--quantization fp8 | works (runtime BF16→FP8 weight conversion) |
| no env vars | needs SYCL_UR_USE_LEVEL_ZERO_V2=0 on Battlemage |
Pull the image
docker pull intel/sglang-dev:latest
Use intel/sglang-dev:latest in all docker run commands below.
Pre-flight: verify XPU is visible inside the container
RENDER_GID=$(getent group render 2>/dev/null | cut -d: -f3)
RENDER_GID=${RENDER_GID:-$(stat -c '%g' /dev/dri/renderD128 2>/dev/null)}
docker run --rm \
--privileged --network host --ipc=host --shm-size=32g \
--device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path \
--group-add "$RENDER_GID" \
-e ZE_AFFINITY_MASK=0 \
-e SYCL_UR_USE_LEVEL_ZERO_V2=0 \
--entrypoint /bin/bash \
intel/sglang-dev:latest \
-lc 'CONDA_SH=$(find /home /root /opt -maxdepth 5 -name activate -path "*/miniforge*/bin/activate" 2>/dev/null | head -1); \
. "${CONDA_SH:-$HOME/miniforge3/bin/activate}" && \
conda activate py3.12 && \
source /opt/intel/oneapi/setvars.sh --force >/dev/null && \
python -c "import torch; n=torch.xpu.device_count(); \
print(\"xpu count:\", n); \
assert n>0, \"NO XPU VISIBLE\""'
If xpu count: 0 see Common errors below.
Why each container flag
--privileged — required on Battlemage with current images. Reducing
to plain --device /dev/dri --group-add render is on the upstream
roadmap but not yet stable.--group-add "$RENDER_GID" — the sglang image runs as non-root user
sdp; /dev/dri/renderD* nodes are mode 660 owned by the render
group. Without this, zeInit fails with EACCES even with
--privileged. vLLM images run as root and skip this issue — that's
why the flag isn't in most vLLM examples.--network host — simplifies port handling (sglang uses 30000 +
internal RPC ports). Drop and add -p 30000:30000 if you don't want
host networking.-v /dev/dri/by-path:/dev/dri/by-path — Level Zero discovers devices
via /by-path symlinks; without it some images report no XPU.--shm-size=32g (or --ipc=host) — sglang scheduler uses shared
memory more aggressively than vLLM. Default Docker shm is too small.SYCL_UR_USE_LEVEL_ZERO_V2=0 — SYCL Unified Runtime's v2 L0 adapter
has a device-discovery bug on Battlemage; v1 is the verified
workaround.source /opt/intel/oneapi/setvars.sh --force — sets oneCCL, oneMKL,
Level Zero env paths inside the conda env.
Quickstart — serve one text-gen model
unset ALL_PROXY all_proxy
hf download Qwen/Qwen3-0.6B --local-dir "$HOME/models/Qwen3-0.6B"
Launch (single GPU — for multi-GPU see below):
RENDER_GID=$(getent group render 2>/dev/null | cut -d: -f3)
RENDER_GID=${RENDER_GID:-$(stat -c '%g' /dev/dri/renderD128 2>/dev/null)}
docker run -d --rm --name sglang-xpu \
--privileged --network host --ipc=host --shm-size=32g \
--device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path \
-v "$HOME/models:/models" \
--group-add "$RENDER_GID" \
-e ZE_AFFINITY_MASK=0 \
-e SYCL_UR_USE_LEVEL_ZERO_V2=0 \
--entrypoint /bin/bash \
intel/sglang-dev:latest \
-lc 'CONDA_SH=$(find /home /root /opt -maxdepth 5 -name activate -path "*/miniforge*/bin/activate" 2>/dev/null | head -1); \
. "${CONDA_SH:-$HOME/miniforge3/bin/activate}" && \
conda activate py3.12 && \
source /opt/intel/oneapi/setvars.sh --force >/dev/null && \
exec python -m sglang.launch_server \
--model /models/Qwen3-0.6B \
--device xpu \
--tp 1 \
--attention-backend intel_xpu \
--disable-overlap-schedule \
--page-size 64 \
--host 0.0.0.0 --port 30000'
Wait for Application startup complete (docker logs -f sglang-xpu),
then confirm the API is reachable and the model is loaded:
for i in $(seq 1 60); do
curl -sf http://127.0.0.1:30000/v1/models >/dev/null && break
sleep 2
done
curl -s http://127.0.0.1:30000/v1/models | python3 -m json.tool
Send a warmup request before trusting latency numbers — the first
request compiles Triton kernels and is 10–60× slower:
curl -s http://127.0.0.1:30000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"/models/Qwen3-0.6B",
"messages":[{"role":"user","content":"warmup"}],
"max_tokens":8}' > /dev/null
Now smoke-test — validate content, not just HTTP 200:
curl -s http://127.0.0.1:30000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"/models/Qwen3-0.6B",
"messages":[{"role":"user","content":"Say hi in one sentence."}],
"max_tokens":64}' | python3 -c "
import sys, json
r = json.load(sys.stdin)
content = r['choices'][0]['message']['content']
print('content:', content[:120])
assert len(content) > 10 and not set(content).issubset(set('! ')), \
'looks like garbage — check quant and XPU placement'
print('SMOKE TEST PASS')
"
Cleanup: docker stop sglang-xpu.
Multi-GPU (TP)
First discover available XPUs and validate the requested TP degree:
GPU_COUNT=$(xpu-smi discovery 2>/dev/null | grep -cE "^\| +[0-9]|Device [0-9]+:")
[ "${GPU_COUNT:-0}" -gt 0 ] || GPU_COUNT=1
TP=${TP:-$GPU_COUNT}
if [ "$TP" -gt "$GPU_COUNT" ]; then
echo "ERROR: requested TP=$TP but only $GPU_COUNT XPU(s) available" >&2
exit 1
fi
MASK=$(python3 -c "print(','.join(str(i) for i in range($TP)))")
RENDER_GID=$(getent group render 2>/dev/null | cut -d: -f3)
RENDER_GID=${RENDER_GID:-$(stat -c '%g' /dev/dri/renderD128 2>/dev/null)}
echo "Launching TP=$TP on XPUs: $MASK (of $GPU_COUNT available)"
Launch:
docker run -d --rm --name sglang-xpu-tp \
--privileged --network host --ipc=host --shm-size=32g \
--device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path \
-v "$HOME/models:/models" \
--group-add "$RENDER_GID" \
-e ZE_AFFINITY_MASK="$MASK" \
-e SYCL_UR_USE_LEVEL_ZERO_V2=0 \
--entrypoint /bin/bash \
intel/sglang-dev:latest \
-lc "CONDA_SH=\$(find /home /root /opt -maxdepth 5 -name activate -path '*/miniforge*/bin/activate' 2>/dev/null | head -1); \
. \"\${CONDA_SH:-\$HOME/miniforge3/bin/activate}\" && \
conda activate py3.12 && \
source /opt/intel/oneapi/setvars.sh --force >/dev/null && \
exec python -m sglang.launch_server \
--model /models/<model> \
--device xpu --tp $TP \
--attention-backend intel_xpu \
--disable-overlap-schedule --page-size 64 \
--host 0.0.0.0 --port 30000"
SGLang flag rationales
--device xpu — explicit. SGLang doesn't auto-detect XPU as cleanly
as vLLM does.--attention-backend intel_xpu — verified-good SYCL kernel path.
Without it you may land on triton, which is slower and has patchier
coverage.--disable-overlap-schedule — sglang's overlapped CPU/GPU scheduler
has known stalls on XPU; keep disabled until upstream fixes land.--page-size 64 — KV-cache page size that holds up best on
Battlemage; default may be smaller.--tp N — ZE_AFFINITY_MASK must expose exactly N XPUs.--trust-remote-code — security opt-in, not in the launch lines above.
Permits arbitrary Python from the model repo to run in the engine. Add it
only when the model declares repo-local code and you trust the publisher.
Check before assuming you need it, per model: the flag is required only if
that model's own config.json contains an auto_map entry. Most mainstream
models do not. Decide from the config, not from a remembered example.--disable-radix-cache — disables RadixAttention prefix caching.
Useful for A/B comparisons against vLLM or to isolate decode
throughput without prefix-cache effects. Omit to keep caching on
(the default and the main reason to use SGLang over vLLM).
Quantization: what works, what silently fails
| Quant | Status |
|---|
fp8 | works — runtime BF16→FP8, ~half BF16 size, KV stays BF16 unless --kv-cache-dtype fp8_e4m3 |
awq | silently broken on XPU — HTTP 200, content is non-language |
gptq, marlin, awq_marlin, bitsandbytes, mxfp8, mxfp4, compressed-tensors, modelopt_* | unverified — validate content |
| AutoRound | supported since Oct 2025; auto-detected via quantization_config.quant_method=auto-round |
Never accept HTTP 200 alone — read choices[0].message.content and
confirm it parses as language.
Speculative decoding, multimodal
See references/spec-decode-and-multimodal.md for EAGLE / MTP
spec-decode flags and verified multimodal models.
Not on this stack today
Route to vllm-xpu-run if you need any of:
- DeepSeek MLA path.
- MoE fused kernels.
- LoRA hot-swap.
torch.compile for the model graph.
Common errors
xpu count: 0 — two independent causes:
- UMD/kernel mismatch: image driver may be newer than the host
kernel supports. Confirm with:
docker run --rm intel/sglang-dev:latest dpkg -l libze-intel-gpu1 | grep "^ii"
and compare to host dpkg -l libze-intel-gpu1.
- Render group: sglang image runs as non-root
sdp; /dev/dri/renderD*
require the render group. Fix: add
--group-add "$(getent group render | cut -d: -f3)" to docker run.
(vLLM images run as root and skip this issue.)
ZE_RESULT_ERROR_UNINITIALIZED from zeInit → UMD mismatch (cause 1
above).EACCES on /dev/dri/renderD* → render group missing (cause 2).triton.compiler.errors.CompilationError on first request → cold
Triton cache. Mount TRITON_CACHE_DIR to a host volume.RuntimeError: NCCL → sglang prints "NCCL" on the XPU path; actual
backend is oneCCL. Usually a --tp / ZE_AFFINITY_MASK mismatch.- HTTP 200 with garbage (
!!!!!!!!...) → AWQ quant. Use fp8 or BF16.
- Server hangs at "Compiling kernel" → cold Triton cache on first run.
Watch
xpu-smi dump -d 0 -m 18 -i 1 — high power = compiling, not
stuck. Persist the cache for next run.
429 Too Many Requests pulling image → Docker Hub rate-limiting
anonymous pulls. Fix: docker login for higher limits, or retry
after 60 seconds.
Env vars
| Variable | Purpose |
|---|
ZE_AFFINITY_MASK | Which XPU(s) the server sees (0, 0,1, …). |
SYCL_UR_USE_LEVEL_ZERO_V2=0 | Force L0 v1 adapter; v2 is buggy on Battlemage. Required. |
TRITON_CACHE_DIR | Persist compiled kernels across runs. |
HF_TOKEN | HF auth when downloading inside the container. |
SGLANG_LOGGING_LEVEL=DEBUG | Verbose engine logs. |
ALL_PROXY / all_proxy | Unset if set to socks:// — hf CLI doesn't support SOCKS. |
What this skill does NOT cover
- Pure PyTorch / Transformers → torch-xpu-run.
- vLLM serving → vllm-xpu-run.
- Benchmarking → sglang-xpu-bench.
References
