# Unofficial Intel XPU Optimization Lab
MiniMax-H3: [owner-approved Comfy-Org published pruned BF16 video package](repro/minimax-h3-pruned-bf16-tp2-b70-20261004/README.md), two B70s, measured 396.6 seconds per clip in an eight-clip batch. Exact scheduling against its own reference; public rebuild incomplete.
Community setup guides, benchmark recipes, troubleshooting notes, and patches
for local AI work on Intel XPUs. This is not a single-model repo. It is a
working lab notebook and reproducibility collection for multiple model lanes
that we revisit as new runtime, compiler, and kernel ideas appear.
This is experimental research, not a supported product. Commands, patches,
and benchmark observations are provided under the repository [LICENSE](LICENSE)
and the risks described in [DISCLAIMER.md](DISCLAIMER.md); review and use them
at your own risk.
## Who This Is For
- Local AI users who want reproducible Intel Arc/B-series commands, patches and
benchmark recipes.
- Anyone deciding whether Intel Arc/B-series hardware is worth it for local
inference, and wanting to see the state of support.
- Optimization agents and contributors who need a map of current work,
archived lessons, and validity rules.
- Upstream vLLM, llama.cpp, oneAPI, SYCL, and Intel/XPU developers looking for
concrete repros and failure signatures.
## Start Here
| Need | Entry Point |
| --- | --- |
| Understand the repo structure | [Docs index](docs/README.md) |
| Find an experiment, retained patch, or cleanup decision | [Experiment index](experiments/README.md), [patch catalogs](patches/README.md), and [cleanup audit](audits/repository-cleanup/README.md) |
| See every active, paused, and archived model lane | [Model effort index](docs/model-effort-index.md) |
| Find a tested result or reproduction artifact | [Results index](results/README.md), [classified reproduction catalog](repro/README.md), and [certification standard](docs/reproduction-guide-certification.md) |
| Try a packaged candidate | Browse the [model guide library](guides.html) or its [machine-readable package catalog](packages/catalog.json); each package states its certification status |
| Review local coding-worker trials | [Trial history](experiments/local-coding-worker/README.md) and [October 7 scoped closeout](experiments/local-coding-worker/scoped-task-20261007/CLOSEOUT.md); further worker tuning and integration are parked |
| Start optimizing a new model | [Model optimization guide](docs/model-optimization-guide.md) |
| See or prepare the next model downloads | [Model intake queue](model-intake/README.md) |
| Follow the Docker and Windows packaging path | [Distribution and packaging roadmap](docs/model-distribution-and-packaging-roadmap.md) |
| Plan PCIe slots, risers, or external GPUs | [PCIe topology and local-LLM inference](docs/pcie-topology-and-llm-inference.md) |
| Diagnose freezes, resets, or silent data corruption | [Host stability and fault diagnosis](docs/host-stability-and-fault-diagnosis.md) |
| Compare expected model performance | [Performance scoreboard](results/scoreboard.md) |
| Contribute a result, patch, or correction | [Contribution guide](CONTRIBUTING.md) and [verification policy](docs/contribution-verification.md) |
| Review or validate incoming work | [Manager playbook](MANAGER.md) |
| Reuse the best research prompts/workflows | [Research workflow playbook](docs/research-workflow-playbook.md) |
| Find recorded host state, restrictions, and active work | [Current workspace state](CURRENT.md) |
| Find retained recipes and their limits | [Current reproducibility map](docs/current-reproducibility-map.md) |
| Submit or audit LocalMaxxing records | [LocalMaxxing submissions](docs/localmaxxing.md) |
| Handle local ops, secrets, sudo, and cross-agent delegation | [Local ops](docs/local-ops.md) |
| Review Intel-facing issues and asks | [Feedback for Intel](docs/feedback-for-intel.md) |
## How The Repo Is Organized
The repo is organized around model lanes, not one-off leaderboard
rows. A serious lane should leave behind:
- `results/<model>-<quant>-<hardware>/`: promoted or closed-out result packet,
validity gates, best commands, invalid fast lanes, and lessons.
- `repro/<model>-.../`: classified reproduction artifacts, ranging from
portable candidates to originating-host lab replays and archived records;
the catalog, not the directory name, states portability.
- `packages/<model>-.../`: user-facing, machine-readable front doors over
exact in-repository guides, commands, dependencies, and evidence.
- `experiments/<model>-.../`: active research lanes that are not production
recipes yet.
- `notes/`: chronological lab notebook entries, including negative results and
postmortems.
- `patches/`: patch snapshots and source/config deltas, including failed
experiments worth preserving.
- `data/`: compact structured benchmark records, payloads, responses, and logs.
- `scripts/`: reusable harnesses, analyzers, launchers, and submission helpers.
- `model-intake/`: revision-pinned candidate catalog, storage budget, and the
boundary between discovery, download, validation, and promotion.
- `community/<contributor>-...`: runnable work contributed from outside the
reference lab, with explicit evidence status and validation history.
- `community/field-reports/`: unverified measurements and observations from
community systems, kept separate from recipes, patches, validation assets,
and promoted results. See [`community/README.md`](community/README.md).
The point is to make model switching cheap. Gemma, Qwen, MiniMax, and future
lanes should all share validation discipline, result-packet shape, and reusable
kernel/runtime lessons without dragging stale generated builds or huge artifacts
forward.
## Representative Results and Public Packages
The [official 27B FP8 quickstart](packages/qwen38-27b-fp8-tp2-b70/README.md)
now uses the qualified October 17 one-launch attention overlay, MTP5 and 32K
input capacity. Its October 19 public-source acceptance passed 12/12 strict
reference outputs, six practical requests with exact repeats, and a clean stop.
It measured 46.013 tok/s on steve-b70s; the historical 90.3 tok/s pair on Turin
remains separately identified. These different hosts and runtime releases are
not a matched speed comparison.
The latest [own Xe runtime one-card recipe](repro/qwen38-flash-next-iq3xxs-ownrt-tp1-b70/README.md)
and [two-card recipe](repro/qwen38-flash-next-iq3xxs-ownrt-tp2-b70/README.md)
now include two full-512 suites and distinct 512-input prefill observations.
They remain expert lab replays: complete outputs repeat exactly, but independent
oracle coverage stops at 128 tokens and portable admission remains incomplete.
The 36.603/63.612 tok/s capped records are documented separately, not substituted
for full-512 results or promoted through an incomplete quality gate.
| Updated text recipe | Decode observations | Prefill / context / concurrency coverage |
| --- | --- | --- |
| [27B FP8 TP2](packages/qwen38-27b-fp8-tp2-b70/README.md) | Current-host acceptance 46.013 tok/s; historical Turin 90.3 tok/s | 512-input synthetic prefill 2164.081 tok/s (six requests); separate overlay screens 2K–32K and validated multi-user profile through 64 requests |
| [Flash-Next IQ3 own runtime TP1](repro/qwen38-flash-next-iq3xxs-ownrt-tp1-b70/README.md) | Full 512 cap 31.510–31.782 tok/s, two loads | 512 / native TTFT reading 32.595–32.752 tok/s; context curve absent; one request at a time |
| [Flash-Next IQ3 own runtime TP2](repro/qwen38-flash-next-iq3xxs-ownrt-tp2-b70/README.md) | Full 512 cap 58.557–58.632 tok/s, two loads | 512 / native TTFT reading 75.884–75.973 tok/s; context curve absent; one request at a time |
| [Gemma 4 Q8](packages/gemma4-26b-a4b-q8-b70/README.md) | Qualified 122.160357 tok/s; existing record | Separate production prefill 1307.54/1167.81/909.27 tok/s at 1867/7623/29400 actual tokens; faster kernel remains unshipped |
| [Nemotron 3.5 UD-Q4_K_M](repro/nemotron-35-lightning-30b-a3b-b70/README.md) | Historical 72.17/72.04 tok/s withheld from strict headline | Raw depth sweep retained; full-suite hashes match 1/12; owner deferred strict replay |
The tables describe different measurement scopes. See each recipe for raw
receipts, exact flags, quality gates, sample counts and missing measurements.
Official **27B FP8 reading speed** now includes a 2,048-token prompt measurement
and a final bounded profiling review. [Results and evidence](experiments/qwen38-27b-b70/notes/2026-09-14-fp8-prefill-focus-results.md)
cover two B70s, one user and fixed MTP1; no new optimization was promoted.
A bounded [AMD-to-B70 transfer test](experiments/qwen38-27b-b70/notes/2026-09-14-amd-transfer-results.md)
adds 512/2K/16K reading measurements for the recommended 27B FP8 setup.
The exact projection-dispatch candidate was neutral. The newer-runtime/V2/
DFlash2 startup froze the host before inference; the user rebooted it and the
original qualified service was restored. No new speedup is claimed.
The [MTP-only source follow-up](community/1337hero-r9700-qwen38-radiance/validation/2026-09-14-mtp-fp8-transfer-review.md)
excludes DFlash and ranks the remaining exactness-preserving ideas.
The [implementation follow-up](experiments/qwen38-27b-b70/notes/2026-09-14-mtp-lossless-transfer-results.md) built metadata and exact
communication candidates. The communication attempt failed exactness and
recorded GPU faults; no speedup or new default was qualified.
[`CURRENT.md`](CURRENT.md) alone owns the recorded host and active research
state. These are evidence-backed examples; the broader expected-performance
view is the [performance index](results/scoreboard.md).
Additional **512-token prefill measurements** now cover 4B/9B on two B70s and
27B INT4 on one B70. [Results and optimization findings](experiments/qwen38-27b-b70/notes/2026-09-14-prefill-followup-results.md)
keep reading speed separate from decode and HTTP first-token wait.
### Current public package headlines
This table is generated from the package manifests. It is a synchronization
view, not a ranking: workloads, context lengths, card counts, quantization,
and speculative-decoding modes differ. Open the detail page for the exact
measurement scope and evidence.
The optional R308 single-request boundary repair is available for
[Qwen3.5 4B](repro/qwen35-4b-w4a16-b70/README.md) and
[9B](repro/qwen35-9b-w4a16-b70/README.md). Both models passed fresh-server
boundary and strict identity gates; the [qualification note](experiments/qwen35-4b-b70/notes/2026-09-13-r308-qualified-single-request.md)
defines its one-card, fixed-depth scope.
<!-- BEGIN GENERATED PUBLIC PACKAGE HEADLINES -->
| Model and deployment | Package status | Measured headline | Exact guide |
| --- | --- | ---: | --- |
| **[Qwen3.5 4B W4A16 with its own MTP head on one or two Intel Arc Pro B70](models/qwen35-4b-w4a16-b70.html)**<br>1× B70 · W4A16 (compressed-tensors INT4 weights, FP16 activations) · vLLM XPU | `candidate` · clean-host replay pending | **`191.727574 tok/s`**<br>Best qualified single-user record: MTP3, 67,248-row draft shortlist; fresh servers 191.87 / 191.58 | [reproduction guide](repro/qwen35-4b-w4a16-b70/README.md) |
| **[LFM2.5 24B A2B Q8_0 on one Intel Arc Pro B70 (exact fast path)](models/lfm25-24b-a2b-q8-b70.html)**<br>1× B70 · Q8_0 · llama.cpp SYCL + lab patch | `candidate` · clean-host replay pending | **`137.61891 tok/s`**<br>strict class-balanced decode | [reproduction guide](repro/lfm25-24b-a2b-q8-b70/README.md) |
| **[LFM2.5 2.6B Q8_0 on one Intel Arc Pro B70](models/lfm25-26b-q8-b70.html)**<br>1× B70 · Q8_0 · llama.cpp SYCL | `candidate` · clean-host replay pending | **`132.137457 tok/s`**<br>strict class-balanced decode | [reproduction guide](repro/lfm25-26b-q8-b70/README.md) |
| **[Laguna S 2.1 INT4 with DFlash on four Intel Arc Pro B70 cards](models/laguna-s-2.1-int4-b70-125tps-20260731.html)**<br>4× B70 · INT4 / BF16 KV · vLLM XPU | `candidate` · clean-host replay pending | **`125.461973 tok/s`**<br>conventional decode median | [reproduction guide](repro/laguna-s-2.1-int4-b70-125tps-20260731/README.md) |
| **[Qwen3.5 9B W4A16 with its own MTP head on one or two Intel Arc Pro B70](models/qwen35-9b-w4a16-b70.html)**<br>1× B70 · W4A16 (compressed-tensors INT4 weights, FP16 activations) · vLLM XPU | `candidate` · clean-host replay pending | **`124.084255 tok/s`**<br>Best qualified single-user record: MTP3, 67,248-row draft shortlist; fresh servers 124.04 / 124.13 | [reproduction guide](repro/qwen35-9b-w4a16-b70/README.md) |
| **[Gemma 4 26B A4B UD-Q8_K_XL on one Intel Arc Pro B70](models/gemma4-26b-a4b-q8-b70-125tps-20260701.html)**<br>1× B70 · UD-Q8_K_XL + Q4_0 MTP · llama.cpp SYCL | `candidate` · clean-host replay pending | **`122.160357 tok/s`**<br>decode (MTP-assisted) | [reproduction guide](repro/gemma4-26b-a4b-q8-b70-125tps-20260701/README.md) |
| **[Qwen3.8 27B AutoRound INT4, fixed-K batch-invariant profile, on two Intel Arc Pro B70 cards](models/qwen38-27b-autoround-int4-b70.html)**<br>2× B70 · AutoRound INT4 W4A16 (group 128, symmetric) · vLLM XPU | `candidate` · clean-host replay pending | **`117.528083 tok/s`**<br>Best qualified TP2 record: MTP4 with the 67k draft shortlist (R299 / R294b) | [reproduction guide](repro/qwen38-27b-autoround-int4-b70/README.md) |
| **[Muse-Glimmer 30B Q8/WOQ with DFlash on four Intel Arc Pro B70 cards](models/muse-glimmer-30b-q8-woq-b70-100tps-20260813.html)**<br>4× B70 · UD-Q8_K_XL / BF16 draft · llama.cpp SYCL | `candidate` · clean-host replay pending | **`100.3685 tok/s`**<br>pooled canonical mean | [reproduction guide](repro/muse-glimmer-30b-q8-woq-b70-100tps-20260813/README.md) |
| **[Qwen3.5 9B FP8-dynamic with its own MTP head on one Intel Arc Pro B70](models/qwen35-9b-fp8-b70.html)**<br>1× B70 · FP8-dynamic (compressed-tensors, per-channel FP8 weights, dynamic activations) · vLLM XPU | `candidate` · clean-host replay pending | **`98.139 tok/s`**<br>MTP depth 3 with the draft-only INT4 lm_head, one card, strict completions suite (center of two fresh servers, 98.251 / 98.027) | [reproduction guide](repro/qwen35-9b-fp8-b70/README.md) |
| **[Qwen3.8 27B official FP8 on two Intel Arc Pro B70 cards](models/qwen38-27b-fp8-vllm-tp2-asrock-b70.html)**<br>2× B70 · FP8 · vLLM XPU | `candidate` · clean-host replay pending | **`90.309841 tok/s`**<br>Historical writing speed · MTP depth 5 · two cards | [reproduction guide](repro/qwen38-27b-fp8-vllm-tp2-asrock-b70/README.md) |
| **[MiniMax M2.7 AutoRound INT4 on four Intel Arc Pro B70 cards](models/minimax-m27-b70-89tps-20260520.html)**<br>4× B70 · AutoRound W4A16 INT4 · vLLM XPU + llm-scaler | `candidate` · clean-host replay pending | **`89.314195 tok/s`**<br>mean output throughput | [reproduction guide](repro/minimax-m27-b70-89tps-20260520/README.md) |
| **[Qwen3.8 27B Q4_K_M + MTP2 on two Intel Arc Pro B70 cards](models/qwen38-27b-q4km-q4mtp-mtp2-tp2-b70.html)**<br>2× B70 · Q4_K_M target + Q4_0 MTP draft / F16 KV · llama.cpp SYCL | `candidate` · clean-host replay pending | **`64.237301 tok/s`**<br>strict varied-prompt decode | [reproduction guide](repro/qwen38-27b-q4km-q4mtp-mtp2-tp2-b70/README.md) |
| **[Qwen3.8 27B official FP8 on one Intel Arc Pro B70](models/qwen38-27b-fp8-vllm-tp1-b70.html)**<br>1× B70 · FP8 · vLLM XPU | `candidate` · clean-host replay pending | **`54.036484 tok/s`**<br>Writing speed · MTP depth 5 · one card | [reproduction guide](repro/qwen38-27b-fp8-vllm-tp1-b70/README.md) |
| **[Qwen3.8 27B Q4_K_M on two Intel Arc Pro B70 cards](models/qwen38-27b-q4km-tp2-asrock-b70.html)**<br>2× B70 · Q4_K_M / F16 KV · llama.cpp SYCL | `candidate` · clean-host replay pending | **`49.717503 tok/s`**<br>conventional decode median | [reproduction guide](repro/qwen38-27b-q4km-tp2-asrock-b70/README.md) |
| **[Qwen3.8 Flash-Next FP8 with lossless MTP1 and the GDN verifier rows in the kernel extension's exact serial mode, on four Intel Arc Pro B70 cards](models/qwen38-flash-next-fp8-tp4-mtp1-exactgdn-b70-47tps-20260913.html)**<br>4× B70 · FP8 block-128 weights / BF16 KV · vLLM XPU | `candidate` · clean-host replay pending | **`46.85425 tok/s`**<br>class-balanced decode median | [reproduction guide](repro/qwen38-flash-next-fp8-tp4-mtp1-exactgdn-b70-47tps-20260913/README.md) |
| **[Qwen3.8 27B Q4_K_M + MTP2 on one Intel Arc Pro B70](models/qwen38-27b-q4km-mtp2-tp1-b70.html)**<br>1× B70 · Q4_K_M target + Q4_0 MTP draft · llama.cpp SYCL | `candidate` · clean-host replay pending | **`42.636988 tok/s`**<br>strict varied-prompt decode | [reproduction guide](repro/qwen38-27b-q4km-mtp2-tp1-b70/README.md) |
| **[Qwen3.8 Flash-Next FP8 with lossless MTP1, never-routed experts host-placed and both reference Triton kernels restored, on four Intel Arc Pro B70 cards](models/qwen38-flash-next-fp8-tp4-mtp1-qsafused-b70-38tps-20260907.html)**<br>4× B70 · FP8 block-128 weights / BF16 KV · vLLM XPU | `candidate` · clean-host replay pending | **`37.825654 tok/s`**<br>class-balanced decode median | [reproduction guide](repro/qwen38-flash-next-fp8-tp4-mtp1-qsafused-b70-38tps-20260907/README.md) |
| **[Qwen3.8 27B Q8_0 + MTP2 on one Intel Arc Pro B70](models/qwen38-27b-q8-q4mtp-mtp2-tp1-b70.html)**<br>1× B70 · Q8_0 target + Q4_0 MTP draft · llama.cpp SYCL | `candidate` · clean-host replay pending | **`37.062028 tok/s`**<br>strict varied-prompt decode | [reproduction guide](repro/qwen38-27b-q8-q4mtp-mtp2-tp1-b70/README.md) |
| **[Qwen3.8 Flash-Next FP8 with lossless MTP1, never-routed experts host-placed and the Triton hyper-connection glue on four Intel Arc Pro B70 cards](models/qwen38-flash-next-fp8-tp4-mtp1-hctriton-b70-37tps-20260907.html)**<br>4× B70 · FP8 block-128 weights / BF16 KV · vLLM XPU | `candidate` · clean-host replay pending | **`37.045844 tok/s`**<br>class-balanced decode median | [reproduction guide](repro/qwen38-flash-next-fp8-tp4-mtp1-hctriton-b70-37tps-20260907/README.md) |
| **[Qwen3.8 27B Q8_0 on two Intel Arc Pro B70 cards](models/qwen38-27b-q8-tp2-asrock-b70.html)**<br>2× B70 · Q8_0 / F16 KV · llama.cpp SYCL | `candidate` · clean-host replay pending | **`36.726447 tok/s`**<br>strict varied-prompt decode | [reproduction guide](repro/qwen38-27b-q8-tp2-asrock-b70/README.md) |
| **[Qwen3.8 Flash-Next FP8 with no speculation, never-routed experts host-placed, both reference Triton kernels restored and the W13 MoE tile at the base width, on four Intel Arc Pro B70 cards](models/qwen38-flash-next-fp8-tp4-mtp0-w13n64-b70-34tps-20260908.html)**<br>4× B70 · FP8 block-128 weights / BF16 KV · vLLM XPU | `candidate` · clean-host replay pending | **`34.495292 tok/s`**<br>class-balanced decode median | [reproduction guide](repro/qwen38-flash-next-fp8-tp4-mtp0-w13n64-b70-34tps-20260908/README.md) |
| **[Qwen3.8 Flash-Next FP8 with lossless MTP1 and never-routed experts host-placed on four Intel Arc Pro B70 cards](models/qwen38-flash-next-fp8-tp4-mtp1-placement-b70-32tps-20260906.html)**<br>4× B70 · FP8 block-128 weights / BF16 KV · vLLM XPU | `candidate` · clean-host replay pending | **`31.929484 tok/s`**<br>class-balanced decode median | [reproduction guide](repro/qwen38-flash-next-fp8-tp4-mtp1-placement-b70-32tps-20260906/README.md) |
| **[Qwen3.8 27B Q4_K_M on one Intel Arc Pro B70](models/qwen38-27b-q4km-tp1-b70.html)**<br>1× B70 · Q4_K_M · llama.cpp SYCL | `candidate` · clean-host replay pending | **`27.825726 tok/s`**<br>decode | [reproduction guide](repro/qwen38-27b-q4km-tp1-b70/README.md) |
| **[Qwen3.8 Flash-Next FP8 with lossless MTP1 on four Intel Arc Pro B70 cards](models/qwen38-flash-next-fp8-tp4-mtp1-lossless-b70-27tps-20260905.html)**<br>4× B70 · FP8 block-128 weights / BF16 KV · vLLM XPU | `candidate` · clean-host replay pending | **`27.048435 tok/s`**<br>class-balanced decode median | [reproduction guide](repro/qwen38-flash-next-fp8-tp4-mtp1-lossless-b70-27tps-20260905/README.md) |
| **[Qwen3.8 27B 256K + vision + MTP draft on one Intel Arc Pro B70](models/qwen38-27b-256k-vision-mtp-b70.html)**<br>1× B70 · UD-Q5_K_S (shipped) / UD-Q4_K_XL (alternative) · llama.cpp SYCL | `candidate` · clean-host replay pending | **`26.668277 tok/s`**<br>decode (MTP-assisted, full package resident) | [reproduction guide](repro/qwen38-27b-256k-vision-mtp-b70/README.md) |
| **[Qwen3.8 27B Q8_0 on one Intel Arc Pro B70](models/qwen38-27b-q8-tp1-b70.html)**<br>1× B70 · Q8_0 · llama.cpp SYCL | `candidate` · clean-host replay pending | **`19.61924 tok/s`**<br>strict varied-prompt decode | [reproduction guide](repro/qwen38-27b-q8-tp1-b70/README.md) |
| **[LTX 2.5 continuous video on four B70 cards](models/ltx25-continuation-stream-b70-145f-20261010.html)**<br>4× B70 · BF16 distilled weights; F32 output checks · ComfyUI / PyTorch XPU | `candidate` · clean-host replay pending | **strict headline pending**<br>Strict headline pending: scoped continuation timing only; seam/audio acceptance, sustained performance and independent replay remain open. | [reproduction guide](repro/ltx25-continuation-stream-b70-145f-20261010/README.md) |
| **[MiniMax-H3 video with audio on two B70 cards](models/minimax-h3-pruned-bf16-tp2-b70-20261004.html)**<br>2× B70 · Comfy-Org pruned BF16 denoiser + INT8 ConvRot encoder · Native diffusers / PyTorch XPU | `candidate` · clean-host replay pending | **strict headline pending**<br>Owner-approved measured batch; strict headline certification pending: baseline raw receipts and independent full-suite repeat are absent. | [reproduction guide](repro/minimax-h3-pruned-bf16-tp2-b70-20261004/README.md) |
| **[Nemotron 3.5 Lightning 30B-A3B UD-Q4_K_M on one Intel Arc Pro B70](models/nemotron-35-lightning-30b-a3b-b70.html)**<br>1× B70 · UD-Q4_K_M · llama.cpp SYCL | `candidate` · clean-host replay pending | **strict headline pending**<br>Historical August22 raw suites, canaries and server logs are now imported and hash-bound. The 72.169452/72.035976 tok/s text-delta observations remain non-headline: native-token-ID quality/determinism replay is incomplete. October19 replay was interrupted and owner-deferred; no qualified new result. | [reproduction guide](repro/nemotron-35-lightning-30b-a3b-b70/README.md) |
| **[Ornith 1.5 35B-A3B Q4_K_M on one Intel Arc Pro B70](models/ornith-15-35b-a3b-q4km-b70.html)**<br>1× B70 · Q4_K_M · llama.cpp SYCL | `candidate` · clean-host replay pending | **strict headline pending**<br>The 131.460231 tok/s two-server observation and matched patch A/B evidence remain valid scoped measurements, but the runtime produced 0/12 identical complete natural-response hashes across fresh stock servers. Keep the mechanism and context evidence; require a stable, registered cross-server oracle before assigning a strict package headline. | [reproduction guide](repro/ornith-15-35b-a3b-q4km-b70/README.md) |
| **[Ornith 1.5 9B Q8_0 on one Intel Arc Pro B70](models/ornith-15-9b-q8-b70.html)**<br>1× B70 · Q8_0 · llama.cpp SYCL | `candidate` · clean-host replay pending | **strict headline withheld**<br>Strict headline withheld after output-gate failure. Two complete fresh-server, cache-zero, 12-prompt/six-class attempts measured 49.593582 and 49.515869 tok/s and passed both objective-canary batteries, but complete token arrays matched only 8/12 across servers. | [reproduction guide](repro/ornith-15-9b-q8-b70/README.md) |
| **[Qwen3.8 Flash-Next UD-IQ3_XXS — own runtime, one GPU](models/qwen38-flash-next-iq3xxs-ownrt-tp1-b70.html)**<br>1× B70 · UD-IQ3_XXS · Own Xe runtime (packet58aj / packet165c) | `candidate` · clean-host replay pending | **strict headline pending**<br>Historical 128-token record: 37.595605 tok/s; separate full-512 timings below. Oracle parity uses BF16 recurrent state. Reference-FP32 fidelity and no-added-loss qualification remain unestablished. | [reproduction guide](repro/qwen38-flash-next-iq3xxs-ownrt-tp1-b70/README.md) |
| **[Qwen3.8 Flash-Next UD-IQ3_XXS — own runtime, two GPUs](models/qwen38-flash-next-iq3xxs-ownrt-tp2-b70.html)**<br>2× B70 · UD-IQ3_XXS · Own Xe runtime (bundle 8 / packet165) | `candidate` · clean-host replay pending | **strict headline pending**<br>Historical 128-token record: 63.611824 tok/s; separate full-512 timings below. Oracle parity uses BF16 recurrent state. Reference-FP32 fidelity and no-added-loss qualification remain unestablished. | [reproduction guide](repro/qwen38-flash-next-iq3xxs-ownrt-tp2-b70/README.md) |
<!-- END GENERATED PUBLIC PACKAGE HEADLINES -->
### Research and historical entry points
These are entry points, not the whole repo:
| Lane | Status | Best Current Pointer |
| --- | --- | --- |
| **Muse-Glimmer-30B UD-Q8_K_XL on 4x B70** | Closed/banked no-training Q8/WOQ record: fixed-N16 oneDNN WOQ plus pretrained BF16 DFlash and distributed ARGMAX. Two canonical full-256 means **`100.088`** and **`100.649 tok/s`**; frozen 15-prompt cold conventional first-100 median **`161.900 tok/s`**, p10 `108.574`, all cache-zero. Target-verified, not BF16/lossless or universally token-exact; LocalMaxxing `cmss8515c00n0ms01n3begqgg` | [result packet](results/muse-glimmer-30b-q8-woq-b70/README.md), [standalone repro](repro/muse-glimmer-30b-q8-woq-b70-100tps-20260813/README.md), [source snapshots](patches/muse-glimmer-30b-b70/README.md) |
| **Poolside Laguna S 2.1 INT4 on 4x B70** | Exact target-verified DFlash depth 11 on the audited width-12 Breakable PIECEWISE graph, with segmented inline draft attention, a decode-only 128-GRF INT4 kernel, and the shared-elementwise M12 fusion: **`125.461973 tok/s`** conventional and **`126.729266 tok/s`** historical compatibility. The final-source cold suite passed 13/13 exact with all `cached_tokens=0`; LocalMaxxing `cms9wuuf300cqpm01t5i285tq` | [qualified result](results/laguna-s-2.1-int4-b70/README.md), [record evidence](experiments/laguna-s-2.1-xpu-b70/notes/2026-07-31-shared-elementwise-m12-record.md), [source snapshots](patches/laguna-s-2.1-xpu-b70/README.md), [package guide](repro/laguna-s-2.1-int4-b70-125tps-20260731/README.md) |
| DeepSeek V4 Flash experimental uniform-K160 on 4x B70 | Paused/closed frontier; target-verified DSpark7 record **`80.820 tok/s`** high and `78.287 tok/s` three-suite median-of-medians; exact source bundles and fail-closed launcher preserved; LocalMaxxing `cmrquta9905w3lg013m5vxoqx` | [result packet](results/deepseek-v4-flash-k160-b70/README.md), [standalone repro](repro/deepseek-v4-flash-k160-b70-80tps-20260718/README.md), [closeout](experiments/deepseek-v4-flash-reap-xpu-b70/notes/2026-07-21-deepseek-v4-flash-frontier-closeout.md) |
| Qwen3.6 27B INT4 AutoRound on 1-2x B70 | **Lane closed 2026-08-18, superseded by the Qwen3.8 INT4 lane above.** July TP2 row **`95.385 tok/s`** under its original metric/bar; LocalMaxxing `cmrh35ct50092mj01h7jgydqj`. The final fixed-RMSNorm screen matched then-sealed controls at `106.663`, but the matched 25-prompt candidate was only 12/25 exact at `93.446`; no new submission. | [final closeout](notes/2026-08-17-qwen36-int4-batch-invariant-rmsnorm-closeout.md), [independent validation](experiments/qwen36-27b-autoround-int4-b70/validation-20260815/README.md), [historical repro](repro/qwen36-27b-autoround-int4-b70/README.md) |
| Qwen3.6 27B GGUF Q8_0 target-only on 2x ASRock B70 | Retained no-speculation TP2 clean-source record: **`36.604128 tok/s` conventional** (`36.973866` historical helper), full-512 after-TTFT `36.533899`; 12/12 512-token outputs exact and all cache counts zero; two-chain DP4A ILP plus recurrent/attention/collective fusions; `+17.981%` over the matched mndodd fork baseline | [handoff](results/qwen36-27b-q8-tp2-asrock-b70/HANDOFF.md), [result packet](results/qwen36-27b-q8-tp2-asrock-b70/README.md), [standalone repro](repro/qwen36-27b-q8-tp2-asrock-b70/README.md), [source patch](patches/qwen36-27b-q8-tp2-asrock-b70/README.md), [mndodd contributor packet](community/mndodd-qwen36-27b-llamacpp-sycl/README.md) |
| **Qwen3.8 27B GGUF on 2x ASRock B70** | Q4_K_M + Q4_0 MTP2 has a replicated strict **`64.237301 tok/s`** one-user headline (two fresh servers, 24/24 complete candidate arrays exact to MTP0). Q4_K_M target-only remains `49.717503 tok/s` for one user and reaches **`192.341954 tok/s aggregate at c96`** with the exact F16 `ffn_down+ffn_gate` cache: two fresh candidates, 96/96 same-shape batch-oracle identity each, cache zero, effective 49,152-token pool, and near-capacity ~30.5 GiB/card use. The separate sequential comparison was 50/96, so this is not a batch-invariant-text claim. Q8_0 target-only is `36.726447 tok/s`. Speculative, target-only, aggregate, and single-stream rates remain distinct. | [Q4 MTP2 repro](repro/qwen38-27b-q4km-q4mtp-mtp2-tp2-b70/README.md), [Q4 target-only repro](repro/qwen38-27b-q4km-tp2-asrock-b70/README.md), [c96 result](experiments/qwen38-27b-b70/notes/2026-08-30-qwen38-q4km-tp2-exact-f16-cache-c96-result.md), [Q8 primary repro](repro/qwen38-27b-q8-tp2-asrock-b70/README.md) |
| **Qwen3.8 27B official FP8/W8A16 on 2x B70** | **Strict R187: MTP1 `54.935 tok/s`, MTP depth-2 `70.142 tok/s`, MTP depth-3 `79.183 tok/s`, MTP depth-4 `82.396 tok/s`, MTP depth-5 `86.182 tok/s`, MTP0 `33.097 tok/s`** (2026-09-03/04 clean boots; every pair 12/12; repeat-exact at 224-300-token prompts; MTP0 output-identical through c64 at 927.9 tok/s aggregate, MTP1 through c16 at 477.7, depth 3 through c16 at 557.0, depth 4 through c16 at 529.4, depth 5 through c16 at 493.2, depth 2 through c4 at 210.8). R187 = R156 served with one whole-graph torch.compile (`splitting_ops=[]`; on the piecewise compile MTP depth 2 emitted a phantom first token on one request in 64, an unfixed upstream defect that also occurs on the unmodified image; no whole-graph pass has shown it). Prebuilt image on GHCR (`ghcr.io/steveseguin/vllm-openai-xpu-qwen38-fp8`, digest `173660ec…`). LocalMaxxing approved R187 MTP1 `cmtmfa23g01tvl601eyqliy9y`, depth-2 `cmtmfa2i701tyl601yrkxjn46`, depth-3 `cmtmfa2t101u1l601gth9lyx6`, depth-4 `cmtmk61qo01x8l601vtqbitg5`, depth-5 `cmtn23c500005mm010ierd4g3`. R156 = the row-invariant R139 W8A16 image (release `qwen38-fp8-tp2-r139-20260902`, LocalMaxxing `cmtkvle7a0428p701k0ttabyy`) plus a Python mixed-step GDN split. Independent-host replay passes all gates; its slower CPU/fabric lowers c1 decode 1.8x graph-off, 94% recovered graph-on. | [fresh-cache audit](experiments/qwen38-27b-b70/notes/2026-09-01-qwen38-fp8-public-reproduction-audit.md), [R54 result](experiments/qwen38-27b-b70/data/2026-09-01-qwen38-fp8-explicit-deterministic-matrix-r54-result.json), [package](packages/qwen38-27b-fp8-tp2-b70/README.md), [reproduction guide](repro/qwen38-27b-fp8-vllm-tp2-asrock-b70/README.md) |
| **Qwen3.8 27B official FP8 on 1x B70** | MTP depth 5 **`53.52 tok/s`** (fresh servers 53.58/53.45) with a 16,384-token context; no-quantization profile `51.61 tok/s` at 12,544 tokens; no MTP `19.40`. Prompt reading 1,987-2,043 tok/s at 2K-12K inputs. Outputs identical to no MTP on the strict suite, long-prompt continuations and a chat quality suite; repeat requests are bitwise identical. Replaces the August one-card eager screen (10.75 tok/s, 8,448 tokens). | [package](packages/qwen38-27b-fp8-tp1-b70/README.md), [recipe](repro/qwen38-27b-fp8-vllm-tp1-b70/README.md) |
| **Qwen3.8 27B AutoRound INT4 on 1–2x B70** | Best qualified shortlist record **117.528 tok/s** on two cards (R299/R294b, pair 117.46/117.59). Current R304 / vLLM v0.29.0 revalidation: TP2 **117.24/116.95**, TP1 **81.17/81.16**, G1/G2/G3 12/12. Multi-user limits are independently tested; see the two-pass matrix. Older output-changing MTP5 rows remain withdrawn. | [recipe](repro/qwen38-27b-autoround-int4-b70/README.md), [package](packages/qwen38-27b-int4-fixed-k-tp2-b70/README.md) |
| **[Flash-Next UD-IQ3_XXS, quantized compressed version · own Xe runtime · 2× B70](models/qwen38-flash-next-iq3xxs-ownrt-tp2-b70.html)** | Current full-512 lab replay **58.557–58.632 tok/s**; 512-input native reading **75.884–75.973 tok/s**. Full-output independent quality, context sweep and concurrency remain pending. Historical packet44 retained separately. | [Recipe](repro/qwen38-flash-next-iq3xxs-ownrt-tp2-b70/README.md), [test details](results/qwen38-flash-next-ud-iq3xxs-b70/README.md) |
| **Qwen3.8 Flash-Next 125B-A6B official FP8 on 4x B70** | Current qualified lossless MTP1 record **46.854250 tok/s**, A367, approved September 13; exact serial GDN, unchanged verifier and output pins. Exact-32K **44.05 tok/s**, four-row median across A382/A394. The no-MTP profile separately measures **34.50 tok/s**, **32.67** at 32K. Multi-user result withheld after output mismatch; clean-host deployment remains pending. | [family](models/qwen-flash-next.html), [record and evidence](results/qwen38-flash-next-fp8-b70/README.md), [recipe](repro/qwen38-flash-next-fp8-tp4-mtp1-exactgdn-b70-47tps-20260913/README.md) |
| Gemma 4 26B A4B Q8 / INT8 on 1x B70 | Production-servable backend plus current strict fresh-response speed frontier; noisy near-record support band | [handoff](results/gemma4-26b-a4b-q8-b70/HANDOFF.md), [production service](results/gemma4-26b-a4b-q8-b70/production-service.md), [125 tok/s repro](repro/gemma4-26b-a4b-q8-b70-125tps-20260701/README.md) |
| Gemma 4 26B long-context/prompt-processing service lane | Separate from short-decode record; approved LocalMaxxing service entry `cmr47ivql0045nv011pfdjlaa`; service gates must not regress short decode | [Gemma result packet](results/gemma4-26b-a4b-q8-b70/README.md), [service gate script](repro/gemma4-26b-a4b-q8-b70/run-vdr2-long-context-service-gate.sh) |
| Rapid one-B70 model snapshots | Historical strict/fresh decode baselines across practical GGUF/vLLM candidates; retained for evidence and regression work rather than current model recommendations | [rapid result ledger](results/rapid-model-snapshots-b70/README.md), [rapid experiment notes](experiments/rapid-model-snapshots-b70/README.md) |
| Qwen3.6 35B A3B Quark W8A8 INT8 on B70 | Closed reference packet for now; preserve lessons for future return | [Qwen result packet](results/qwen36-35b-quark-int8-b70/README.md), [research map](docs/qwen36-research-map.md) |
| MiniMax M2.7 INT4 AutoRound on 4x B70 | Deployable baseline plus older strict-speed and source-fusion research leads | [result packet](results/minimax-m27-int4-autoround-b70/README.md), [Ubuntu 24 deploy repro](repro/minimax-m27-b70-110tps-ubuntu24-20260523/README.md), [production service notes](docs/minimax-production-c1-service.md) |
| Gemma 4 12B IT INT4 AutoRound | Historical deployment and multimodal research profiles; host state is recorded in [CURRENT.md](CURRENT.md) | [experiment packet](experiments/gemma4-12b-int4-autoround-vllm/README.md), [slot switching](docs/model-slot-switching.md) |
For the full queue and archive, use [docs/model-effort-index.md](docs/model-effort-index.md).
### Qwen3.8 27B Model Board
Last audited **2026-08-30**. Qwen3.8 retains Qwen3.6's exact 64-layer,
three-GDN-to-one-full-attention tensor geometry, so the accepted exact-shape
TP2 stack transfers mechanically. Every new weight set is still independently
gated. Every row states whether it is target-only or speculative; promoted
measurements are cache-zero. Other B70 hosts
and agents should start with the [multi-host handoff](experiments/qwen38-27b-b70/MULTI-HOST-HANDOFF.md)
and [do-not-repeat index](experiments/qwen38-27b-b70/DO-NOT-REPEAT.md).
| Target and route | Hardware | Best captured decode result | Evidence boundary / pointer |
| --- | --- | ---: | --- |
| **GGUF Q4_K_M + external Q4_0 draft, MTP2** | 2x ASRock B70, TP2 target; draft on first card | **`64.237301 tok/s` strict paired headline** | Median of two fresh candidate servers (`64.180644`/`64.293959`) versus matched fresh MTP0 `49.787366` (+29.02%); full 12-prompt/six-class cache-zero suites, objective canaries passed, and 24/24 complete candidate arrays target-exact. Short-context one-user result only; no target-only 32K or concurrency value transfers. [standalone repro](repro/qwen38-27b-q4km-q4mtp-mtp2-tp2-b70/README.md), [promotion attestation](experiments/qwen38-27b-b70/data/2026-08-30-qwen38-q4km-q4mtp-tp2-mtp2-promotion-attestation.json) |
| GGUF Q4_K_M, lab TP2 stack + device-local dense-FFN fusion | 2x ASRock B70, TP2 | **one user `50.219700` (`49.717503` conventional); c96 aggregate `192.341954`** | No speculation; the one-user suite remains 12/12 complete-output exact. The separate two-family exact-cache c96 profile used two fresh candidate servers, prompt cache off, and 96/96 same-shape batch-oracle identity in each. It used an effective 49,152-token pool and about 30.5 GiB/device; isolated sequential identity remained 50/96 and one-user MMVQ decode is unchanged. [standalone repro](repro/qwen38-27b-q4km-tp2-asrock-b70/README.md), [c96 result](experiments/qwen38-27b-b70/notes/2026-08-30-qwen38-q4km-tp2-exact-f16-cache-c96-result.md), [source increment](patches/qwen38-27b-q4km-tp2-asrock-b70/README.md) |
| GGUF Q8_0, accepted packaged TP2 stack | 2x ASRock B70, TP2 | **`36.726447` strict paired headline** (`36.772932` historical raw-completions capture) | No speculation; two fresh full 12-prompt/six-class, 512-cap attempts passed cache-zero and objective-canary gates with 12/12 complete token arrays exact. The current outputs also match 12/12 historical response hashes. SG16 added `+0.257%`, SG24 added `+0.356%`, and DP4A2×SG24 beat one-chain SG24 in both endpoint pairs; [strict result](experiments/qwen38-27b-b70/notes/2026-08-27-qwen38-q8-tp2-strict-reasoningoff-native-r2-result.md), [standalone repro](repro/qwen38-27b-q8-tp2-asrock-b70/README.md), [base summary](experiments/qwen38-27b-b70/data/2026-08-15-q8-tp2-transfer-summary.json), [DP4A2×SG24 result](experiments/qwen38-27b-b70/data/2026-08-17-q8-dp4a2-sg24-accepted.json) |
| GGUF Q8_0, accepted stack, two simultaneous requests | 2x ASRock B70, TP2, c2 | **`57.398122` aggregate (`28.70` per request)** | Narrow fixed-prompt capacity result, not single-stream or a general quality guarantee; broader prompt pairs confirmed schedule-dependent outputs; no speculation; [c2 repro](repro/qwen38-27b-q8-tp2-c2-asrock-b70/README.md), [capture](experiments/qwen38-27b-b70/data/2026-08-16-q8-tp2-c2-summary.json), [broader audit](experiments/qwen38-27b-b70/data/2026-08-16-q8-c2-batch-shape-audit.json) |
| **Official block-scaled FP8 + block-W8A16** | 2x ASRock B70, TP2 | **strict MTP1 `51.808087 tok/s`; target-only `1,112.570323 tok/s` aggregate c128** | The R53/R54 matrix used two fresh MTP1 candidates and two same-image MTP0 controls. Every complete natural-512 workload/canary gate passed and all four target/candidate comparisons were 12/12 exact. MTP0 centers at `33.733520 tok/s`; dynamic MTP stays withheld. The historical exact-depth profile separately measured `46.636241 tok/s` at 32K; [audit](experiments/qwen38-27b-b70/notes/2026-09-01-qwen38-fp8-public-reproduction-audit.md), [R54 result](experiments/qwen38-27b-b70/data/2026-09-01-qwen38-fp8-explicit-deterministic-matrix-r54-result.json), [32K result](experiments/qwen38-27b-b70/notes/2026-08-28-qwen38-fp8-w8a16-mtp1-exact-depth-r33-result.md), [package](packages/qwen38-27b-fp8-tp2-b70/README.md), [standalone repro](repro/qwen38-27b-fp8-vllm-tp2-asrock-b70/README.md) |
| Official block-scaled FP8, older Intel vLLM `0.21.0-b3.1` | 2x ASRock B70, TP2 | Not promoted | Superseded negative: artifact loaded, but bounded initialization hit device-lost/out-of-resource errors; [bring-up note](experiments/qwen38-27b-b70/notes/2026-08-15-bringup-checkpoint.md) |
| **AutoRound INT4 W4A16, MTP4, XPU graphs + 67k draft shortlist** | 2x B70, TP2 | **117.528 tok/s** pair center (117.46/117.59, R299/R294b) | G1/G2/G3 12/12; approved shortlist record. Current R304 replay 117.24/116.95; one-card 81.17/81.16. Output-identical concurrency is scoped separately in the [recipe](repro/qwen38-27b-autoround-int4-b70/README.md). |
| AutoRound INT4 W4A16, vLLM/XPU MTP4 speculative | 2x B70, TP2 | historical **`100.497 tok/s` all-25; `96.627` selection-12** | Preserved as a measured speed observation of the exact margin-on configuration, not a promotion-grade exactness result. The output-changing margin and margin-on quality baseline prevent clean comparison with margin-free rows; upstream disposition remains a human decision. LocalMaxxing [`cmszarna10e0nms0103hv0tve`](https://www.localmaxxing.com/en/runs/cmszarna10e0nms0103hv0tve); [audit and position](experiments/qwen38-27b-b70/notes/2026-08-23-lmx-position-and-candidates.md) |
| **AutoRound INT4 W4A16, vLLM XPU nightly, target-only, XPU graph on** | 1x B70, TP1 | **`30.22 / 30.26 tok/s` conventional (boot pair)** | Fastest single-card lane for this model (llama.cpp Q4_K_M TP1 is `27.82`). No speculation, cache-zero gated, objective battery pass (code canary `14`, repeats, 8K needle) on the exact config. The captures used diagnostic `ignore_eos=true`; no real baseline comparison was run. Caveat: deterministic within a boot but NOT across boots (autotuned kernels; 19/25 boot-pair output agreement), so no cross-boot token-exactness claim and no sealed record/LMX submission yet; [finding](experiments/qwen38-27b-b70/notes/2026-08-22-qwen38-tp1-vllm-nightly-bringup-finding.md), [data](experiments/qwen38-27b-b70/data/2026-08-22-qwen38-tp1-vllm-nightly-matrix.json) |
| **AutoRound INT4 W4A16, vLLM XPU nightly, target-only, XPU graph on** | 4x B70, TP4 | **`71.67 / 71.55 tok/s` conventional (boot pair)** | Fastest target-only Qwen3.8 result measured in this lab for this AutoRound/nightly identity; the objective canaries, 8-run repeat, cache-zero checks, and 8K needle passed, with 0.17% speed spread. The captures used diagnostic `ignore_eos=true`; no real baseline comparison was run. The two boots matched 21/25 complete outputs, and vLLM explicitly labels multi-GPU XPU Graph unsupported/experimental. Eager mode measured `17.4`; TP3 is impossible (16 GDN K heads % 3); TP4 needs `gpu-memory-utilization <= 0.6`; [TP-scale finding](experiments/qwen38-27b-b70/notes/2026-08-23-qwen38-tpscale-nightly-finding.md), [data](experiments/qwen38-27b-b70/data/2026-08-23-qwen38-tpscale-nightly-matrix.json) |
Community-reported alternatives are kept outside the promoted rows above:
| Target and route | Hardware | Best captured result | Evidence boundary / pointer |
| --- | --- | ---: | --- |
| GGUF Q4_K_M, oneAPI 2025.3-family JIT Docker using the lab TP2 patches | 1x/2x B70 on contributor host | Contributor reports **33.4 TP1 / 51.1 TP2 tok/s** target-only | `community-reported`; no raw benchmark data, fixed-suite evidence, cache telemetry, output hashes, or reference-lab execution. The container disables the lab's Q4K fusion under JIT after reported corruption. No files were vendored because the source has no explicit license; provenance and the useful packaging delta are retained in our [internal status and review](community/0xsero-qwen38-27b-q4km-docker/STATUS.md). |
| GPTQ INT4 G128, vLLM XPU target-only / native MTP4 | 1x ASRock B70 local; 1x Intel B70 contributor | **34.160467 / 87.605425 tok/s local** with native FP16 KV | Experimental performance only: native FP16 KV beat FP8 at 8K, MTP4 accepted 511/540 drafts and matched its target, and loaded MTP parameters were verified FP16. The GPTQ target failed a deterministic code-result canary (`30` versus correct Q8/Q4 result `14`), so this is not the no-quality-loss default or a promoted headline; [decision](community/sergiiob-qwen38-27b-vllm-xpu/validation/2026-08-16-quality-kv-dtype-decision.md), [community packet](community/sergiiob-qwen38-27b-vllm-xpu/STATUS.md) |
| GPTQ INT4 G128, vLLM **0.27.1** XPU Docker, fp8 KV, MTP ladder | 1x Arc Pro B70 32 GB (Reddit poster, single-card = TP1) | Reddit report **33.2 off / 47.1 MTP1 / 52.2 MTP2 tok/s**, 128K context | `community-reported` field report; poster's numbers, raw logs not captured, not reference-lab run. fp8 KV, prefix-caching, PLAIN eager+inductor, `--max-num-seqs 1` (MTP + concurrent requests crash the engine on this hybrid model). Newer 0.27.1 image than our pinned lane; our archived GPTQ A/B measured higher (`68.23` MTP2) but the GPTQ target fails our code-result quality canary, so GPTQ stays quality-rejected as a default; [field report](community/field-reports/reddit-arc-b70-vllm-52tps/README.md) |
#### TP1 single-card context + KV-cache sweep (Q4_K_M, llama.cpp)
Measured 2026-08-22 on one Arc Pro B70 with the accepted llama.cpp SYCL TP1
lane build, Q4_K_M target-only, flash-attn on, `llama-bench` pp2048 + tg128,
5 reps per depth. These are **raw-engine** rates for the decode/prefill *shape*
and the KV-dtype delta - not the promoted `27.83 tok/s` class-balanced
realistic-suite metric (depth-0 tg128 here is `24.81`; different harness, both
real). Raw data and chart: [sweep JSON](experiments/qwen38-27b-b70/data/2026-08-22-q4km-tp1-context-kv-sweep.json),
[chart SVG](experiments/qwen38-27b-b70/data/2026-08-22-q4km-tp1-context-kv-sweep.svg),
[finding](experiments/qwen38-27b-b70/notes/2026-08-22-qwen38-q4km-tp1-context-kv-sweep-finding.md).
| context depth | decode KV **f16** | decode KV **q8_0** | q8 vs f16 | prefill KV f16 | prefill KV q8_0 |
| ---: | ---: | ---: | ---: | ---: | ---: |
| 0 | 24.81 | 24.27 | -2.2% | 825 | 818 |
| 4096 | 24.25 | 21.05 | -13.2% | 893 | 887 |
| 8192 | 23.83 | 18.68 | -21.6% | 851 | 843 |
| 16384 | 23.10 | 14.86 | -35.7% | 780 | 772 |
| 32768 | 21.77 | 10.66 | **-51.0%** | 668 | 663 |
**KV8-vs-KV16 finding:** the q8_0-KV decode penalty *grows with context* -
~2% at 0 ctx to **-51% at 32K** - because the per-token KV dequant scales with
cached length. f16-KV decode stays nearly flat (`24.81 -> 21.77`, -12% to 32K).
Prefill is KV-dtype-independent (<1.5% at every depth). **Practical rule:**
keep KV at **f16 for speed** on this lane; use q8_0 KV only to fit longer
context in 32 GiB, accepting a large long-context decode hit. (The Reddit vLLM
report ran fp8 KV by default on the newer 0.27.1 XPU path. Our own vLLM TP1 is
now unblocked on the XPU nightly image - see the matrix below; there,
fp8_e4m3 KV measured decode-neutral at short context but output-divergent,
and fp8_e5m2 is refused outright by FlashAttention on this device.)
#### TP1 weight-quant ladder (Q4_K_M vs UD-Q4_K_XL vs UD-Q5_K_S, f16 KV)
Same lane, same config, f16 KV, three weight files. Decode tg128 (tok/s):
| context | **Q4_K_M** (17.67 GiB) | UD-Q5_K_S (17.38 GiB) | UD-Q4_K_XL (16.35 GiB) |
| ---: | ---: | ---: | ---: |
| 0 | **24.81** | 22.72 | 21.81 |
| 8192 | **23.83** | 21.79 | 21.06 |
| 16384 | **23.10** | 21.19 | 20.49 |
| 32768 | **21.77** | 20.09 | 19.45 |
**Counter-intuitive result:** decode speed is *inverse* to file size here -
the smallest file (UD-Q4_K_XL, 16.35 GiB) is the **slowest** decoder, and the
largest (plain Q4_K_M) is the **fastest** at every depth. Decode is therefore
not purely bandwidth-bound on this build: it is Q4_K-tuned (Q4_K reorder +
MMVQ), so uniform Q4_K_M hits the fast path while the unsloth UD mixed-precision
dynamic quants do not. This ordering is a property of the tuned lane build, not
a universal ranking of the formats. **Takeaway:** on this lane, prefer Q4_K_M
for speed (it is also the quality-validated promoted lane); pick a UD quant only
for its quality/size, accepting slower decode here.
[data](experiments/qwen38-27b-b70/data/2026-08-22-qwen38-tp1-weight-ladder-sweep.json),
[chart](experiments/qwen38-27b-b70/data/2026-08-22-qwen38-tp1-weight-ladder-sweep.svg),
[finding](experiments/qwen38-27b-b70/notes/2026-08-22-qwen38-q4km-tp1-context-kv-sweep-finding.md).
#### TP1 vLLM on the XPU nightly image (AutoRound INT4): unblocked; graph +25%; MTP verify-bound
Measured 2026-08-22/23 on one B70 (GPU0), same 25-prompt realistic suite and
conventional metric as the promoted lanes, cache-zero gated, prefix caching
off, `--max-num-seqs 1`. Image: `vllm-openai-xpu` nightly `e9d1398d9`
(dated 11 calendar days after v0.27.1 but not its descendant, torch 2.13+xpu).
The pinned `0.20.2rc1` TP1 crashes do not
reproduce here; container bring-up needs `CCL_ZE_IPC_EXCHANGE=sockets` + a
read-only `/dev/dri/by-path` mount. The full quality battery (code canary
`14`, repeats, 8K needle) passed on both certified configs: MTP-off f16
graph-off AND graph-on. Those battery runs did not supply `--baseline-json`,
so their `baseline_match_all=true` compatibility field is vacuous and is not
used as evidence of oracle parity.
| Config (f16 KV unless noted) | conventional decode tok/s | acceptance | outputs vs MTP-off oracle |
| --- | ---: | ---: | --- |
| MTP off (two boots) | **23.72 / 24.25** | - | oracle pair (20/25 cross-boot agreement) |
| MTP off, **XPU graph on** (two boots) | **30.22 / 30.26** | - | faithful (23/25, within boot envelope) |
| MTP off, fp8_e4m3 KV | 24.10 | - | **divergent (3/25)** - capacity lever only |
| MTP1 / MTP2 / MTP3 | 4.51 / 4.41 / 4.30 | 1.91 / 2.70 / 3.47 | faithful (23-24/25) |
| MTP1 + XPU graph | 7.63 | **0.00** | **corrupt (0/25) - do not use** |
| any MTP, fp8_e5m2 KV | fails to boot | - | `NotImplementedError` |
**TP scaling on the same container** (2026-08-23, same suite/metric,
characterization-only): eager multi-GPU decode is FLAT (~17 tok/s at both
TP2 and TP4, below single-card - per-step container collective tax), while
**XPU graph restores scaling: 30.2 / 48.8 / 71.7 tok/s at TP1 / TP2 / TP4**.
The 71.7 TP4 row is the fastest target-only Qwen3.8 result for this
AutoRound/nightly identity, not the lab-wide target-only record. All three
graph configs passed the objective battery, but multi-GPU XPU Graph is
explicitly unsupported/experimental in this vLLM build. The only TP4 MTP2
probe is infrastructure-invalid: concurrent ranks lost shared Triton-cache
artifacts, and the later `shm_broadcast` starvation was a downstream symptom.
It does not establish a TP>1 speculative-decode deadlock. Prefill
scales even in eager (281 -> ~500 -> ~860 tok/s); TTFT drops to 0.09 s at
TP4. TP3 is architecturally impossible (16 GDN K heads % 3). TP4 needs
`gpu-memory-utilization <= 0.6` (XPU single-allocation cap).
[TP-scale data](experiments/qwen38-27b-b70/data/2026-08-23-qwen38-tpscale-nightly-matrix.json),
[TP-scale finding](experiments/qwen38-27b-b70/notes/2026-08-23-qwen38-tpscale-nightly-finding.md).
Key findings: **XPU graph** (`VLLM_XPU_ENABLE_XPU_GRAPH=1`, default off on the
nightly) is worth **+25%** MTP-off and is output-faithful. **MTP at TP1
works and is output-faithful with excellent acceptance, but the verify step
costs ~190-200 ms per extra verify token** (~4-5x a whole MTP-off step), so
net decode collapses ~5x and deeper drafts cannot amortize it - opposite to
the community GPTQ result, pointing at the INC/AutoRound W4A16 small-batch
verify path or GDN serial verify. **Graph + MTP corrupts outputs** (0%
acceptance and 0/25 oracle match) - quarantined. The lane is deterministic
within a boot but not across boots (autotuned kernel selection), so it makes
no cross-boot token-exactness claim yet.
[data](experiments/qwen38-27b-b70/data/2026-08-22-qwen38-tp1-vllm-nightly-matrix.json),
[finding](experiments/qwen38-27b-b70/notes/2026-08-22-qwen38-tp1-vllm-nightly-bringup-finding.md),
[driver](experiments/qwen38-27b-b70/scripts/run-20260822-qwen38-tp1-nightly-docker-bench.sh).
Plain-language explainers for these results live in the Learn library:
[Quantization vs decode speed](https://neural.download/learn/quantization-and-speed.html),
[KV cache precision](https://neural.download/learn/kv-cache-precision.html),
[Context length](https://neural.download/learn/context-length.html),
[The MTP ladder](https://neural.download/learn/mtp-ladder.html).
### Qwen3.6 27B Model Board
Last audited **2026-08-15**. These rows share a model family, not a quality,
runtime, or benchmark class. “Target only” means no speculative draft; MTP and
DFlash rows retain the declared target as verifier. The first number is the
repository's historical published 100-event/99-interval rate where that
harness was used; the parenthesized value is conventional 99-interval
accounting from the same timestamps. Relative A/B gains are unchanged.
| Target and route | Hardware | Best captured decode result | Evidence boundary / pointer |
| --- | --- | ---: | --- |
| AutoRound INT4 W4A16, vLLM MTP3 | 2x B70, TP2 | historical **95.384868** (`94.431019` conventional) | **Historical campaign closeout, 2026-08-18**; later Qwen3.8 INT4 qualifications and limits live in its [maintained recipe](repro/qwen38-27b-autoround-int4-b70/README.md). The July row is retained and unbeaten on its own 12-prompt suite. Two durable findings from the closeout: complete-token parity against a differently-configured reference is unsatisfiable at fp16, and XPU batch invariance is dead code behind `is_cuda_alike()` gates; [determinism/speed closeout](notes/2026-08-18-qwen36-int4-determinism-speed-tradeoff.md), [source packet](patches/qwen36-27b-autoround-int4-b70/determinism-closeout-20260818/README.md), [prior closeout](notes/2026-08-17-qwen36-int4-batch-invariant-rmsnorm-closeout.md) |
| AutoRound INT4 W4A16, vLLM ReplaySSM MTP3 | 1x B70 | **68.236263** (`67.553901` conventional) | Valid quality-gated historical high; July 11 isolated reconfirmation was `65.4-66.7 tok/s`, so do not treat the high as every-run expectation; [TP1 attribution packet](results/qwen36-27b-autoround-int4-b70/tp1-draftgraph-attribution-reconfirm-20260711.json) |
| GGUF Q4_0, DFlash5 | 1x B70 | **47.818818** (`47.340630` conventional) | Strict fixed-suite speculative record; unchanged Q4_0 target verifies accepted tokens; [closure](notes/2026-07-13-qwen27-dflash-sycl-closure.md) |
| GGUF Q4_K_M, intrinsic MTP2 | 1x B70 | **38.112 tok/s** | One fixed greedy 128-token request, visible bytes matched its 25.307 tok/s target-only control; not token-exact and not a fixed-suite median; [community validation](community/dominick253-qwen36-27b-llamacpp-sycl/validation/2026-08-08-reference-lab-validation.md) |
| GGUF UD-Q4_K_XL, intrinsic MTP7 with `p_min=0.65` | 1x B70 | **31.480049** (`31.165249` conventional) | Best valid p-min support row; cache-zero fixed suite; [Q4 packet](results/qwen36-27b-mtp-gguf-q4-b70/README.md) |
| GGUF Q4_0, target only | 1x B70 | **25.937011** (`25.677641` conventional) | Strict cold-suite target-only control from the closed DFlash campaign; [timeline evidence](experiments/qwen27-dflash-sycl-b70/notes/2026-07-12-cycle-timeline-tooling.md) |
| GGUF Q8_0, mndodd base plus lab exact fusions, target only | 2x ASRock B70, TP2 | **36.973866** (**36.604128** conventional) | Current clean-source record; 12/12 cold 512-token output hashes exact and cache zero; full-512 after-TTFT `36.533899`; `+17.981%` over matched fork baseline; [result packet](results/qwen36-27b-q8-tp2-asrock-b70/README.md) |
| GGUF Q8_0, mndodd fork baseline, target only | 2x ASRock B70, TP2 | **31.338765** (**31.025377** conventional) | Matched contributor-fork baseline; 12/12 complete hashes match the upstream-derived control, cache zero; [community packet](community/mndodd-qwen36-27b-llamacpp-sycl/README.md) |
| GGUF Q8_0, mndodd fork, target only | 1x ASRock B70 | **17.955800** (**17.776242** conventional) | Matched raw-completions fixed suite; quality-cleared, cache zero; [community validation](community/mndodd-qwen36-27b-llamacpp-sycl/validation/2026-08-12-asrock-b70-validation.md) |
| GGUF Q8_0, upstream-derived VDR2, target only | 1x ASRock B70 | **17.297038** (`17.124067` conventional) | Matched one-card control for the fork; cache zero; [community validation](community/mndodd-qwen36-27b-llamacpp-sycl/validation/2026-08-12-asrock-b70-validation.md) |
| GGUF Q8_0, mndodd fork, intrinsic MTP4 | 1x ASRock B70 | **39.618445** (`39.222260` conventional) | Support row pending the packet's explicit-greedy rerun; F16 target/draft KV; [community packet](community/mndodd-qwen36-27b-llamacpp-sycl/README.md) |
| GGUF Q8_0, mndodd fork, DFlash5 | 1x ASRock B70 | **38.084045** (`37.703205` conventional) | Support row, not target-only; the `65.00 tok/s` observation was one favorable prompt, not the fixed-suite median; [community packet](community/mndodd-qwen36-27b-llamacpp-sycl/README.md) |
| Native FP8 Safetensors, vLLM | 2x B70, TP2 | **30.171 tok/s** median decode | Different 15-row prompt-length benchmark, so not rank-comparable to the fixed suite; [community validation](community/dominick253-qwen36-27b-fp8-tp2-docker/STATUS.md) |
The official BF16 target was not downloaded in the current campaign. At
55.586 GB of weights, two 608 GB/s B70s have a target-only one-weight-read
roofline of about `21.9 tok/s` before collective and runtime overhead, so a
plain lossless TP2 goal above 30 tok/s is not physically credible. The Q8
target-only TP2 lane remains active toward the 40 tok/s stretch goal;
speculative rows stay separate.
## Validity Rules For Speed Claims
Diagnostic runs are allowed and useful, but headline records require the
model-lane gate. For Gemma/Qwen-style fresh-response records, that means:
- fixed realistic prompt suite;
- each prompt run once as a cold response;
- `cached_tokens=0` for every request;
- no prompt/KV cache reuse, context checkpoints, response reuse, warmed
repeated prompts, or n-gram/history acceleration;
- target model and quantization unchanged;
- speculative decoding/MTP allowed only when accepted tokens are verified by
the declared target model;
- primary metric is the median conventional rate across the 99 inter-token
intervals between timestamps 1 and 100 after TTFT, with p10, mean, TTFT,
wall-clock full-output throughput, full-output after-TTFT throughput, hashes,
runtime identity, env vars, flags, and logs. Historical 100-event/99-interval
compatibility fields must be labeled as such.
Synthetic, repeated, warmed, cached, or history-assisted rows stay diagnostic
unless revalidated by the lane's promotion gate. This matters because several
very fast historical rows were useful optimization clues but not real-world
fresh-response claims.
## Hardware Scope
The measuring host has four Intel Arc Pro B70 32 GB cards and about 125 GiB
system RAM. A second host has **two ASRock Arc Pro B70 32 GB cards** (`64 GB`
aggregate VRAM), full-size ReBAR, PCIe 5.0 x16 links, and only about 15 GiB
system RAM; it is currently restricted to source/build/op-level work. Hardware,
count, and host identity remain part of every result. B70 is the platform on
which maintainers can independently reproduce and verify submitted patches,
but this is an Intel Arc/XPU project, not a B70-only repository.
Results, fixes, and portability reports from Intel Arc Pro B50, B60, B65, and
B70 owners are welcome, as are useful observations from other Intel Arc and
XPU systems. A B70 rerun verifies what a patch does on B70; it does not certify
a contributor's score on hardware the maintainers do not possess. Hardware and
verification status therefore stay explicit in every promoted result.
The four-card host enables TP4 or independent one-/two-GPU screens. The
two-card host has enough VRAM for TP2 but not enough system RAM for the current
Qwen3.8 AutoRound server, so GPU count alone is not a safety signal.
### Arc Pro B65 orientation
Intel specifies B65 and B70 with the same 32-GiB VRAM capacity and `608 GB/s`
memory bandwidth, but B65 has 20 versus 32 Xe cores and 160 versus 256 XMX
engines. A community report relayed from boyter claims roughly `42 tok/s` with
MTP, `22 tok/s` without MTP, and `19 tok/s` at a 150-W limit on one B65. The
command, prompt, metric, and exact power/MTP identity were not captured, so
these are purchasing-orientation anecdotes rather than verified comparisons.
Also note that this lab's `49.717503 tok/s` Qwen3.8 Q4_K_M result uses **two**
B70s, not one. See the [B65 field report](community/field-reports/boyter/arc-pro-b65-qwen38/README.md).
Steve Seguin maintains this repo and posts ongoing build notes at
<https://x.com/xyster>.
<img height="600" alt="quad" src="https://github.com/user-attachments/assets/e6ce5633-17ff-4a73-924d-31dcbc913ede" />
<img width="1666" height="478" alt="Gemma 4 26B B70 result context" src="https://github.com/user-attachments/assets/14602c6b-5c72-483d-aec7-415fbc4a8114" />
## How To Contribute
This remains primarily a working optimization lab, but careful outside
contributions are welcome. Start with [CONTRIBUTING.md](CONTRIBUTING.md) and
the [manual verification policy](docs/contribution-verification.md). No result
needs to be a record to be useful.
Useful contributions include:
- reproducing a result on another Intel/XPU stack and sharing exact versions;
- testing a failed lane after a driver, PyTorch XPU, vLLM, llama.cpp, or oneAPI
update;
- turning local patches into clean upstream issues or PRs;
- adding quality canaries for new model families;
- sharing high-signal failure logs with model, quantization, graph mode, and
hardware identity intact;
- providing temporary access to larger Intel hardware for models that do not
fit cleanly on 32 GB cards.
When opening an issue or discussion, include GPU, OS, model, quantization,
runtime, exact command, benchmark shape, quality gate, result JSON/log paths,
and what changed from the closest known-good run.
## Reusable Optimization Lessons
The durable value of this lab is not only its fastest rows. Start with the
[cross-model pattern catalog](docs/research-workflow-playbook.md#cross-model-patterns-worth-reusing)
for evidence-linked strategies that transfer across model lanes, and the
[model optimization guide](docs/model-optimization-guide.md) for the
identity, quality, A/B, variance, and negative-result process used to test them.
Model-specific packets retain the commands and caveats behind each lesson.
## Deep Historical Notes Below
The former top-level chronology was preserved verbatim in
[notes/2026-07-11-readme-historical-b70-archive.md](notes/2026-07-11-readme-historical-b70-archive.md).
It remains useful for chronology and link recovery, but maintained navigation
now lives in the result packets, model-effort index, reproduction recipes, and
optimization guides.
## Historical B70 Findings And Open Leads
Use the [historical archive](notes/2026-07-11-readme-historical-b70-archive.md)
for the original long-form findings. Current model-specific starting points are
the [results index](results/README.md), including the
[MiniMax M2.7 packet](results/minimax-m27-int4-autoround-b70/README.md), and the
[cross-model effort index](docs/model-effort-index.md). Underlying notes, data,
patches, experiments, and reproduction paths were intentionally left in place.
## Layout
Use the [docs index](docs/README.md) for maintained navigation and the
[repository organization](#how-the-repo-is-organized) above for placement
rules. The former file-by-file inventory is retained in the
[historical archive](notes/2026-07-11-readme-historical-b70-archive.md); it is
not duplicated here because detailed inventories become stale quickly.