| name | qwen38-hga-optimization |
| description | Diagnose, benchmark, and optimize the repository's HGA Qwen3.8-27B inference path on 16 GB NVIDIA GPUs, especially VRAM residency, graph reuse, speculative decoding, and split-FFN streaming. Use for performance regressions or deployment tuning in Inference/Qwen3_8_27B_VRAM16GB; do not use for unrelated models or generic llama.cpp tuning. |
Qwen3.8 HGA optimization
Optimize the live Qwen3.8-27B HGA deployment using matched measurements, while preserving model correctness, VRAM limits, and the user's existing worktree changes.
Required context
Before diagnosing or changing this inference path, read references/optimization-details.md. It records the model-specific memory layout, benchmark method, fixed bugs, measured alternatives, current recommendation, and relevant source locations.
Work from Inference/Qwen3_8_27B_VRAM16GB. Treat the recorded numbers as a 2026-08-29 calibration for an RTX 5060 Ti-class 16 GB host, not as universal performance guarantees.
Workflow
- Inspect the current diff and running process before editing. Existing modifications may belong to the user.
- Record a matched baseline with
deployment/deploy.py and tools/bench_8k_api.py. Use the same fixed --nonce, prompt size, generation length, model, and server restart policy for every A/B candidate.
- Verify the effective process environment and startup log. Do not infer the running configuration merely from
api.env.
- Distinguish intentional host-resident exchange weights from an unintended CPU fallback. Use the startup tensor census and pin check described in the reference.
- Compare graph BUILD/HIT counts and total build time before blaming graph reconstruction. CPU graph metadata is not evidence that compute was offloaded.
- When changing speculative depth, compare accepted draft tokens, target-batch count, output correctness, and decode throughput together.
- When changing split FFN, validate tile lifetime, quantization alignment, host pinning, stream/event ownership, numerical output, and the whole-layer fallback before judging speed.
- Rebuild, redeploy, wait for
/health, run the matched benchmark, then run the focused tests. Leave the fastest verified safe configuration deployed unless the user requests otherwise.
Benchmark invariants
- Prefer
--nonce <fixed-value> for deterministic uncached A/B prompts. --stable-prompt can exercise the lazy prefix cache and invalidate a comparison.
- Restart the model between candidate configurations so weights, prefix state, and graph caches begin comparably.
- Do not compare runs with materially different speculative acceptance or target-batch counts as if they measured the same compute schedule.
- Separate prefill and decode. Split FFN is a DECODE/VERIFY optimization and should not be expected to improve PREFILL.
- Preserve raw JSON results and the corresponding server log. Report exact tokens/s and wall times, not only percentages.
- A speed change within ordinary run-to-run noise needs repeated trials before becoming a default.
Current recommendation
The best configuration measured through 2026-08-30 is whole-layer exchange with HGA_SPEC=3, HGA_SPLIT_FFN=0, HGA_F16_TRANSPORT=1, GGML_CUDA_CUBLAS_COMPUTE_TYPE=auto, calibrated routing and packing widths, and CUDA graphs disabled. On the measured host calibration selected HGA_THREADS=24 and HGA_PACK_THREADS=4. Routing stays on its cache-friendly OpenMP team; append/quantize/packing runs on a separate persistent pool pinned one worker per physical core. The calibrated 8K/64-token run measured 277.52 prefill tok/s and 12.42 decode tok/s, effectively unchanged from the earlier 12-pack-worker run because packing is mostly hidden by the larger route/GPU critical path.
Do not implement the split as alternating 24- and 12-worker OpenMP regions. That made routing substantially slower. OMP_PLACES=cores also hurt routing, and a 24-worker region with only 12 active workers retained the 24-worker barrier cost. Read the 2026-08-30 threading section in the reference before touching CPU parallelism.
Keep split FFN opt-in. Keep FP16 transport limited to the large PREFILL HGA boundary: decode transfers are tiny, and making the full ggml graph FP16 violates CUDA kernel type contracts. Read the FP16 section in the reference before extending this path.
Keep HGA_GPU_KV_I8=0 by default. The experimental CUDA Q8_0 K/V wire is functional and reduces D2H bytes, but its end-to-end gain was smaller than normal run variance and it adds a second quantization plus graph memory. Use it only for controlled A/B work.
Completion criteria
An optimization task is complete only when:
- the live process uses the intended configuration;
/health succeeds;- no expected-GPU tensor unexpectedly resides on CUDA host or CPU memory;
- graph-build evidence is interpreted quantitatively;
- the matched benchmark improves the requested metric without correctness failures;
- focused unit tests, benchmark self-test, syntax checks, and
git diff --check pass; and
- the final report states what is deployed, measured tradeoffs, and any remaining bottleneck.
