ascend-llm-cost-optimization

name	ascend-llm-cost-optimization
description	Benchmark and reduce LLM inference token cost on Huawei Ascend 910 NPUs with vLLM-Ascend and a Mooncake DRAM KV tier. Use when working on Qwen3.6-35B-A3B serving, 64K-context agent workloads, prefix-cache tiering, data-parallel session affinity, concurrency tuning, or when a throughput/cost benchmark gives results that look wrong.

Ascend LLM inference cost optimization

Field-tested playbook from taking Qwen3.6-35B-A3B on 16× Ascend 910 dies from USD 24.77 → 13.96 per million output tokens (829 output tok/s, zero errors) on a 64K-context agent workload with ~90% repeated prompt text.

Read REPORT.md for the full write-up. This file is the operating checklist.

Start here when a benchmark result looks wrong

Three measurement defects caused most of the apparent plateau in this project. Check them before believing any throughput or cost number.

1. Is the load generator actually driving the concurrency it claims? aiohttp.ClientSession(timeout=...) without an explicit connector caps at 100 connections (TCPConnector(limit=100)). Fingerprint: peak running tracks nominal concurrency at C32/C64, then freezes at ~95-99 for C128 and C256 alike. Fix with TCPConnector(limit=0, limit_per_host=0). Gate: peak running must scale linearly with nominal concurrency.

2. Did the KV pool evict during the measured window? Compare master_evicted_size_bytes before and after. Any increase means the warm tier was destroyed mid-run — discard the result. Reset the pool before every run; the churn phase writes hundreds of GB.

3. Is a stale server sharing the port? vLLM binds with SO_REUSEPORT, so a second instance on the same port ; two listeners split traffic by 4-tuple hash and a dead-engine leftover returns 500s. Assert is 0 before launch and 1 after. Smoke-test with , never — a zombie can still 200.