dlinfer

Pull and analyze inference timelines on Ascend with lmdeploy+dlinfer. Covers profiling script templates, delay selection, raw timeline handling, fixed-real-batch decode/KV-context audit, and analysis reporting.

Start, monitor, and stop lmdeploy REST API services for any supported device/backend. Use when Codex needs to run `lmdeploy serve api_server` for a model path, choose model names from the model being served, set backend/device/max-batch/cache and user-provided parallel strategy flags, run outside the sandbox when devices are unavailable inside it, preserve server logs, check readiness, handle port conflicts, and cleanly stop the service before another run.

Run RESTful pressure benchmarks with lmdeploy's profile_restful_api.py after starting an lmdeploy API service. Use when Codex needs to choose ShareGPT native-length or random-length workloads, set tokenizer/model/client parameters without hard-coding qwen, compare models or max-batch settings, preserve logs, inspect running batch and KV cache behavior, and report throughput and cache/OOM handling.

The operator call chain in lmdeploy+dlinfer — the five layers every op crosses from model.forward() down to torch_npu/aclnn kernels, and how to trace or add a new op. Uses fused MoE (Qwen3.5 MoE) on Ascend as the worked example.

Understand the complete graph mode flow in lmdeploy+dlinfer, covering the runner architecture, buffer management, vendor differences, and common pitfalls.

Debug precision regressions in lmdeploy+dlinfer on domestic AI hardware (Ascend / CAMB / MACA) by comparing against a reference implementation.

Add support for a new model (already in lmdeploy's CUDA backend) on domestic AI hardware (Ascend / CAMB / MACA) via dlinfer.