ascend-memory-profiling

name	ascend-memory-profiling
description	Profile and attribute HBM memory usage on Ascend NPU for vLLM serving scenarios. Breaks down memory into fixed overhead, model weights, KV cache, HCCL buffers, activations, and runtime, with traceable evidence chains. Use for requests like "分析显存占用", "显存 profiling", "HBM 用了多少", "内存各部分拆分". Do not use for performance profiling (kernel timing, throughput), offline inference, or non-Ascend hardware.

Ascend Memory Profiling

Collect and analyze HBM memory usage on Ascend NPU devices running vLLM serving workloads. Produces a structured breakdown of memory by component, with every value traceable to its data source.

Remote substrate rule: use .remote-dev remote tools for ad hoc remote read/edit/bash/search/patch work around memory profiling setup and output inspection. Use this skill for the domain HBM workflow and keep its scripts as the compatibility backend for managed VAWS sessions.

Use this skill when

• the user asks to profile or analyze GPU/NPU memory (显存) usage
• the user wants to understand what consumes HBM in a vLLM serving scenario
• the user asks "权重/KV cache/HCCL/激活各占多少显存"
• the user wants to verify memory allocation against theoretical expectations
• the user asks to compare memory usage across different configurations

Do not use this skill when

• the task is performance profiling (kernel timing, bubble analysis, step/layer/operator breakdown, cross-rank diagnosis) → use ascend-profiling-analysis (consumes an ascend-profiling-collection manifest or a remote profile root)
• the task is starting/stopping a service without memory analysis → use vllm-ascend-serving
• the task involves non-Ascend hardware
• the task is offline (non-serving) inference only

Data source priority

Priority	Source	Role	Trustworthiness
P0	msprof --application wrapping	Full component breakdown (APP, HCCL, RUNTIME, SLOG)	Highest -- sees memory torch cannot manage
P1	npu-smi info	Static baseline + phased delta	High -- hardware-level
P2	vLLM startup logs	Weights, KV cache, num_gpu_blocks	Medium-high -- application-reported
P3	safetensors file headers	Tensor shapes, dtypes, byte sizes (byte-accurate); component classification and shard strategy are rule-based inference	High for byte sizes; medium for per-device attribution
P4	Model config.json	Theoretical weight calculation (fallback)	Reference only

Memory components

Component	Source	How derived
Fixed overhead (driver)	npu-smi Phase 0	HBM_Used with no user process
Model weights	safetensors + vLLM logs	weight_inspector.py 解析文件头 → 按 TP/EP/DP 分片 → 与 vLLM DeviceMemoryProfiler 交叉验证
KV cache	vLLM logs	"Available KV cache memory: X GiB"
ACL Graph 编译缓冲	vLLM logs	"Graph capturing finished in X secs, took Y GiB"
HCCL buffers	msprof	npu_module_mem.csv Component=HCCL (per-device when PROF→device mapping available, otherwise process-level)
CANN Runtime	msprof	npu_module_mem.csv Component=RUNTIME (same scoping as HCCL)
Activations	npu-smi delta	HBM during inference minus HBM at idle
未归因残差	Residual	所有已知组件加总后的余量 (有 msprof 时通常 < 200 MB)

Residual handling

All memory attribution is based on measured data — no estimation or guessing is performed.

• With msprof: HCCL, RUNTIME, SLOG are precise msprof measurements. Any remaining residual after all components is small (typically < 200 MB) and reported as "未归因残差".
• Without msprof: The residual is reported as "未归因 (缺少 msprof 数据)" with an explicit note that msprof collection is needed for a complete breakdown. No attempt is made to split the residual into sub-components.

Workflow

Always use the vllm-ascend-serving skill for service lifecycle management. This profiling skill only collects and analyzes data — it attaches to a running service. msprof wrapping is mandatory for a complete, traceable memory breakdown.

For parallel agent work, create or reuse a session-management session first, then use --session-id <id> or --session-file <session.json> everywhere below. In session mode, this skill reads serving state from .vaws-local/sessions/<session-id>/serving.json and talks only to that session's dedicated container.

Session-scoped memory profiling must use --attach. Standalone mode starts its own service and is kept only for legacy single-tenant machine flows; it is blocked when --session-id or --session-file is used so it cannot bypass session service-port leases or shared state isolation.

Step 0: Check msprof availability

Before starting the service, verify that msprof is available on the remote machine:

python3 -c "
import sys; sys.path.insert(0, '.agents/skills/ascend-memory-profiling/scripts')
from _common import check_msprof_available, resolve_execution_target
target = resolve_execution_target('<alias-or-none>', session_id='<session-id-or-none>')
ep = target['endpoint']
print(check_msprof_available(ep))
"

If this fails, stop and fix the remote environment before proceeding.

Step 1: Prepare the service with msprof (via vllm-ascend-serving)

Upload the msprof wrapper, then start the service with --wrap-script:

# Upload msprof wrapper to remote
python3 -c "
import sys; sys.path.insert(0, '.agents/skills/ascend-memory-profiling/scripts')
from _common import resolve_execution_target, upload_msprof_wrapper
target = resolve_execution_target('<alias-or-none>', session_id='<session-id-or-none>')
ep = target['endpoint']
print(upload_msprof_wrapper(ep, mem_freq=50))
"
# Start service with msprof wrapping
python3 .agents/skills/vllm-ascend-serving/scripts/serve_start.py \
  (--machine <alias> | --session-id <id>) --model <path> --tp <N> \
  --wrap-script /tmp/_vaws_msprof_wrap.sh \
  [-- --speculative-config '...' --compilation-config '...' ...]

Note: upload_msprof_wrapper internally calls check_msprof_available as a safety net and writes a unique wrapper path under /tmp for each call.

For baseline npu-smi data, collect npu-smi info before starting the service.

Step 2: Collect memory data (attach mode)

python3 .agents/skills/ascend-memory-profiling/scripts/mem_collect.py \
  --session-id <id> \
  --attach \
  [--baseline-from <previous-run-dir>] \
  [--tag <experiment-name>]

Use --machine <alias-or-ip> for legacy single-tenant workflows, or --session-file <session.json> when the session file path is the stable handle.

What happens:

• Reads .vaws-local/serving/<alias>.json or .vaws-local/sessions/<session-id>/serving.json to discover port, PID, model path, tp/dp, extra args
• Auto-extracts --speculative-config, --compilation-config, etc. from the serving state
• Collects npu-smi snapshot, vLLM logs (from serving's runtime dir), weight manifest
• Sends inference request to collect activation delta
• Does NOT start or stop the service

Step 3: Stop service (via vllm-ascend-serving)

python3 .agents/skills/vllm-ascend-serving/scripts/serve_stop.py \
  --session-id <id>

Use the same target form that was used for serve_start.py.

Step 4: Collect msprof data

After stop, run mem_collect --attach again with --resume-run pointing to the run directory from Step 2. This merges the msprof CSV data into the same run, producing a single manifest with both live npu-smi data and post-stop msprof CSVs:

python3 .agents/skills/ascend-memory-profiling/scripts/mem_collect.py \
  --session-id <id> --attach \
  --resume-run .vaws-local/memory-profiling/<run-dir-from-step-2>/

This works because attach mode accepts stopped services — it skips health check and inference, and focuses on collecting msprof CSVs that are now available. The --resume-run flag ensures all data lands in one directory.

Step 5: Analyze and generate report

python3 .agents/skills/ascend-memory-profiling/scripts/mem_analyze.py \
  .vaws-local/memory-profiling/<run-dir>/ \
  [--format json|text]

Default --format json outputs machine-readable JSON to stdout (matching the repo-wide wrapper contract). Use --format text for a human-readable report on stdout. Both modes write report.json and report.txt files to the run directory.

Baseline strategy

Situation	How to get baseline
Fresh profiling	Collect npu-smi info before serve_start, save to file, use --baseline-from <file>
Repeat profiling on same machine	Reuse baseline from a previous run via --baseline-from <old-run-dir>
Quick analysis (no baseline needed)	Omit --baseline-from — report shows "固定开销" as 0 with note

--baseline-from accepts either a previous run directory (containing baseline_npu_smi.txt or manifest.json) or a raw npu-smi info output text file.

Fallback: Standalone mode

When the serving skill is unavailable (e.g. bootstrap scenario), mem_collect.py can manage the service internally. This is a fallback — prefer the serving skill workflow above.

python3 .agents/skills/ascend-memory-profiling/scripts/mem_collect.py \
  --session-id <id> \
  --model <remote-weight-path> \
  --tp <N> [--dp <N>] [--tag <name>] \
  [--speculative-config '...'] [--compilation-config '...'] ...

This runs all phases internally: baseline → start (with msprof) → health check → snapshot → inference → stop → msprof export. msprof is always enabled in standalone mode; a pre-flight check verifies msprof availability before starting.

Standalone: Analyze and generate report

python3 .agents/skills/ascend-memory-profiling/scripts/mem_analyze.py \
  .vaws-local/memory-profiling/<run-dir>/

Outputs:

• report.txt -- human-readable report with evidence chains
• report.json -- machine-readable structured report

Example output (S2: MoE 35B, TP=4, DP=2, MTP=3, FULL_DECODE_ONLY, with msprof)

[Device 0] 总 HBM: 32.00 GiB | 已用: 27.42 GiB

组件                           |    占用 (MB) |   占用 (GiB) |     占比 | 主数据源
--------------------------------------------------------------------------------------------------------------
固定开销 (driver/runtime)        |     2930.0 |      2.861 |  10.4% | npu-smi Phase 0 (baseline)
模型权重                         |     9692.6 |      9.465 |  34.5% | safetensors 文件头精确计算
KV Cache 预留                  |    14080.0 |     13.750 |  50.2% | vLLM 日志
ACL Graph 编译缓冲               |      501.8 |      0.490 |   1.8% | vLLM 日志
HCCL 缓冲                      |      597.7 |      0.584 |   2.1% | msprof npu_module_mem
CANN Runtime                   |      125.3 |      0.122 |   0.4% | msprof npu_module_mem
激活峰值                         |       72.0 |      0.070 |   0.3% | npu-smi delta
  └ 未归因残差                    |       74.7 |      0.073 |   0.3% | 残差

[交叉验证]
  npu-smi 已用:          28,074 MB
  组件加总:              27,999.3 MB
  未归因:                74.7 MB (0.3%)
  msprof APP:            27,500 MB

Interaction with other skills

Skill	Interaction
vllm-ascend-serving	Service lifecycle: Use serve_start.py to start (with --wrap-script for msprof), serve_stop.py to stop. This skill reads serving state from .vaws-local/serving/<alias>.json or .vaws-local/sessions/<session-id>/serving.json in session mode. The serving skill is agnostic to msprof — it only knows about the wrapper script.
session-management	Parallel isolation: Use session_create.py before parallel remote work and pass --session-id to serving and memory collection. Session mode stores serving state under .vaws-local/sessions/<session-id>/serving.json.
machine-management	SSH endpoint resolution: Both skills share the machine inventory via inventory from .agents/lib/.
remote-code-parity	Automatic via serving: The serving skill calls parity sync before service start.

Critical rules

• Service lifecycle belongs to vllm-ascend-serving — this skill only collects and analyzes data.
• In attach mode, mem_collect will never start or stop the service itself. Service stop is done by the agent calling serve_stop.py directly. After stop, mem_collect --attach can be run again to export and collect msprof data.
• In standalone mode (fallback), the service is started and stopped within the profiling run.
• msprof export (8 卡) 可能需要几分钟。
• Keep collected data under .vaws-local/memory-profiling/ only (untracked).

Weight analysis methodology

The skill uses weight_inspector.py to parse safetensors file headers on the remote machine, extracting tensor names, shapes, dtypes, and byte sizes. Individual tensor byte sizes are byte-accurate from the file headers. Component classification (e.g. "Attention Q", "MoE Expert") and shard strategy assignment (col/row/expert parallel) are rule-based inferences from tensor name patterns, not direct measurements.

How it works

1. Parse safetensors headers: Each .safetensors file starts with a JSON header containing tensor metadata (name, shape, dtype, offsets). No weight data is read — only the header (typically a few KB per file).

2. Classify tensors: Each tensor is categorized by name pattern:

   • embed_tokens → Embedding
   • self_attn.{q,k,v,o}_proj → Full Attention
   • linear_attn.{in_proj_qkv,in_proj_z,out_proj,conv1d,...} → Linear/Mamba Attention
   • mlp.experts.* → MoE Experts
   • mlp.shared_expert.* → MoE Shared Expert
   • visual.* → Vision Encoder
   • mtp.* → MTP (Multi-Token Prediction)
   • norm, layernorm → Layer Norms

3. Determine shard strategy: Each tensor is assigned a parallelism strategy:

   • col_parallel / row_parallel → divided by TP
   • expert_parallel → divided by EP (= TP × DP when EP enabled)
   • replicated → full copy on each device (norms, gates, small params)

4. Calculate per-device weight: Sum up per-device bytes across all tensors.

5. Cross-validate with vLLM: Compare safetensors-derived per-device value with vLLM's DeviceMemoryProfiler measurement. Expected differences:

   • MTP embedding/lm_head sharing: MTP reuses base model's embeddings (reduces actual memory)
   • F32 → BF16 conversion: Some small parameters (A_log, dt_bias) stored as F32 but may load as BF16
   • Vision encoder in text-only mode: Still loaded for multimodal architectures

When the script isn't sufficient

If the safetensors-based analysis shows unexpected results, the agent should:

1. Check weight_manifest.json for unclassified ("other") tensors
2. Inspect vLLM's model implementation (load_weights) for weight sharing or skipping logic
3. Compare model.named_parameters() output on the remote machine with the safetensors manifest
4. Check if enable_ep_weight_filter affects loading behavior

Limitations

• Activation measurement relies on npu-smi delta between idle and inference states. This captures peak but not fine-grained activation lifetime.
• torch_npu.profiler via vLLM's /start_profile//stop_profile endpoints currently does not produce device-side data (device_0/data is empty). Use msprof wrapping instead.
• msprof wrapping profiles the main process only. TP worker processes are separate -- each gets its own PROF directory with per-device data.
• msprof export for 8-card runs may take several minutes. The timeout is set to 1800s.

References

• references/methodology.md -- Detailed methodology and data source descriptions
• references/msprof_fields.md -- msprof CSV field reference