菜单
Performance analysis coordination workflow. Guides profiling delegation, bottleneck classification (compute/memory/launch/communication/sync), and structured report generation. Use when the user asks to analyze performance, profile a workload, check MFU/SOL, or diagnose bottlenecks.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | perf-analysis |
| description | Performance analysis coordination workflow. Guides profiling delegation, bottleneck classification (compute/memory/launch/communication/sync), and structured report generation. Use when the user asks to analyze performance, profile a workload, check MFU/SOL, or diagnose bottlenecks. |
| tags | ["analysis","profiling","bottleneck"] |
| license | Apache-2.0 |
| metadata | {"author":"NVIDIA Corporation"} |
When diagnosing performance issues, classify the primary bottleneck:
| Type | Indicator | Description |
|---|---|---|
| Compute-bound | High GPU utilization, low memory bandwidth usage | Limited by compute capacity (FLOPs) |
| Memory-bound | High memory bandwidth, low compute utilization | Limited by DRAM throughput |
| Launch-overhead | Many small kernels, high CPU time | CPU becoming bottleneck from kernel launch overhead |
| Communication-bound | Significant time in collective operations | Limited by inter-GPU or inter-node communication |
| Sync-bound | Excessive CPU-GPU synchronization points | Stalls from unnecessary synchronization |
When delegating to specialists, describe the desired outcome -- not the tool methodology.
DO include:
DO NOT include:
--set=full, --section SpeedOfLight)Specialists have their own skills that encode best practices for tool usage
and their own workspace artifacts for output. Prescribing commands in the
delegation overrides their skills and may lead to suboptimal profiling
strategies (e.g., collecting 8000+ metrics with --set=full when a targeted
section analysis would be faster and more surgical).
Profile the batched GEMM kernel in bmm_workload.py with NCU.
The workload uses cudaProfilerStart/Stop markers to isolate the region of interest.
Collect kernel-level metrics: SOL%, compute/memory throughput, DRAM bandwidth,
tensor core utilization, occupancy, warp stall reasons, and roofline classification.
The batched GEMM performs 68.72 GFLOP per call (B=32, M=512, N=1024, K=2048, FP16).
Calculate MFU against the GPU's peak FP16 tensor core TFLOP/s.
Run NCU with --set=full --profile-from-start off --target-processes all.
If --set=full fails, try --set=detailed. Parse the CSV output for
sm__throughput.avg.pct_of_peak_sustained_elapsed.
Save raw NCU output to /workspace/.../ncu_output.txt.
When profiling on a remote SLURM cluster, include the Remote Execution Context block in the delegation prompt with the SSH+srun wrapper for the target cluster. The perf-profiling-specialist will prefix its commands (nsys, ncu, nvidia-smi) with this wrapper.
The perf-profiling-specialist does not need the remote-slurm skill — the
context block provides everything it needs to execute remotely.
Delegate profiling and domain-specific analysis to these specialists:
Structure every analysis report with these four sections:
## Summary
Training at 42% MFU, memory-bound due to large attention tensors.
## Metrics
- Throughput: 1,247 samples/sec
- MFU: 42% (vs 65% theoretical for this model)
- % of SOL: 58% (room for 1.7x improvement)
- GPU Utilization: 45%
- Memory Bandwidth: 850 GB/s (89% of peak)
- Kernel Count: 1,247 per iteration
## Findings
1. Self-attention consumes 60% of memory bandwidth
2. Optimizer step has 3 unnecessary synchronizations
3. Batch size could be increased by 2x
## Recommendations
1. Enable FlashAttention (expected: +15% MFU)
2. Remove synchronizations in optimizer (expected: +5% throughput)
3. Increase batch size to improve GPU utilization
Onboard a HuggingFace multimodal model (vision/audio/video + text) to the TensorRT-LLM PyTorch backend. Use when writing a new `tensorrt_llm/_torch/models/modeling_<vlm>.py` plus its input processor and weight mapper, or extending an existing VLM. Not for AutoDeploy — use `ad-model-onboard` for that path.
Adds sharding-aware IR hints (op substitutions, sharding kwargs, all_reduce insertions) directly into an existing AutoDeploy custom model (modeling_*.py). Edits the file in place — no separate _ir.py copy. Validates with apply_sharding_hints and end-to-end multi-GPU runs.
Review, design, and refactor TensorRT-LLM PyTorch MoE code for architecture fit, clean code, maintainability, and testability. Always use for any modification, review, refactor, or design planning that touches MoE modules, including tensorrt_llm/_torch/modules/fused_moe, ConfigurableMoE, MoE backends, MoEScheduler/moe_scheduler.py, forward execution/chunking, communication strategies, EPLB, quantization/weight handling, routing, factories, MoE docs, or MoE tests. Also use when the user asks whether a MoE design follows the current architecture or whether a MoE refactor is reasonable.
Compile TensorRT-LLM on a SLURM cluster. Covers submitting a batch job with a container image, monitoring the job, and verifying the build. Use when the user wants to compile TRT-LLM remotely via SLURM rather than on a local compute node.
Translates a HuggingFace model into a prefill-only AutoDeploy custom model using reference custom ops, validates with hierarchical equivalence tests.
Debug AutoDeploy accuracy regressions vs a reference score (PyTorch backend or published baseline). Use when an AutoDeploy model's eval score is significantly below the reference and the root cause is unknown.