Menu
Rank collective ops by total and p99 latency
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
| name | comm_bottleneck |
| description | Rank collective ops by total and p99 latency |
| category | distributed |
| tables | ["nccl.coll_perf","nccl.proxy_ops","python.comm_collective","rdma.mlx_hca"] |
| tags | ["NCCL","collective","communication","通信","all_reduce"] |
| keywords | {"en":["communication slow","NCCL","collective","comm bottleneck"],"zh":["通信慢","NCCL","all_reduce","通信瓶颈","带宽"]} |
| parameters | {"step_window":{"type":"integer","default":50},"use_global":{"type":"boolean","default":true}} |
按 collective op 聚合延迟与传输量，定位通信热点。 适用于「计算不慢但 step 时间长」或「通信占比高」的场景。
| 层 | 表 | 计时 |
|---|---|---|
| NCCL profiler 插件（优先） | nccl.coll_perf、nccl.proxy_ops | NCCL 原生事件，执行时间/带宽精准（见 timing_source） |
| Torch API 层插桩（回退） | python.comm_collective | Python 墙钟，launch 层粗粒度；独有 global_step 上下文 |
NCCL 插件启用时 Torch 侧插桩默认关闭（probing.torch.collective.enable=1 可强制同开）。
step_window (integer, default 50):use_global (boolean, default True):nccl.proxy_ops → skill: nccl_culprit_victim（culprit/victim 归因）NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps