Menu
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
| name | nccl_culprit_victim |
| description | NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait) |
| category | distributed |
| tables | ["nccl.proxy_ops","nccl.net_qp","global.nccl.proxy_ops"] |
| tags | ["NCCL","culprit","victim","straggler","wait","proxy"] |
| keywords | {"en":["NCCL culprit","NCCL victim","recv wait","gpu wait","slow rank NCCL","proxy wait"],"zh":["NCCL 慢 rank","罪魁祸首","受害者","recv_wait","gpu_wait","等待分解"]} |
| parameters | {"seq_window":{"type":"integer","default":20},"use_global":{"type":"boolean","default":true}} |
Uses NCCL profiler plugin wait decomposition (nccl.proxy_ops) with peer-topology attribution (rank↔peer edges, not global wait sorting):
send_gpu_wait_ns on a send edge (rank → peer)recv_wait_ns on recv edge; check attributed_root_ranksend_gpu_wait_ns is also high — root cause is upstreamsend_peer_wait_ns — this rank's
sends are stalled waiting for the peer's clear-to-send credits; investigate
that peerRequires NCCL_PROFILER_PLUGIN + probing-nccl-profiler (or mock data via PROBING_NCCL_MOCK on macOS).
seq_window (integer, default 20): recent collective sequences to analyzeuse_global (boolean, default true): fan-out with global.nccl.proxy_opscomm_bottleneck / slow_rank (python.comm_collective)nccl.net_qp when NetPlugin mask enabledOne-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
Find slowest PyTorch modules in recent steps