| name | nccl_culprit_victim |
| description | NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
|
| category | distributed |
| tables | ["nccl.proxy_ops","nccl.net_qp","global.nccl.proxy_ops"] |
| tags | ["NCCL","culprit","victim","straggler","wait","proxy"] |
| keywords | {"en":["NCCL culprit","NCCL victim","recv wait","gpu wait","slow rank NCCL","proxy wait"],"zh":["NCCL 慢 rank","罪魁祸首","受害者","recv_wait","gpu_wait","等待分解"]} |
| parameters | {"seq_window":{"type":"integer","default":20},"use_global":{"type":"boolean","default":true}} |
NCCL culprit / victim attribution
Uses NCCL profiler plugin wait decomposition (nccl.proxy_ops) with peer-topology attribution (rank↔peer edges, not global wait sorting):
- Culprit (slow local GPU): high
send_gpu_wait_ns on a send edge (rank → peer)
- Victim (waiting on peers / network): high
recv_wait_ns on recv edge; check attributed_root_rank
- Propagated victim: recv wait high but peer's
send_gpu_wait_ns is also high — root cause is upstream
- Receiver congestion (v4 ABI only): high
send_peer_wait_ns — this rank's
sends are stalled waiting for the peer's clear-to-send credits; investigate
that peer
Requires NCCL_PROFILER_PLUGIN + probing-nccl-profiler (or mock data via PROBING_NCCL_MOCK on macOS).
Parameters
seq_window (integer, default 20): recent collective sequences to analyzeuse_global (boolean, default true): fan-out with global.nccl.proxy_ops
Related skills
- Coarse collective latency →
comm_bottleneck / slow_rank (python.comm_collective)
- IB QP timing →
nccl.net_qp when NetPlugin mask enabled
