메뉴
Rank collective ops by total and p99 latency
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | comm_bottleneck |
| description | Rank collective ops by total and p99 latency |
| category | distributed |
| tables | ["nccl.coll_perf","nccl.proxy_ops","python.comm_collective","rdma.mlx_hca"] |
| tags | ["NCCL","collective","communication","通信","all_reduce"] |
| keywords | {"en":["communication slow","NCCL","collective","comm bottleneck"],"zh":["通信慢","NCCL","all_reduce","通信瓶颈","带宽"]} |
| parameters | {"step_window":{"type":"integer","default":50},"use_global":{"type":"boolean","default":true}} |
按 collective op 聚合延迟与传输量，定位通信热点。 适用于「计算不慢但 step 时间长」或「通信占比高」的场景。
| 层 | 表 | 计时 |
|---|---|---|
| NCCL profiler 插件（优先） | nccl.coll_perf、nccl.proxy_ops | NCCL 原生事件，执行时间/带宽精准（见 timing_source） |
| Torch API 层插桩（回退） | python.comm_collective | Python 墙钟，launch 层粗粒度；独有 global_step 上下文 |
NCCL 插件启用时 Torch 侧插桩默认关闭（probing.torch.collective.enable=1 可强制同开）。
step_window (integer, default 50):use_global (boolean, default True):nccl.proxy_ops → skill: nccl_culprit_victim（culprit/victim 归因）