Menu
Find straggler ranks via collective latency imbalance
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | slow_rank |
| description | Find straggler ranks via collective latency imbalance |
| category | distributed |
| tables | ["python.comm_collective","global.python.comm_collective","nccl.proxy_ops"] |
| tags | ["rank","straggler","distributed","collective","拖后腿"] |
| keywords | {"en":["slow rank","straggler","lagging rank","which rank"],"zh":["慢 rank","拖后腿","straggler","掉队","哪个 rank","rank 慢"]} |
| parameters | {"step_window":{"type":"integer","default":20},"use_global":{"type":"boolean","default":true}} |
对比各 rank 的 collective 延迟，找出明显偏慢的 straggler。 单机多卡时 rank 列来自 torch.distributed；多机时使用 global.* 并带 _host/_rank 标签。
step_window (integer, default 20): Include collectives from the last N global_stepsuse_global (boolean, default True): Query global.python.comm_collective for cross-node fan-out