Menü
Find straggler ranks via collective latency imbalance
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Basierend auf der SOC-Berufsklassifikation
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | slow_rank |
| description | Find straggler ranks via collective latency imbalance |
| category | distributed |
| tables | ["python.comm_collective","global.python.comm_collective","nccl.proxy_ops"] |
| tags | ["rank","straggler","distributed","collective","拖后腿"] |
| keywords | {"en":["slow rank","straggler","lagging rank","which rank"],"zh":["慢 rank","拖后腿","straggler","掉队","哪个 rank","rank 慢"]} |
| parameters | {"step_window":{"type":"integer","default":20},"use_global":{"type":"boolean","default":true}} |
对比各 rank 的 collective 延迟，找出明显偏慢的 straggler。 单机多卡时 rank 列来自 torch.distributed；多机时使用 global.* 并带 _host/_rank 标签。
step_window (integer, default 20): Include collectives from the last N global_stepsuse_global (boolean, default True): Query global.python.comm_collective for cross-node fan-out