القائمة
Find straggler ranks via collective latency imbalance
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
استنادا إلى تصنيف SOC المهني
| name | slow_rank |
| description | Find straggler ranks via collective latency imbalance |
| category | distributed |
| tables | ["python.comm_collective","global.python.comm_collective","nccl.proxy_ops"] |
| tags | ["rank","straggler","distributed","collective","拖后腿"] |
| keywords | {"en":["slow rank","straggler","lagging rank","which rank"],"zh":["慢 rank","拖后腿","straggler","掉队","哪个 rank","rank 慢"]} |
| parameters | {"step_window":{"type":"integer","default":20},"use_global":{"type":"boolean","default":true}} |
对比各 rank 的 collective 延迟，找出明显偏慢的 straggler。 单机多卡时 rank 列来自 torch.distributed；多机时使用 global.* 并带 _host/_rank 标签。
step_window (integer, default 20): Include collectives from the last N global_stepsuse_global (boolean, default True): Query global.python.comm_collective for cross-node fan-out