メニュー
Find straggler ranks via collective latency imbalance
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | slow_rank |
| description | Find straggler ranks via collective latency imbalance |
| category | distributed |
| tables | ["python.comm_collective","global.python.comm_collective","nccl.proxy_ops"] |
| tags | ["rank","straggler","distributed","collective","拖后腿"] |
| keywords | {"en":["slow rank","straggler","lagging rank","which rank"],"zh":["慢 rank","拖后腿","straggler","掉队","哪个 rank","rank 慢"]} |
| parameters | {"step_window":{"type":"integer","default":20},"use_global":{"type":"boolean","default":true}} |
对比各 rank 的 collective 延迟，找出明显偏慢的 straggler。 单机多卡时 rank 列来自 torch.distributed；多机时使用 global.* 并带 _host/_rank 标签。
step_window (integer, default 20): Include collectives from the last N global_stepsuse_global (boolean, default True): Query global.python.comm_collective for cross-node fan-out