메뉴
Find ranks that are slow on many steps (worst_fraction), not just once
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
SOC 직업 분류 기준
| name | persistent_straggler |
| description | Find ranks that are slow on many steps (worst_fraction), not just once |
| category | distributed |
| tables | ["python.comm_collective","global.python.comm_collective"] |
| tags | ["straggler","persistent","worst_fraction"] |
| keywords | {"en":["persistent straggler","worst fraction","always slow","chronic straggler"],"zh":["长期慢","一直慢","worst_fraction","总是最慢"]} |
| parameters | {"step_window":{"type":"integer","default":300},"worst_fraction_threshold":{"type":"number","default":0.25},"use_global":{"type":"boolean","default":true}} |
slow_rank 看 当前窗口里谁平均最慢；本 skill 看 谁在多少 step 上拿「最慢 rank」。
对每个 rank 统计：在窗口内有多少 step 上，该 rank 的 collective 总耗时等于该 step 的最大值。
worst_fraction ≈ 1.0 — 几乎每步都是最慢（ chronic straggler）worst_fraction ≈ 0.05 — 偶发step_window (default 300)worst_fraction_threshold (default 0.25) — persistent_offenders 步骤的过滤阈值use_global (default true)probing -t rank0:8080 skill run persistent_straggler --global
probing -t rank0:8080 skill run persistent_straggler --set worst_fraction_threshold=0.4 --set step_window=500
job_healthslow_ranknccl_culprit_victim