메뉴
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
Find slowest PyTorch modules in recent steps
SOC 직업 분류 기준
| name | watchdog_timeout |
| description | Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment. |
| category | distributed |
| tables | ["python.torch_nccl_flight_record","global.python.torch_nccl_flight_record","python.torch_nccl_pg_status"] |
| tags | ["NCCL","watchdog","timeout","Flight Recorder","desync","collective"] |
| keywords | {"en":["NCCL watchdog","timeout","flight recorder","collective mismatch","desync"],"zh":["NCCL watchdog","超时","Flight Recorder","collective 不一致","反同步","卡住"]} |
| parameters | {"seq_window":{"type":"integer","default":200},"use_global":{"type":"boolean","default":true}} |
用于 PyTorch NCCL watchdog timeout、collective deadlock、rank 间 collective 顺序不一致等场景。
这个 skill 会先触发 PyTorch Flight Recorder snapshot，并写入：
python.torch_nccl_flight_recordpython.torch_nccl_pg_status然后按 pg_id + collective_seq_id 对齐各 rank，检查 missing rank、collective type mismatch、shape/dtype mismatch、state mismatch。
torch._C._distributed_c10d._dump_nccl_traceTORCH_NCCL_TRACE_BUFFER_SIZE=2000TORCH_NCCL_DUMP_ON_TIMEOUT=1ranks_seen < expected_world_size：某些 rank 没有进入同一个 collective seq，常见于 CPU-side hang 或分支发散。op_min != op_max：同一个 seq 上不同 rank 发起了不同 collective。input_sizes_min != input_sizes_max 或 dtype 不同：collective 参数不一致。state_min != state_max：某些 rank 未完成、未开始或处于不同状态。nccl_culprit_victimtraining_hangcomm_bottleneck