القائمة
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
Find slowest PyTorch modules in recent steps
| name | watchdog_timeout |
| description | Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment. |
| category | distributed |
| tables | ["python.torch_nccl_flight_record","global.python.torch_nccl_flight_record","python.torch_nccl_pg_status"] |
| tags | ["NCCL","watchdog","timeout","Flight Recorder","desync","collective"] |
| keywords | {"en":["NCCL watchdog","timeout","flight recorder","collective mismatch","desync"],"zh":["NCCL watchdog","超时","Flight Recorder","collective 不一致","反同步","卡住"]} |
| parameters | {"seq_window":{"type":"integer","default":200},"use_global":{"type":"boolean","default":true}} |
用于 PyTorch NCCL watchdog timeout、collective deadlock、rank 间 collective 顺序不一致等场景。
这个 skill 会先触发 PyTorch Flight Recorder snapshot，并写入：
python.torch_nccl_flight_recordpython.torch_nccl_pg_status然后按 pg_id + collective_seq_id 对齐各 rank，检查 missing rank、collective type mismatch、shape/dtype mismatch、state mismatch。
torch._C._distributed_c10d._dump_nccl_traceTORCH_NCCL_TRACE_BUFFER_SIZE=2000TORCH_NCCL_DUMP_ON_TIMEOUT=1ranks_seen < expected_world_size：某些 rank 没有进入同一个 collective seq，常见于 CPU-side hang 或分支发散。op_min != op_max：同一个 seq 上不同 rank 发起了不同 collective。input_sizes_min != input_sizes_max 或 dtype 不同：collective 参数不一致。state_min != state_max：某些 rank 未完成、未开始或处于不同状态。nccl_culprit_victimtraining_hangcomm_bottleneck