Menu
Crash triage: grep PROBING CRASH in rank logs; read spill JSON under {PROBING_DATA_DIR}/crash/<pid>/latest.json
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
| name | crash_triage |
| description | Crash triage: grep PROBING CRASH in rank logs; read spill JSON under {PROBING_DATA_DIR}/crash/<pid>/latest.json |
| category | triage |
| tags | ["crash","error","exception","startup","报错","崩溃"] |
| keywords | {"en":["crash","error","exception","failed","startup failure"],"zh":["崩溃","报错","异常","启动失败","挂了"]} |
训练崩溃或启动失败时：
PROBING CRASH{PROBING_DATA_DIR}/crash/<pid>/latest.json（与 memtable 同根目录，默认 Linux /dev/shm/probing，macOS $TMPDIR/probing）signal-latest.jsontorchrun 级联退出后 memtable 不可用——以日志 + spill 为准。
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps