القائمة
Crash triage: grep PROBING CRASH in rank logs; read spill JSON under {PROBING_DATA_DIR}/crash/<pid>/latest.json
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | crash_triage |
| description | Crash triage: grep PROBING CRASH in rank logs; read spill JSON under {PROBING_DATA_DIR}/crash/<pid>/latest.json |
| category | triage |
| tags | ["crash","error","exception","startup","报错","崩溃"] |
| keywords | {"en":["crash","error","exception","failed","startup failure"],"zh":["崩溃","报错","异常","启动失败","挂了"]} |
训练崩溃或启动失败时：
PROBING CRASH{PROBING_DATA_DIR}/crash/<pid>/latest.json（与 memtable 同根目录，默认 Linux /dev/shm/probing，macOS $TMPDIR/probing）signal-latest.jsontorchrun 级联退出后 memtable 不可用——以日志 + spill 为准。