メニュー
Crash triage: grep PROBING CRASH in rank logs; read spill JSON under {PROBING_DATA_DIR}/crash/<pid>/latest.json
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | crash_triage |
| description | Crash triage: grep PROBING CRASH in rank logs; read spill JSON under {PROBING_DATA_DIR}/crash/<pid>/latest.json |
| category | triage |
| tags | ["crash","error","exception","startup","报错","崩溃"] |
| keywords | {"en":["crash","error","exception","failed","startup failure"],"zh":["崩溃","报错","异常","启动失败","挂了"]} |
训练崩溃或启动失败时：
PROBING CRASH{PROBING_DATA_DIR}/crash/<pid>/latest.json（与 memtable 同根目录，默认 Linux /dev/shm/probing，macOS $TMPDIR/probing）signal-latest.jsontorchrun 级联退出后 memtable 不可用——以日志 + spill 为准。