메뉴
Crash triage: grep PROBING CRASH in rank logs; read spill JSON under {PROBING_DATA_DIR}/crash/<pid>/latest.json
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | crash_triage |
| description | Crash triage: grep PROBING CRASH in rank logs; read spill JSON under {PROBING_DATA_DIR}/crash/<pid>/latest.json |
| category | triage |
| tags | ["crash","error","exception","startup","报错","崩溃"] |
| keywords | {"en":["crash","error","exception","failed","startup failure"],"zh":["崩溃","报错","异常","启动失败","挂了"]} |
训练崩溃或启动失败时：
PROBING CRASH{PROBING_DATA_DIR}/crash/<pid>/latest.json（与 memtable 同根目录，默认 Linux /dev/shm/probing，macOS $TMPDIR/probing）signal-latest.jsontorchrun 级联退出后 memtable 不可用——以日志 + spill 为准。