メニュー
Training stalled — capture stacks and thread/blocking hints
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | training_hang |
| description | Training stalled — capture stacks and thread/blocking hints |
| category | reliability |
| tables | ["python.backtrace","cpu.utilization","python.trace_event","python.comm_collective"] |
| tags | ["hang","stall","deadlock","卡住","不动"] |
| keywords | {"en":["hang","stuck","stall","frozen","deadlock","not progressing"],"zh":["卡住","hang","不动","停住","deadlock","死锁","loss 不更新"]} |
| parameters | {"stack_depth":{"type":"integer","default":20}} |
Loss 不更新、进程仍存活但无进展时使用。 组合：实时混合栈 + CPU 线程 wchan + 最近 trace 活动。
stack_depth (integer, default 20): Backtrace frames to show