Menu
Training stalled — capture stacks and thread/blocking hints
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | training_hang |
| description | Training stalled — capture stacks and thread/blocking hints |
| category | reliability |
| tables | ["python.backtrace","cpu.utilization","python.trace_event","python.comm_collective"] |
| tags | ["hang","stall","deadlock","卡住","不动"] |
| keywords | {"en":["hang","stuck","stall","frozen","deadlock","not progressing"],"zh":["卡住","hang","不动","停住","deadlock","死锁","loss 不更新"]} |
| parameters | {"stack_depth":{"type":"integer","default":20}} |
Loss 不更新、进程仍存活但无进展时使用。 组合：实时混合栈 + CPU 线程 wchan + 最近 trace 活动。
stack_depth (integer, default 20): Backtrace frames to show