القائمة
Training stalled — capture stacks and thread/blocking hints
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
| name | training_hang |
| description | Training stalled — capture stacks and thread/blocking hints |
| category | reliability |
| tables | ["python.backtrace","cpu.utilization","python.trace_event","python.comm_collective"] |
| tags | ["hang","stall","deadlock","卡住","不动"] |
| keywords | {"en":["hang","stuck","stall","frozen","deadlock","not progressing"],"zh":["卡住","hang","不动","停住","deadlock","死锁","loss 不更新"]} |
| parameters | {"stack_depth":{"type":"integer","default":20}} |
Loss 不更新、进程仍存活但无进展时使用。 组合：实时混合栈 + CPU 线程 wchan + 最近 trace 活动。
stack_depth (integer, default 20): Backtrace frames to showNCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps