Menu
Detect monotonic GPU memory growth across training steps
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Find slowest PyTorch modules in recent steps
Based on SOC occupation classification
| name | memory_leak |
| description | Detect monotonic GPU memory growth across training steps |
| category | memory |
| tables | ["python.torch_trace","gpu.utilization"] |
| tags | ["memory","leak","OOM","显存","泄漏"] |
| keywords | {"en":["memory leak","OOM","memory growing","out of memory"],"zh":["泄漏","OOM","显存涨","内存涨","out of memory","阶梯"]} |
| parameters | {"min_steps":{"type":"integer","default":10},"step_skip":{"type":"integer","default":2}} |
检测 python.torch_trace 中 allocated 是否随 step 单调上涨， 并结合 gpu.utilization 看设备级显存趋势。
min_steps (integer, default 10): Minimum steps required for trend analysisstep_skip (integer, default 2): Skip first N steps (discovery / warmup)