メニュー
Detect monotonic GPU memory growth across training steps
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Find slowest PyTorch modules in recent steps
| name | memory_leak |
| description | Detect monotonic GPU memory growth across training steps |
| category | memory |
| tables | ["python.torch_trace","gpu.utilization"] |
| tags | ["memory","leak","OOM","显存","泄漏"] |
| keywords | {"en":["memory leak","OOM","memory growing","out of memory"],"zh":["泄漏","OOM","显存涨","内存涨","out of memory","阶梯"]} |
| parameters | {"min_steps":{"type":"integer","default":10},"step_skip":{"type":"integer","default":2}} |
检测 python.torch_trace 中 allocated 是否随 step 单调上涨， 并结合 gpu.utilization 看设备级显存趋势。
min_steps (integer, default 10): Minimum steps required for trend analysisstep_skip (integer, default 2): Skip first N steps (discovery / warmup)