القائمة
Detect monotonic GPU memory growth across training steps
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Find slowest PyTorch modules in recent steps
| name | memory_leak |
| description | Detect monotonic GPU memory growth across training steps |
| category | memory |
| tables | ["python.torch_trace","gpu.utilization"] |
| tags | ["memory","leak","OOM","显存","泄漏"] |
| keywords | {"en":["memory leak","OOM","memory growing","out of memory"],"zh":["泄漏","OOM","显存涨","内存涨","out of memory","阶梯"]} |
| parameters | {"min_steps":{"type":"integer","default":10},"step_skip":{"type":"integer","default":2}} |
检测 python.torch_trace 中 allocated 是否随 step 单调上涨， 并结合 gpu.utilization 看设备级显存趋势。
min_steps (integer, default 10): Minimum steps required for trend analysisstep_skip (integer, default 2): Skip first N steps (discovery / warmup)