Menü
Job-level health: cluster alive, step lag, slowdown proxy, comm trend
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Basierend auf der SOC-Berufsklassifikation
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | job_health |
| description | Job-level health: cluster alive, step lag, slowdown proxy, comm trend |
| category | triage |
| tables | ["cluster.nodes","python.comm_collective","global.python.comm_collective","python.trace_event"] |
| tags | ["job","slowdown","health","作业"] |
| keywords | {"en":["job health","job slow","slowdown","training degraded","step lag"],"zh":["作业健康","job 慢","整体变慢","slowdown","step 落后"]} |
| parameters | {"step_window":{"type":"integer","default":200},"step_lag_threshold":{"type":"integer","default":5},"use_global":{"type":"boolean","default":true}} |
从 整 job 视角看训练是否健康，而不是单个 rank 的单次 collective。
| 步骤 | 含义 |
|---|---|
| cluster_snapshot | 多少节点 alive / dead |
| rank_step_lag | 哪些 rank 的 global_step 明显落后（hang 信号） |
| job_slowdown_proxy | 各 step 上最慢 rank vs 最快 rank 的比值均值（§ federation 3.2） |
| comm_trend_by_step | 最近 step 通信总耗时是否爬升 |
| train_step_pacing | train.step span 的 wall time（若已开启 phase hook） |
step_window (default 200): 分析最近多少 global_stepstep_lag_threshold (default 5): 落后 cluster max step 多少步算异常use_global (default true): rank0 上 fan-out 到全集群probing -t rank0:8080 skill run job_health --global
probing -t rank0:8080 skill run job_health --set step_window=500
persistent_stragglerslow_ranktraining_hang