メニュー
Job-level health: cluster alive, step lag, slowdown proxy, comm trend
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
| name | job_health |
| description | Job-level health: cluster alive, step lag, slowdown proxy, comm trend |
| category | triage |
| tables | ["cluster.nodes","python.comm_collective","global.python.comm_collective","python.trace_event"] |
| tags | ["job","slowdown","health","作业"] |
| keywords | {"en":["job health","job slow","slowdown","training degraded","step lag"],"zh":["作业健康","job 慢","整体变慢","slowdown","step 落后"]} |
| parameters | {"step_window":{"type":"integer","default":200},"step_lag_threshold":{"type":"integer","default":5},"use_global":{"type":"boolean","default":true}} |
从 整 job 视角看训练是否健康，而不是单个 rank 的单次 collective。
| 步骤 | 含义 |
|---|---|
| cluster_snapshot | 多少节点 alive / dead |
| rank_step_lag | 哪些 rank 的 global_step 明显落后（hang 信号） |
| job_slowdown_proxy | 各 step 上最慢 rank vs 最快 rank 的比值均值（§ federation 3.2） |
| comm_trend_by_step | 最近 step 通信总耗时是否爬升 |
| train_step_pacing | train.step span 的 wall time（若已开启 phase hook） |
step_window (default 200): 分析最近多少 global_stepstep_lag_threshold (default 5): 落后 cluster max step 多少步算异常use_global (default true): rank0 上 fan-out 到全集群probing -t rank0:8080 skill run job_health --global
probing -t rank0:8080 skill run job_health --set step_window=500
persistent_stragglerslow_ranktraining_hang