메뉴
SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)
One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health
Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.
GPU memory and utilization headroom
Detect monotonic GPU memory growth across training steps
Find slowest PyTorch modules in recent steps
SOC 직업 분류 기준
| name | sre_triage |
| description | SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks. |
| category | sre |
| tables | ["cluster.nodes","cpu.utilization","gpu.utilization","python.comm_collective","python.torch_nccl_flight_record","nccl.proxy_ops","rdma.mlx_hca","process.kmsg"] |
| tags | ["SRE","runbook","oncall","triage","NCCL","hang","OOM","network"] |
| keywords | {"en":["SRE","oncall","runbook","incident","triage","NCCL timeout","hang"],"zh":["SRE","值班","应急","runbook","分诊","事故","卡住","NCCL 超时"]} |
| parameters | {"use_global":{"type":"boolean","default":true},"seq_window":{"type":"integer","default":200},"sample_limit":{"type":"integer","default":20}} |
面向值班第一响应：把原本需要人工 ssh、grep、py-spy、看 NCCL/IB/GPU/内核日志的流程，尽量转成 probing 可执行证据链。
这个 skill 不追求一次性给出最终根因，而是自动回答：
incident_snapshot：当前有哪些表，决定能自动化到什么深度。cluster_nodes：rank/host 是否齐，是否有 dead/stale 节点。blocked_threads / kernel_wait_points：类似人工 py-spy/strace 的第一层替代。flight_recorder_alignment：自动化 Flight Recorder / fr_trace 的核心对齐判断。nccl_wait_summary：进入 NCCL 后的 culprit/victim 信号。rdma_network_hints / kernel_fault_hints：网络或硬件灰故障线索。watchdog_timeoutnccl_culprit_victimtraining_hangmemory_leak / gpu_pressure