probing

NCCL proxy wait decomposition — culprit (send_gpu_wait) vs victim (recv_wait)

One-shot health check: CPU, GPU, tables, torch step progress, TorchProbe overhead, cluster nodes, NCCL profiler health

SRE first-response triage for distributed training incidents. Automates the manual checks from PyTorch/NCCL debugging runbooks.

Diagnose PyTorch NCCL watchdog timeouts with Flight Recorder collective sequence alignment.

GPU memory and utilization headroom

Detect monotonic GPU memory growth across training steps

Find slowest PyTorch modules in recent steps

Demo skill from probing-acme vendor package. Use to verify pip-installed extension discovery (skills + magics).

Rank collective ops by total and p99 latency

Crash triage: grep PROBING CRASH in rank logs; read spill JSON under {PROBING_DATA_DIR}/crash/<pid>/latest.json

Job-level health: cluster alive, step lag, slowdown proxy, comm trend

Find ranks that are slow on many steps (worst_fraction), not just once

Find straggler ranks via collective latency imbalance

Training stalled — capture stacks and thread/blocking hints