debug

name

debug

description

Evidence-before-action diagnosis of failing ML experiments. Probes the system before guessing causes, process list, dmesg, GPU stats, log scrollback, checkpoint state, then states a hypothesis as a hypothesis and runs a smoke before claiming a root cause. Use when the user asks why a run is failing, diverging, OOMing, hanging, slow, producing weird metrics, has crashed, or asks to debug, diagnose, troubleshoot, or investigate a training issue.

Debug: evidence-before-action investigation

The most expensive class of mistake in ML debugging is asserting a cause based on plausibility, then attempting a "fix" that masks the real problem. This skill enforces the discipline of probe → hypothesis → smoke → controls → claim, in that order.

The agentic Stop hook routes here from reason when an assistant claims a cause without backing tool output.

When to run

The user just said any of:

• "why is X failing / diverging / NaN / OOM / hung / slow / crashed"
• "the loss is going up", "metrics look weird", "GPU util is 0"
• "debug this", "diagnose", "troubleshoot", "investigate this run"
• pasted a log excerpt asking what's wrong

Five-step protocol

Step 1: cheap probes

Before forming any hypothesis, gather the cheap evidence. None of these cost more than a few seconds:

Process state:

ps aux | grep -E '(python|train|torchrun|accelerate)' | grep -v grep

Is the process still running? Zombie? Defunct? Multiple instances?

Kernel / system events:

dmesg | tail -100 # OOM kills, hardware errors, NFS errors
journalctl -xe --since "1 hour ago" | tail -50