Triage a dead or degraded GPU on a CoreWeave node fast — decide reschedule
vs GPU-reset vs node-reboot vs RMA from an Xid code or a pasted dmesg /
nvidia-smi blob, so a bad card does not silently kill a multi-day training run.
Use when a GPU throws an Xid error, a node "fell off the bus", a training run
stalls or NCCL hangs on one rank, or you need to know whether to replace,
reset, or just reschedule. Trigger with "xid error", "gpu fell off the bus",
"coreweave gpu dead", "should I RMA this GPU", "gpu node triage".
Triage a dead or degraded GPU on a CoreWeave node fast — decide reschedule
vs GPU-reset vs node-reboot vs RMA from an Xid code or a pasted dmesg /
nvidia-smi blob, so a bad card does not silently kill a multi-day training run.
Use when a GPU throws an Xid error, a node "fell off the bus", a training run
stalls or NCCL hangs on one rank, or you need to know whether to replace,
reset, or just reschedule. Trigger with "xid error", "gpu fell off the bus",
"coreweave gpu dead", "should I RMA this GPU", "gpu node triage".
Community-contributed. Not affiliated with, endorsed by, or sponsored by
CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.
"NVIDIA" and "Xid" are trademarks of NVIDIA Corporation; Xid semantics are
cited from NVIDIA's public documentation.
Triages a dead or degraded GPU on a CoreWeave node in seconds and returns one
grounded move — reschedule, reset-gpu, reboot-node, rma, watch, or
app-bug-not-hardware — from an NVIDIA Xid code or a pasted dmesg /
nvidia-smi blob. The classification is deterministic: a bundled script does the
mapping so the agent never guesses whether a card is dead, degraded, or fine.
Overview
A single bad GPU can kill a 64-GPU, multi-day training run — thousands of dollars
and days of wall-clock gone — because one rank stalls the whole collective. The
expensive mistakes are triage mistakes: RMAing a healthy card for an app bug,
restarting a job onto a GPU whose memory error was uncontained, or manually
uncordoning a node the lifecycle controller is trying to replace. This skill kills
that ambiguity.
The decision logic is grounded in the NVIDIA Xid error catalog
(https://docs.nvidia.com/deploy/xid-errors/) and CoreWeave's node-lifecycle /
cordon behavior. The math-of-the-matter — which Xid means what, and how
row-remapper state overrides it — lives in scripts/triage.py as a table the LLM
does not get to re-litigate. Deep domain knowledge (the full code→action table,
the row-remap decision, the cordon rules) loads from references/ on demand.
The headline is the Xid 94-vs-95 split. A contained memory error (94) cost
you one job restart on a healthy node; an uncontained one (95) means the GPU
could not isolate the fault and everything it touched is suspect. Getting that one
bit wrong is the difference between a 30-second reschedule and a run that quietly
trained on corrupt gradients. The script decides it; the skill never eyeballs it.
This skill is diagnostic, not destructive: it recommends the cordon / drain /
reset / RMA next-step but its tools are scoped read-only (nvidia-smi -q,
kubectl get, dmesg) — it never runs a reset, a reboot, or an uncordon itself.
Prerequisites
The failure evidence. Either an Xid number, or a pasted dmesg /
nvidia-smi dump. The skill works from a paste alone — no live cluster access
required — which is the common case (an operator pastes what the run's logs
showed).
Optional live access for corroboration: kubectl context on the CoreWeave
cluster (read-only is enough), and nvidia-smi on the node. If neither is
available the skill still triages from the paste.
python3 to run the deterministic classifier (scripts/triage.py, stdlib
only — no dependencies).
No secrets are handled. All commands are read-only queries.
Instructions
The pipeline is capture → classify → act. The classifier is authoritative for
the verdict; references/ supplies the "why" when a case needs depth.
Step 1: Capture the evidence
If the user has not already pasted it, ask for (or read) the fault signal. The two
richest sources:
Feed the Xid code, or the whole blob, to the classifier. Do not classify by
hand — the script owns the Xid→action mapping and the row-remap override.
# From an Xid code:
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 95
# With row-remapper state (Xid 63/64 or a DBE):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 63 --pending yes
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 48 --remap-failure yes# From a pasted dmesg / nvidia-smi blob (file or stdin):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --blob /path/to/dmesg.txt
dmesg -T | grep -i xid | python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py"
The script emits a VERDICT block plus a JSON object
({classification, severity, action, why, next_command, cordon_rule?, unverified?}).
Add --json for machine-readable output only.
When a blob carries several Xids, the most severe one governs the move and the
rest are reported as co_occurring_xids — a co-occurring app-side Xid 43 next to a
hardware Xid 79 does not soften the "reboot the node" verdict.
Step 3: Read the verdict and act on the action
Present the action and the next_command to the user in plain language. The six
actions and what each means live in
${CLAUDE_SKILL_DIR}/references/xid-triage-table.md.
Load it when the user wants the full table or asks about an Xid not in the summary.
reschedule — restart the failed rank; the node stays in service.
reset-gpu — drain the GPU and reset it; re-run any job that shared it.
reboot-node — the card is off the bus; only a bare-metal reboot returns it.
rma — terminal hardware fault; the card must be replaced.
watch — correctable / trending; monitor, do not act yet.
app-bug-not-hardware — the app faulted, not the GPU. Do NOT RMA.
Step 4: The 94-vs-95 headline (load-bearing — get this right)
If the Xid is 94 or 95, state the containment explicitly, because the two look
almost identical in the logs and lead to opposite actions:
Xid 94 (CONTAINED) → reschedule. The error was isolated to the app's
context; the GPU and node are healthy. Restart the job. Do not reset or RMA.
Xid 95 (UNCONTAINED) → reset-gpu. The GPU could not isolate it; every
context it touched is suspect. Drain, reset, and re-run anything that shared the
card — otherwise the run may continue on corrupt state.
For any ECC/DBE or row-remap Xid, the nvidia-smi -q -d ROW_REMAPPER fields
override the base verdict. Pass them in (--pending, --remap-failure) and let
the script decide. The rule:
Whenever the action is a hardware move (reset-gpu, reboot-node, rma), the
verdict carries a cordon_rule. Surface it verbatim:
Never manually uncordon a CoreWeave health cordon — the node-lifecycle
controller owns cordon/uncordon and is driving remediation. Uncordoning
re-admits the run onto known-bad hardware and races the controller.
A verdict — the deterministic {classification, severity, action, why, next_command} for the governing signal, presented in plain language with the
single next command an operator runs.
The 94-vs-95 call stated explicitly when either fires — contained →
reschedule, uncontained → reset — never conflated.
The cordon rule attached to every hardware action, verbatim, so nobody
uncordons a node the controller is replacing.
Co-occurring Xids listed when a blob carried several, with the most-severe
one named as governing.
[unverified] hedges surfaced honestly — e.g. CoreWeave's exact auto-RMA
thresholds and health-cordon taint strings are not publicly published, and the
skill says so rather than inventing a number.
Error Handling
Situation
Cause
Response
Unmapped Xid
Xid not in the triage table
Script returns a conservative reset-gpu default, flags it [unverified], and points at the NVIDIA catalog. Do not RMA on an unmapped Xid.
Xid 94 vs 95 ambiguity
Both are "ECC error" in casual logs
Never infer from prose — use the exact code. 94 = contained (reschedule), 95 = uncontained (reset).
Row-remap without an Xid
Only nvidia-smi pasted
Script reads Remapping Failure Occurred / Pending and decides from those alone.
App-side Xid mistaken for hardware
Xid 13/31/43/45
Classed app-bug-not-hardware; the fix is compute-sanitizer on the job, never an RMA.
Multiple Xids in one blob
Cascade (e.g. 43 then 79)
Most-severe governs; others listed as co_occurring_xids.
Empty / no signal
Blob has no Xid, remap, or thermal line
Script returns watch and asks for fresh dmesg + nvidia-smi -q evidence.
Thermal throttle, not a fault
HW/SW Thermal Slowdown : Active
Classed watch — healthy but hot; investigate cooling, not the card.
Tempted to uncordon a stalled node
Health cordon owned by the controller
Refuse. Surface the cordon rule; let remediation run.
Examples
Example 1: "Xid 94 on one rank — do I need to replace the GPU?"
VERDICT: Xid 94 — Contained ECC/memory error [severity: HIGH]
ACTION: reschedule
WHY: The error was CONTAINED to the faulting application's context — the GPU
and node are healthy. Just reschedule/restart the job; no reset or RMA needed.
NEXT: Restart or reschedule the failed rank; the node stays in service.
No replacement. The containment did its job — restart the rank and keep the node.
Example 2: "Xid 95 — same run, different node"
VERDICT: Xid 95 — Uncontained ECC/memory error [severity: CRITICAL]
ACTION: reset-gpu
WHY: The error was UNCONTAINED — the GPU could not isolate it, so every context
it touched is suspect. Drain the GPU and reset it; treat all in-flight work as corrupt.
NEXT: Cordon + drain, GPU-reset (nvidia-smi -r) or node reset; re-run any job that shared this GPU.
CORDON: Do NOT manually uncordon a CoreWeave health cordon — the node-lifecycle
controller owns cordon/uncordon. Let it drain and replace the node.
Opposite of Example 1 despite looking identical in the logs: drain, reset, and
re-run the shared work — do not just restart.
Example 3: "GPU fell off the bus"
python3 scripts/triage.py --xid 79 → reboot-node. The card is unreachable on
the PCIe bus and returns only after a bare-metal reboot; expect the controller to
cordon and reboot — let it, and RMA only if it recurs after the reboot.
Example 4: "Xid 63 with a remap failure"
python3 scripts/triage.py --xid 63 --remap-failure yes → rma. The remapper
tried to swap in a spare row and physically could not — terminal. Cordon, drain,
open the RMA. (The script flags that CoreWeave's exact auto-RMA threshold is
[unverified].)
Example 5: "Xid 43 killed my job"
python3 scripts/triage.py --xid 43 → app-bug-not-hardware. Software-induced
fault — the fix is in the application (run it under compute-sanitizer), not an RMA.
Pulling the card would waste healthy hardware and never fix the job.
