CoreWeave GPU Node Forensics
Community-contributed. Not affiliated with, endorsed by, or sponsored by
CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.
"NVIDIA" and "Xid" are trademarks of NVIDIA Corporation; Xid semantics are
cited from NVIDIA's public documentation.
Triages a dead or degraded GPU on a CoreWeave node in seconds and returns one
grounded move — reschedule, reset-gpu, reboot-node, rma, watch, or
app-bug-not-hardware — from an NVIDIA Xid code or a pasted dmesg /
nvidia-smi blob. The classification is deterministic: a bundled script does the
mapping so the agent never guesses whether a card is dead, degraded, or fine.
Overview
A single bad GPU can kill a 64-GPU, multi-day training run — thousands of dollars
and days of wall-clock gone — because one rank stalls the whole collective. The
expensive mistakes are triage mistakes: RMAing a healthy card for an app bug,
restarting a job onto a GPU whose memory error was uncontained, or manually
uncordoning a node the lifecycle controller is trying to replace. This skill kills
that ambiguity.
The decision logic is grounded in the NVIDIA Xid error catalog
(https://docs.nvidia.com/deploy/xid-errors/) and CoreWeave's node-lifecycle /
cordon behavior. The math-of-the-matter — which Xid means what, and how
row-remapper state overrides it — lives in scripts/triage.py as a table the LLM
does not get to re-litigate. Deep domain knowledge (the full code→action table,
the row-remap decision, the cordon rules) loads from references/ on demand.
The headline is the Xid 94-vs-95 split. A contained memory error (94) cost
you one job restart on a healthy node; an uncontained one (95) means the GPU
could not isolate the fault and everything it touched is suspect. Getting that one
bit wrong is the difference between a 30-second reschedule and a run that quietly
trained on corrupt gradients. The script decides it; the skill never eyeballs it.
This skill is diagnostic, not destructive: it recommends the cordon / drain /
reset / RMA next-step but its tools are scoped read-only (nvidia-smi -q,
kubectl get, dmesg) — it never runs a reset, a reboot, or an uncordon itself.
Prerequisites
- The failure evidence. Either an Xid number, or a pasted
dmesg /
nvidia-smi dump. The skill works from a paste alone — no live cluster access
required — which is the common case (an operator pastes what the run's logs
showed).
- Optional live access for corroboration:
kubectl context on the CoreWeave
cluster (read-only is enough), and nvidia-smi on the node. If neither is
available the skill still triages from the paste.
python3 to run the deterministic classifier (scripts/triage.py, stdlib
only — no dependencies).
No secrets are handled. All commands are read-only queries.
Instructions
The pipeline is capture → classify → act. The classifier is authoritative for
the verdict; references/ supplies the "why" when a case needs depth.
Step 1: Capture the evidence
If the user has not already pasted it, ask for (or read) the fault signal. The two
richest sources:
dmesg -T | grep -i xid
nvidia-smi -q -d ROW_REMAPPER,ECC,PERFORMANCE
On a CoreWeave node you can also check who owns any cordon before acting:
kubectl get node NODE -o json | jq '{unschedulable: .spec.unschedulable, taints: .spec.taints}'
Step 2: Run the deterministic triage
Feed the Xid code, or the whole blob, to the classifier. Do not classify by
hand — the script owns the Xid→action mapping and the row-remap override.
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 95
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 63 --pending yes
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 48 --remap-failure yes
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --blob /path/to/dmesg.txt
dmesg -T | grep -i xid | python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py"
The script emits a VERDICT block plus a JSON object
({classification, severity, action, why, next_command, cordon_rule?, unverified?}).
Add --json for machine-readable output only.
When a blob carries several Xids, the most severe one governs the move and the
rest are reported as co_occurring_xids — a co-occurring app-side Xid 43 next to a
hardware Xid 79 does not soften the "reboot the node" verdict.
Step 3: Read the verdict and act on the action
Present the action and the next_command to the user in plain language. The six
actions and what each means live in
${CLAUDE_SKILL_DIR}/references/xid-triage-table.md.
Load it when the user wants the full table or asks about an Xid not in the summary.
reschedule — restart the failed rank; the node stays in service.reset-gpu — drain the GPU and reset it; re-run any job that shared it.reboot-node — the card is off the bus; only a bare-metal reboot returns it.rma — terminal hardware fault; the card must be replaced.watch — correctable / trending; monitor, do not act yet.app-bug-not-hardware — the app faulted, not the GPU. Do NOT RMA.
Step 4: The 94-vs-95 headline (load-bearing — get this right)
If the Xid is 94 or 95, state the containment explicitly, because the two look
almost identical in the logs and lead to opposite actions:
- Xid 94 (CONTAINED) →
reschedule. The error was isolated to the app's
context; the GPU and node are healthy. Restart the job. Do not reset or RMA.
- Xid 95 (UNCONTAINED) →
reset-gpu. The GPU could not isolate it; every
context it touched is suspect. Drain, reset, and re-run anything that shared the
card — otherwise the run may continue on corrupt state.
Step 5: Row-remap (Xid 63 / 64 / 48) — routine vs terminal
For any ECC/DBE or row-remap Xid, the nvidia-smi -q -d ROW_REMAPPER fields
override the base verdict. Pass them in (--pending, --remap-failure) and let
the script decide. The rule:
Remapping Failure Occurred: Yes → rma (terminal — sparing failed).Pending: Yes (no failure) → reset-gpu (routine — applies on reset).
Full logic + how to read the histogram:
${CLAUDE_SKILL_DIR}/references/row-remap-decision.md.
Step 6: The cordon hard rule (never violate)
Whenever the action is a hardware move (reset-gpu, reboot-node, rma), the
verdict carries a cordon_rule. Surface it verbatim:
Never manually uncordon a CoreWeave health cordon — the node-lifecycle
controller owns cordon/uncordon and is driving remediation. Uncordoning
re-admits the run onto known-bad hardware and races the controller.
Determine cordon provenance before touching schedulability. Full guidance:
${CLAUDE_SKILL_DIR}/references/cordon-rules.md.
Output
- A verdict — the deterministic
{classification, severity, action, why, next_command} for the governing signal, presented in plain language with the
single next command an operator runs.
- The 94-vs-95 call stated explicitly when either fires — contained →
reschedule, uncontained → reset — never conflated.
- The cordon rule attached to every hardware action, verbatim, so nobody
uncordons a node the controller is replacing.
- Co-occurring Xids listed when a blob carried several, with the most-severe
one named as governing.
[unverified] hedges surfaced honestly — e.g. CoreWeave's exact auto-RMA
thresholds and health-cordon taint strings are not publicly published, and the
skill says so rather than inventing a number.
Error Handling
| Situation | Cause | Response |
|---|
| Unmapped Xid | Xid not in the triage table | Script returns a conservative reset-gpu default, flags it [unverified], and points at the NVIDIA catalog. Do not RMA on an unmapped Xid. |
| Xid 94 vs 95 ambiguity | Both are "ECC error" in casual logs | Never infer from prose — use the exact code. 94 = contained (reschedule), 95 = uncontained (reset). |
| Row-remap without an Xid | Only nvidia-smi pasted | Script reads Remapping Failure Occurred / Pending and decides from those alone. |
| App-side Xid mistaken for hardware | Xid 13/31/43/45 | Classed app-bug-not-hardware; the fix is compute-sanitizer on the job, never an RMA. |
| Multiple Xids in one blob | Cascade (e.g. 43 then 79) | Most-severe governs; others listed as co_occurring_xids. |
| Empty / no signal | Blob has no Xid, remap, or thermal line | Script returns watch and asks for fresh dmesg + nvidia-smi -q evidence. |
| Thermal throttle, not a fault | HW/SW Thermal Slowdown : Active | Classed watch — healthy but hot; investigate cooling, not the card. |
| Tempted to uncordon a stalled node | Health cordon owned by the controller | Refuse. Surface the cordon rule; let remediation run. |
Examples
Example 1: "Xid 94 on one rank — do I need to replace the GPU?"
VERDICT: Xid 94 — Contained ECC/memory error [severity: HIGH]
ACTION: reschedule
WHY: The error was CONTAINED to the faulting application's context — the GPU
and node are healthy. Just reschedule/restart the job; no reset or RMA needed.
NEXT: Restart or reschedule the failed rank; the node stays in service.
No replacement. The containment did its job — restart the rank and keep the node.
Example 2: "Xid 95 — same run, different node"
VERDICT: Xid 95 — Uncontained ECC/memory error [severity: CRITICAL]
ACTION: reset-gpu
WHY: The error was UNCONTAINED — the GPU could not isolate it, so every context
it touched is suspect. Drain the GPU and reset it; treat all in-flight work as corrupt.
NEXT: Cordon + drain, GPU-reset (nvidia-smi -r) or node reset; re-run any job that shared this GPU.
CORDON: Do NOT manually uncordon a CoreWeave health cordon — the node-lifecycle
controller owns cordon/uncordon. Let it drain and replace the node.
Opposite of Example 1 despite looking identical in the logs: drain, reset, and
re-run the shared work — do not just restart.
Example 3: "GPU fell off the bus"
python3 scripts/triage.py --xid 79 → reboot-node. The card is unreachable on
the PCIe bus and returns only after a bare-metal reboot; expect the controller to
cordon and reboot — let it, and RMA only if it recurs after the reboot.
Example 4: "Xid 63 with a remap failure"
python3 scripts/triage.py --xid 63 --remap-failure yes → rma. The remapper
tried to swap in a spare row and physically could not — terminal. Cordon, drain,
open the RMA. (The script flags that CoreWeave's exact auto-RMA threshold is
[unverified].)
Example 5: "Xid 43 killed my job"
python3 scripts/triage.py --xid 43 → app-bug-not-hardware. Software-induced
fault — the fix is in the application (run it under compute-sanitizer), not an RMA.
Pulling the card would waste healthy hardware and never fix the job.
Resources
