coreweave-gpu-node-forensics

name	coreweave-gpu-node-forensics
description	Triage a dead or degraded GPU on a CoreWeave node fast — decide reschedule vs GPU-reset vs node-reboot vs RMA from an Xid code or a pasted dmesg / nvidia-smi blob, so a bad card does not silently kill a multi-day training run. Use when a GPU throws an Xid error, a node "fell off the bus", a training run stalls or NCCL hangs on one rank, or you need to know whether to replace, reset, or just reschedule. Trigger with "xid error", "gpu fell off the bus", "coreweave gpu dead", "should I RMA this GPU", "gpu node triage".
allowed-tools	Read, Bash(python3:*), Bash(nvidia-smi -q:*), Bash(kubectl get:*), Bash(dmesg:*)
version	1.11.0
author	Jeremy Longshore <[e-mail]>
license	MIT
compatibility	Designed for Claude Code, also compatible with Codex
tags	["saas","coreweave","gpu-cloud","reliability","xid"]

CoreWeave GPU Node Forensics

Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc. "NVIDIA" and "Xid" are trademarks of NVIDIA Corporation; Xid semantics are cited from NVIDIA's public documentation.

Triages a dead or degraded GPU on a CoreWeave node in seconds and returns one grounded move — reschedule, reset-gpu, reboot-node, rma, watch, or app-bug-not-hardware — from an NVIDIA Xid code or a pasted dmesg / nvidia-smi blob. The classification is deterministic: a bundled script does the mapping so the agent never guesses whether a card is dead, degraded, or fine.

Overview

A single bad GPU can kill a 64-GPU, multi-day training run — thousands of dollars and days of wall-clock gone — because one rank stalls the whole collective. The expensive mistakes are triage mistakes: RMAing a healthy card for an app bug, restarting a job onto a GPU whose memory error was uncontained, or manually uncordoning a node the lifecycle controller is trying to replace. This skill kills that ambiguity.

The decision logic is grounded in the NVIDIA Xid error catalog (https://docs.nvidia.com/deploy/xid-errors/) and CoreWeave's node-lifecycle / cordon behavior. The math-of-the-matter — which Xid means what, and how row-remapper state overrides it — lives in scripts/triage.py as a table the LLM does not get to re-litigate. Deep domain knowledge (the full code→action table, the row-remap decision, the cordon rules) loads from references/ on demand.

The headline is the Xid 94-vs-95 split. A contained memory error (94) cost you one job restart on a healthy node; an uncontained one (95) means the GPU could not isolate the fault and everything it touched is suspect. Getting that one bit wrong is the difference between a 30-second reschedule and a run that quietly trained on corrupt gradients. The script decides it; the skill never eyeballs it.

This skill is diagnostic, not destructive: it recommends the cordon / drain / reset / RMA next-step but its tools are scoped read-only (nvidia-smi -q, kubectl get, dmesg) — it never runs a reset, a reboot, or an uncordon itself.

Prerequisites

• The failure evidence. Either an Xid number, or a pasted dmesg / nvidia-smi dump. The skill works from a paste alone — no live cluster access required — which is the common case (an operator pastes what the run's logs showed).