coreweave-fabric-diagnostics

name	coreweave-fabric-diagnostics
description	Diagnose the most expensive silent failure on a CoreWeave multi-node GPU job: GPUDirect RDMA falling back from InfiniBand to TCP. When NCCL drops from NET/IB to NET/Socket, collectives keep running with NO error but throughput collapses (commonly 5-20x slower) while every GPU still bills at full rate — 5x the GPU bill for the same work, invisibly. Paste an NCCL_DEBUG=INFO log (and/or a pod-spec, ibstat, or all_reduce_perf output) and the bundled deterministic script verdicts whether RDMA is actually engaged, which of the three required conditions is missing, and the fix. Use when multi-node training is slow, when checking whether RDMA/InfiniBand is engaged, or when all-reduce bandwidth looks low. Trigger with "coreweave slow training", "is RDMA working", "NCCL fell back to TCP", "NET/Socket", "GPUDirect RDMA", "infiniband not used", "multi-node training slow".
allowed-tools	Read, Write, Edit, Glob, Bash(kubectl get:*), Bash(python3:*)
version	1.11.0
author	Jeremy Longshore <[e-mail]>
license	MIT
compatibility	Designed for Claude Code, also compatible with Codex
tags	["saas","coreweave","gpu-cloud","rdma","nccl"]

CoreWeave Fabric Diagnostics

Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Detects when a CoreWeave multi-node GPU job has silently fallen off the InfiniBand fabric onto TCP — the failure that makes distributed training run at a fraction of the hardware's speed while every GPU keeps billing at the full rate — and gives the exact fix.

Overview

On a CoreWeave multi-node job, NCCL should carry collectives over InfiniBand with GPUDirect RDMA (NET/IB). If any one of three conditions is missing, NCCL silently falls back to TCP sockets (NET/Socket): the job still runs, still converges, and raises no error — but all-reduce throughput collapses (commonly cited as 5-20x slower [nccl]) because it now crosses the Ethernet control plane instead of the 400 Gb/s-class fabric. You keep paying full GPU rate for a multi-node run that performs like a badly-connected one. This is the single highest-dollar invisible failure on the platform, and nothing in the default output flags it.

The diagnosis is deterministic: the bundled scripts/fabric-check.py greps the pasted NCCL_DEBUG=INFO log for the decisive Using network line (and NET/IB vs NET/Socket), parses the pod-spec's resources block for the RDMA device request, reads ibstat port state, and echoes any all_reduce_perf bus bandwidth — then emits a VERDICT with the rdma_engaged/transport call, the missing conditions, and the fix. The LLM never eyeballs which transport is in use; the script decides. Deep grounding lives in references/, loaded only when a leg of the diagnosis needs it.

Prerequisites

• An NCCL_DEBUG=INFO log from the actual run — the primary signal. Re-run the job (or one rank) with NCCL_DEBUG=INFO set and capture stderr. The decisive line is Using network IB (good) vs Using network Socket (the fallback). This is the one input the skill really needs; everything else corroborates.
• Optional, for a full diagnosis: the pod/job spec (kubectl get pod NAME -o yaml) to check the RDMA device request; output from the node for port health; and results from CoreWeave's to measure bus bandwidth.