cluster-network-topology

name

cluster-network-topology

description

Discover, diagram, and diagnose the RDMA/GPU network topology of a GPU cluster (rails, NICs, GID/RoCE config, GPU<->NIC PCIe affinity), determine whether the fabric supports cross-rail communication, and localize a failure to a fabric tier (NIC / leaf / spine) using only unprivileged host-side probes. Picks up where `mori check` stops: `mori check` answers "is this host configured correctly?" and drives its peer over SSH, while this answers "what is the fabric, and which tier broke?" from inside a scheduler allocation where SSH to compute nodes is unavailable. Handles IPv4-mapped AND IPv6-ULA RoCE addressing and classifies at BOTH the IP and RDMA layers — a fabric can be IP-routable cross-rail yet rail-only for RDMA. Includes the addressing-plan decode that recovers rail/leaf/pod grouping without switch access, spine enumeration that settles shared-vs-partitioned upper tiers from an unprivileged shell, a confound register for the measurement tools themselves, and an sbatch harness for multi-node tests under Slurm/Spur. Use when asked to "map the cluster", "draw how NICs are connected", "cross-rail diagram", "can we enable cross-rail RDMA", "get the switch topology", or to diagnose RDMA/RoCE QP-setup failures (e.g. rc=110 ETIMEDOUT on INIT->RTR), cross-rail unreachability, rail affinity, or EP / KV-transfer connectivity problems on AMD (ionic/bnxt/mlx5) or NVIDIA (mlx5) fabrics.

Cluster network topology

Companion to mori check (tools/env_check.sh) and the deploy-mori skill: those answer "is this host configured correctly?", this answers "what is the fabric, and which tier broke?". The scripts referenced throughout ship beside this file — rail_detect.sh, probe_topology.sh, xrail_worker.sh, xrail_worker.sbatch, xrail_matrix.sh, xrail_matrix.sbatch, make_diagrams.py, make_report.py.

rail_detect.sh is the shared rail-discovery library the other scripts source: it auto-detects each device's global RoCEv2 GID index and address (IPv4-mapped or IPv6 ULA/GUA, skipping fe80:: link-local), so the tools never disagree about the same machine. Run it alone to see what a node looks like: bash rail_detect.sh --dump.

A "rail-optimized" GPU cluster gives each GPU its own NIC, and each NIC lives on its own isolated L2/L3 domain (a "rail"). Rails may or may not be routable to each other:

• Full-mesh fabric: any NIC can reach any NIC (cross-rail works) at both IP and RDMA.
• Rail-only fabric: a NIC can only reach the same rail on other nodes; cross-rail is unroutable. Symptom: same-rail RDMA works, cross-rail QP INIT->RTR fails (ibv_modify_qp returns ETIMEDOUT/110, or ping shows 100% loss cross-rail).
• IP-routable but RDMA rail-only (seen in the wild): cross-rail ICMP/IP works (the rails are routed), but cross-rail RoCEv2 RDMA still fails. So do not classify from ping alone — confirm at the RDMA layer (Step 2).

The goal of this skill is to (1) discover the layout, (2) classify the fabric at both layers, (3) localize a failure to a tier and say what could change it, and (4) produce a diagram + report + guidance for RDMA workloads.

---

Measurement discipline (read before running anything)

Three rules, each learned by getting it wrong. They cost more time to re-do than to follow.

1. Every negative needs a positive control from the same tool, in the same job, at the same instant. A cross-rail FAIL means nothing on its own — the peer might be down, the server might not have bound, the GID index might be wrong. It means something next to a same-rail OK from the same binary, same allocation, same minute. Sweeps (service level, DSCP/traffic class, MTU) are the usual offenders: a sweep where every value fails and no control was run is indistinguishable from a broken harness and carries zero evidential weight. Build the control into the harness, not into a follow-up run.

2. Never let one tool be the sole source of a verdict. Tools embed assumptions in their address handles, and a fabric-shaped conclusion drawn from one binary is really a conclusion about that binary. See the confound register in Step 2c — the canonical case is ibv_rc_pingpong, which hardcodes hop_limit = 1 and never uses RDMA CM. Before reporting "the fabric cannot do X", reproduce X's failure with at least two tools that build their path differently (one hand-built AH, one RDMA CM).

3. Capture the cheap artifacts every run, whether or not you think you need them. ip -6 route show table all, ip -6 neigh, ip -o addr, /sys/class/net/*/statistics, and the tool inventory cost milliseconds and are the first things you will wish you had when the verdict is questioned weeks later. ip route is the single most often forgotten and most load-bearing artifact — it decides whether a rail plane is on-link (pure L2) or routed, which in turn decides whether the hop_limit confound is fatal or harmless. If you get one allocation, get the routes.

Corollary: record the confounds you did not close in the report itself. A verdict with a named, dated open question is usable; a verdict that quietly rests on an unexamined assumption is a liability.

---

Step 1 — Discover the layout

Prefer the bundled probe_topology.sh (auto-detects NICs, GIDs incl. IPv6-ULA rails, GPUs, GPU model, same-PCI-domain affinity; emits topo_report.<host>.txt plus a raw topology_<host>_node.auto.{mmd,dot}). Run it inside the container/host that owns the devices:

./probe_topology.sh                 # local node
PEER_IPS='<rail0_ip> <rail1_ip> ...' ./probe_topology.sh --peer <host>   # + IP reachability matrix
GID_INDEX=1 ./probe_topology.sh     # force a RoCE GID index (else auto)

--peer needs PEER_IPS (the peer's addresses in rail order) because deriving them would mean guessing the site's addressing plan; without it the section says so rather than printing an empty pass. It is IP only — prefer xrail_worker.sbatch, which discovers both sides itself and tests the RDMA layer too.

If doing it by hand, collect these facts per node:

Fact	Where
RDMA devices	ls /sys/class/infiniband (or ibv_devices)
Port state / link layer / rate	/sys/class/infiniband/<dev>/ports/1/{state,rate}, ibv_devinfo
RoCE GID + type + addr	/sys/class/infiniband/<dev>/ports/1/gids/<i> and .../gid_attrs/types/<i>. Pick the RoCEv2 global entry — this may be IPv4-mapped (::ffff:AABBCCDD, tail = rail IP) or a global IPv6, commonly a ULA fc00::/7 (each rail its own /64). Skip fe80:: link-local. Note the GID index (RDMA apps need it: NCCL_IB_GID_INDEX, MORI_IB_GID_INDEX).
netdev per NIC	.../gid_attrs/ndevs/<i> (e.g. enP2p0s9)
Rail IP (v4 or v6)	IPv4 from the ::ffff: tail; for IPv6 rails read ip -o -6 addr show <ndev> scope global. If a rail has no IPv4 and only a fe80:: GID, look again — its routable address is usually an IPv6 ULA at a higher GID index (this is easy to miss).
MTU per netdev	cat /sys/class/net/<ndev>/mtu. Rails are usually jumbo (9000); a mgmt NIC usually is not. A 1500-MTU path still works, so this is easy to miss while it quietly taxes throughput.
L3 shape	ip -6 route show table all, ip -4 route show table all, ip route show default, ip -6 neigh show. See Step 3a — this is what tells you whether a rail is on-link or routed.
NIC PCI bus + NUMA	readlink -f /sys/class/infiniband/<dev>/device; .../device/numa_node
Firmware + driver	cat /sys/class/infiniband/<dev>/fw_ver; modinfo <mod> vs /sys/module/<mod>/version. Compare as a tuple, not lexically — a larger trailing build number can hide a smaller patch number (1.117.1-a-63 is older than 1.117.5-a-58).
GPU PCI bus + model	rocm-smi --showbus / rocm-smi --showproductname (AMD, verify the actual model — don't infer from PCI DID) or nvidia-smi --query-gpu=index,pci.bus_id,name (NVIDIA)

GPU↔NIC affinity. PCI addresses are domain:bus:dev.func (e.g. 0002:00:01.0). First isolate the rail NICs — drop the management/front-end NIC(s): the one on the default route (ip route show default) and any RDMA device with no global address. This matters on single-PCI-domain boxes where mgmt NICs are interspersed among the rails (e.g. mlx5 eth0/eth1 sitting among rdma0..7); if you don't drop them the pairing shifts. Then pair within the same PCI domain by ordinal: sort the rail NICs and the GPUs in each domain by full PCI address and zip them (k-th GPU ↔ k-th NIC). A naive "nearest bus number" instead collapses on multi-domain boxes (the bus field is 00 for everything). On a rail-optimized box the two halves land on the two NUMA nodes / PCI domains (first-half GPUs with first-half NICs, second half with second half); on single-domain boxes take the ordinal over the rails.

Note whether the mgmt NIC is cross-socket for half the GPUs — it usually is, and it matters if you later fall back to it (Step 5).

Script portability. The probe must run where the devices live. Two gotchas the bundled script already handles, but watch for if you hand-roll it:

• No /dev/fd in some container / scheduler-step namespaces → bash process substitution (while read … < <(cmd)) fails with /dev/fd/63: No such file. Read from a temp file instead.
• Parse the full PCI address (domain included), per the affinity note above.

---

Step 2 — Classify the fabric (both layers)

Pick two nodes A and B. Test per rail, both layers, and as a full N×N matrix rather than a couple of spot checks — the shape of the matrix is the finding.

2a — The N×N matrix

For an N-rail node, run all N² source-rail × destination-rail combinations at both layers. What you are looking for is the pattern:

Matrix shape	Meaning
All N² pass	Full mesh.
Perfect diagonal (N pass, N²−N fail)	Rail-partitioned for that tool. The clean diagonal is itself the positive control — it proves the harness, the GID index, the ports and the peer are all fine.
Ragged / asymmetric	Not a fabric property. Suspect the harness, a wedged server, a per-device config difference, or a sick NIC. Re-run before interpreting.
Diagonal fails too	Wrong GID index, wrong port, peer down. Fix this before reading anything else.

IP layer (quick proxy). Bind to the rail's netdev. ping -I <ifname> sets SO_BINDTODEVICE, which genuinely forces egress out that device — so a cross-rail ping success is real evidence that L3 crosses planes. Caveat: -I only forces the outbound path. The return path is chosen by the far node and is not observed; a success therefore proves "there is a route out and some route back", not "the reverse traversed the same rail".

# rail-aligned: same subnet on both nodes  -> expect 0% loss
ping    -c2 -I <ethX_railN> <B_ip4_railN>
ping -6 -c2 -I <ethX_railN> <B_ip6_railN>
# cross-rail: source railN device to B's railM addr (M != N) -> loss if rail-only
ping -6 -c2 -I <ethX_railN> <B_ip6_railM>

Confirm the packets really left the rail NIC rather than leaking to the default route:

cat /sys/class/net/<ndev>/statistics/tx_packets     # before
ping -6 -c 20 -I <ndev> <peer cross-rail addr>
cat /sys/class/net/<ndev>/statistics/tx_packets     # after -> must rise by ~20

RDMA layer (authoritative). ICMP passing cross-rail does not prove RDMA works. Server on B, client on A, using the chosen RoCEv2 GID index (-g / -x):

# same-rail:  server & client both on rail 0
B$ ibv_rc_pingpong -d <dev0> -g <gid> -p 18500          # server
A$ ibv_rc_pingpong -d <dev0> -g <gid> -p 18500 <B_host> # client  -> expect success
# cross-rail: client on rail 0, server on rail 1
B$ ibv_rc_pingpong -d <dev1> -g <gid> -p 18501          # server
A$ ibv_rc_pingpong -d <dev0> -g <gid> -p 18501 <B_host> # client  -> hangs/times out if rail-only

The out-of-band handshake goes over the management network by hostname; the RDMA path follows -d/-g. Note ibv_rc_pingpong exits after one connection, so the server side must re-listen in a loop for a matrix run; the destination rail is selected by which server port you connect to, the source rail by the client's -d.

Classification:

• Same-rail RDMA OK and cross-rail RDMA fails ⇒ rail-only for RDMA (the important case) — regardless of whether ICMP crossed. But see 2c before writing it down.
• Cross-rail RDMA also OK ⇒ full-mesh.
• To see the real errno when a library asserts, enable the provider's debug (e.g. AMD ionic: IONIC_DEBUG=1 IONIC_DEBUG_FILE=/tmp/ionic_dbg; look for modify qp ... state 1 -> 2 rc <errno>, where state 1->2 = INIT->RTR).

2b — The paired IP-vs-RDMA control

Run both layers against the same node pair, in the same job, back to back. This is the single highest-value experiment in the whole skill, because the interesting fabrics are the ones where the two layers disagree, and a disagreement measured hours apart on different node pairs is not a disagreement — it is two unrelated observations.

Emit one row per (src rail, dst rail) with both verdicts side by side:

rail0 -> rail0 :  RDMA OK    IP OK
rail0 -> rail1 :  RDMA FAIL  IP OK      <- this row is the finding

2c — Confound register: the tool is part of the experiment

Before a matrix result becomes a fabric verdict, check what the tool did to the path:

Tool	Path construction	Hop limit	Confound
ibv_rc_pingpong	hand-built AH, TCP out-of-band	source says grh.hop_limit = 1	1 would permit zero router hops, which alone could reproduce a perfect diagonal with no fabric involvement. But the kernel overwrites it from the route, so in practice it is usually not in force — measured: pingpong succeeds across a 3-router path. Settle it by hop count (below), not by reading the source.
ib_write_bw / ib_write_lat (perftest)	hand-built AH, TCP out-of-band	sets a high hop limit	Not subject to the above, but still never consults RDMA CM. Good second opinion.
ib_write_bw -R, rping, ucmatose	RDMA CM (rdma_resolve_addr/rdma_resolve_route)	from the kernel route	Uses the same resolution path a real library uses. If cross-rail works here and not with a hand-built AH, the fabric is fine and your address handle was wrong.
ping -I	kernel route + SO_BINDTODEVICE	kernel default (64)	Egress forced, return path unobserved.

Closing the hop-limit confound cheaply. You do not need a hop-limit sweep, and a sweep is hard to run correctly anyway (a failed run leaves the peer's server mid-timeout, so the next cell fails for harness reasons and the column fills with noise). Instead count the router hops on the path the tool already succeeded on. If the same-rail run passes and mtr/traceroute shows that same-rail crosses one or more routers, then a hop limit of 1 was demonstrably never in force — whatever the source says — and it cannot explain the cross-rail failures. One mtr run retires the whole question.

Minimum bar for a "cross-rail is dead" verdict: the failure reproduces under at least two tools that build the path differently, each with a same-rail positive control in the same job, and ip route has been captured so the on-link-vs-routed question is settled. Prefer a second hand-built-AH tool over an RDMA CM tool if the CM tools cannot be made to source-bind — CM will not pick a cross-rail path on its own anyway (see 3c).

# RDMA CM re-test — cheap, one 2-node job, and it is the test most often skipped
B$ rping -s -a <B rail1 addr>
A$ rping -c -a <B rail1 addr> -C 10
A$ ib_write_bw -R <B rail1 addr>        # and a same-rail run as the control

2d — Running multi-node tests under a scheduler (Slurm / Spur)

You need coordinated processes on two different nodes. Pitfalls learned the hard way:

• Interactive srun --overlap from the login node lands on the first node only (even with -w / --nodelist / -N2 -n2), and direct SSH to compute nodes is often blocked (publickey-only, key not installed). So drive it from a batch job, whose steps do spread across the allocation. This is also why mori check, which drives its mesh over SSH, cannot be used directly on such clusters.
• Spur-like schedulers: srun may be non-blocking (returns on dispatch, so the batch script must wait on its own sentinel files or it falls off the end and kills the workers); srun may not propagate the submitter's environment (pass knobs via files in a shared run dir); scontrol show hostnames may be unsupported and --ntasks-per-node rejected. Don't expand the nodelist — where srun exists, launch srun -N2 -n2 --overlap; either way have the workers self-organize by $(hostname) (sort the names; lower = tester A, higher = target B), which is what makes the same worker correct under both fan-out models below. The batch script may run from a spool copy, so use $SLURM_SUBMIT_DIR, not $BASH_SOURCE.