cluster-network-topology

# Cluster network topology Companion to `mori check` (`tools/env_check.sh`) and the `deploy-mori` skill: those answer "is this host configured correctly?", this answers "what is the fabric, and which tier broke?". The scripts referenced throughout ship beside this file — `rail_detect.sh`, `probe_topology.sh`, `xrail_worker.sh`, `xrail_worker.sbatch`, `xrail_matrix.sh`, `xrail_matrix.sbatch`, `make_diagrams.py`, `make_report.py`. `rail_detect.sh` is the shared rail-discovery library the other scripts source: it auto-detects each device's global RoCEv2 GID index and address (IPv4-mapped or IPv6 ULA/GUA, skipping `fe80::` link-local), so the tools never disagree about the same machine. Run it alone to see what a node looks like: `bash rail_detect.sh --dump`. A "rail-optimized" GPU cluster gives each GPU its own NIC, and each NIC lives on its own isolated L2/L3 domain (a "rail"). Rails may or may not be routable to each other: - **Full-mesh fabric**: any NIC can reach any NIC (cross-rail works) at both IP and RDMA. - **Rail-only fabric**: a NIC can only reach the *same* rail on other nodes; cross-rail is unroutable. Symptom: same-rail RDMA works, cross-rail QP `INIT->RTR` fails (`ibv_modify_qp` returns `ETIMEDOUT`/110, or `ping` shows 100% loss cross-rail). - **IP-routable but RDMA rail-only** (seen in the wild): cross-rail *ICMP/IP* works (the rails are routed), but cross-rail *RoCEv2 RDMA* still fails. So **do not classify from ping alone** — confirm at the RDMA layer (Step 2). The goal of this skill is to (1) discover the layout, (2) classify the fabric at both layers, (3) **localize** a failure to a tier and say what could change it, and (4) produce a diagram + report + guidance for RDMA workloads. --- ## Measurement discipline (read before running anything) Three rules, each learned by getting it wrong. They cost more time to re-do than to follow. **1. Every negative needs a positive control from the same tool, in the same job, at the same instant.** A cross-rail `FAIL` means nothing on its own — the peer might be down, the server might not have bound, the GID index might be wrong. It means something next to a same-rail `OK` from the same binary, same allocation, same minute. Sweeps (service level, DSCP/traffic class, MTU) are the usual offenders: a sweep where *every* value fails and no control was run is indistinguishable from a broken harness and carries **zero** evidential weight. Build the control into the harness, not into a follow-up run. **2. Never let one tool be the sole source of a verdict.** Tools embed assumptions in their address handles, and a fabric-shaped conclusion drawn from one binary is really a conclusion about that binary. See the confound register in Step 2c — the canonical case is `ibv_rc_pingpong`, which hardcodes `hop_limit = 1` and never uses RDMA CM. Before reporting "the fabric cannot do X", reproduce X's failure with **at least two tools that build their path differently** (one hand-built AH, one RDMA CM). **3. Capture the cheap artifacts every run, whether or not you think you need them.** `ip -6 route show table all`, `ip -6 neigh`, `ip -o addr`, `/sys/class/net/*/statistics`, and the tool inventory cost milliseconds and are the first things you will wish you had when the verdict is questioned weeks later. **`ip route` is the single most often forgotten and most load-bearing artifact** — it decides whether a rail plane is on-link (pure L2) or routed, which in turn decides whether the `hop_limit` confound is fatal or harmless. If you get one allocation, get the routes. Corollary: **record the confounds you did not close** in the report itself. A verdict with a named, dated open question is usable; a verdict that quietly rests on an unexamined assumption is a liability. --- ## Step 1 — Discover the layout Prefer the bundled `probe_topology.sh` (auto-detects NICs, GIDs incl. IPv6-ULA rails, GPUs, GPU model, same-PCI-domain affinity; emits `topo_report.<host>.txt` plus a raw `topology_<host>_node.auto.{mmd,dot}`). Run it inside the container/host that owns the devices: ```bash ./probe_topology.sh # local node PEER_IPS='<rail0_ip> <rail1_ip> ...' ./probe_topology.sh --peer <host> # + IP reachability matrix GID_INDEX=1 ./probe_topology.sh # force a RoCE GID index (else auto) ``` `--peer` needs `PEER_IPS` (the peer's addresses in rail order) because deriving them would mean guessing the site's addressing plan; without it the section says so rather than printing an empty pass. It is **IP only** — prefer `xrail_worker.sbatch`, which discovers both sides itself and tests the RDMA layer too. If doing it by hand, collect these facts per node: | Fact | Where | |---|---| | RDMA devices | `ls /sys/class/infiniband` (or `ibv_devices`) | | Port state / link layer / rate | `/sys/class/infiniband/<dev>/ports/1/{state,rate}`, `ibv_devinfo` | | **RoCE GID + type + addr** | `/sys/class/infiniband/<dev>/ports/1/gids/<i>` and `.../gid_attrs/types/<i>`. Pick the **RoCEv2 global** entry — this may be IPv4-mapped (`::ffff:AABBCCDD`, tail = rail IP) **or a global IPv6, commonly a ULA `fc00::/7`** (each rail its own `/64`). **Skip `fe80::` link-local.** Note the GID **index** (RDMA apps need it: `NCCL_IB_GID_INDEX`, `MORI_IB_GID_INDEX`). | | netdev per NIC | `.../gid_attrs/ndevs/<i>` (e.g. `enP2p0s9`) | | Rail IP (v4 or v6) | IPv4 from the `::ffff:` tail; **for IPv6 rails** read `ip -o -6 addr show <ndev> scope global`. If a rail has **no IPv4 and only a `fe80::` GID**, look again — its routable address is usually an IPv6 **ULA at a higher GID index** (this is easy to miss). | | **MTU per netdev** | `cat /sys/class/net/<ndev>/mtu`. Rails are usually jumbo (9000); a mgmt NIC usually is not. A 1500-MTU path still *works*, so this is easy to miss while it quietly taxes throughput. | | **L3 shape** | `ip -6 route show table all`, `ip -4 route show table all`, `ip route show default`, `ip -6 neigh show`. See Step 3a — this is what tells you whether a rail is on-link or routed. | | NIC PCI bus + NUMA | `readlink -f /sys/class/infiniband/<dev>/device`; `.../device/numa_node` | | **Firmware + driver** | `cat /sys/class/infiniband/<dev>/fw_ver`; `modinfo <mod>` vs `/sys/module/<mod>/version`. Compare as a **tuple**, not lexically — a larger trailing build number can hide a smaller patch number (`1.117.1-a-63` is *older* than `1.117.5-a-58`). | | GPU PCI bus + model | `rocm-smi --showbus` / `rocm-smi --showproductname` (AMD, verify the actual model — don't infer from PCI DID) or `nvidia-smi --query-gpu=index,pci.bus_id,name` (NVIDIA) | **GPU↔NIC affinity.** PCI addresses are `domain:bus:dev.func` (e.g. `0002:00:01.0`). First **isolate the rail NICs** — drop the management/front-end NIC(s): the one on the default route (`ip route show default`) and any RDMA device with no global address. This matters on **single-PCI-domain** boxes where mgmt NICs are interspersed among the rails (e.g. mlx5 `eth0`/`eth1` sitting among `rdma0..7`); if you don't drop them the pairing shifts. Then **pair within the same PCI domain by ordinal**: sort the rail NICs and the GPUs in each domain by full PCI address and zip them (k-th GPU ↔ k-th NIC). A naive "nearest bus number" instead collapses on multi-domain boxes (the bus field is `00` for everything). On a rail-optimized box the two halves land on the two NUMA nodes / PCI domains (first-half GPUs with first-half NICs, second half with second half); on single-domain boxes take the ordinal over the rails. Note whether the **mgmt NIC is cross-socket** for half the GPUs — it usually is, and it matters if you later fall back to it (Step 5). **Script portability.** The probe must run where the devices live. Two gotchas the bundled script already handles, but watch for if you hand-roll it: - **No `/dev/fd`** in some container / scheduler-step namespaces → bash **process substitution** (`while read … < <(cmd)`) fails with `/dev/fd/63: No such file`. Read from a temp file instead. - Parse the **full** PCI address (domain included), per the affinity note above. --- ## Step 2 — Classify the fabric (both layers) Pick two nodes A and B. Test **per rail**, **both layers**, and **as a full N×N matrix** rather than a couple of spot checks — the shape of the matrix is the finding. ### 2a — The N×N matrix For an N-rail node, run all N² source-rail × destination-rail combinations at both layers. What you are looking for is the *pattern*: | Matrix shape | Meaning | |---|---| | All N² pass | Full mesh. | | **Perfect diagonal** (N pass, N²−N fail) | Rail-partitioned *for that tool*. The clean diagonal is itself the positive control — it proves the harness, the GID index, the ports and the peer are all fine. | | Ragged / asymmetric | Not a fabric property. Suspect the harness, a wedged server, a per-device config difference, or a sick NIC. Re-run before interpreting. | | Diagonal fails too | Wrong GID index, wrong port, peer down. Fix this before reading anything else. | **IP layer (quick proxy).** Bind to the rail's netdev. `ping -I <ifname>` sets `SO_BINDTODEVICE`, which genuinely forces egress out that device — so a cross-rail ping success is real evidence that L3 crosses planes. **Caveat: `-I` only forces the *outbound* path.** The return path is chosen by the far node and is not observed; a success therefore proves "there is a route out and *some* route back", not "the reverse traversed the same rail". ```bash # rail-aligned: same subnet on both nodes -> expect 0% loss ping -c2 -I <ethX_railN> <B_ip4_railN> ping -6 -c2 -I <ethX_railN> <B_ip6_railN> # cross-rail: source railN device to B's railM addr (M != N) -> loss if rail-only ping -6 -c2 -I <ethX_railN> <B_ip6_railM> ``` Confirm the packets really left the rail NIC rather than leaking to the default route: ```bash cat /sys/class/net/<ndev>/statistics/tx_packets # before ping -6 -c 20 -I <ndev> <peer cross-rail addr> cat /sys/class/net/<ndev>/statistics/tx_packets # after -> must rise by ~20 ``` **RDMA layer (authoritative).** ICMP passing cross-rail does **not** prove RDMA works. Server on B, client on A, using the chosen RoCEv2 GID index (`-g` / `-x`): ```bash # same-rail: server & client both on rail 0 B$ ibv_rc_pingpong -d <dev0> -g <gid> -p 18500 # server A$ ibv_rc_pingpong -d <dev0> -g <gid> -p 18500 <B_host> # client -> expect success # cross-rail: client on rail 0, server on rail 1 B$ ibv_rc_pingpong -d <dev1> -g <gid> -p 18501 # server A$ ibv_rc_pingpong -d <dev0> -g <gid> -p 18501 <B_host> # client -> hangs/times out if rail-only ``` The out-of-band handshake goes over the management network by hostname; the RDMA path follows `-d`/`-g`. Note `ibv_rc_pingpong` **exits after one connection**, so the server side must re-listen in a loop for a matrix run; the destination rail is selected by which server port you connect to, the source rail by the client's `-d`. Classification: - Same-rail RDMA OK **and** cross-rail RDMA fails ⇒ **rail-only for RDMA** (the important case) — regardless of whether ICMP crossed. But see 2c before writing it down. - Cross-rail RDMA also OK ⇒ full-mesh. - To see the real errno when a library asserts, enable the provider's debug (e.g. AMD ionic: `IONIC_DEBUG=1 IONIC_DEBUG_FILE=/tmp/ionic_dbg`; look for `modify qp ... state 1 -> 2 rc <errno>`, where `state 1->2` = INIT->RTR). ### 2b — The paired IP-vs-RDMA control Run **both layers against the same node pair, in the same job, back to back**. This is the single highest-value experiment in the whole skill, because the interesting fabrics are the ones where the two layers disagree, and a disagreement measured hours apart on different node pairs is not a disagreement — it is two unrelated observations. Emit one row per (src rail, dst rail) with both verdicts side by side: ``` rail0 -> rail0 : RDMA OK IP OK rail0 -> rail1 : RDMA FAIL IP OK <- this row is the finding ``` ### 2c — Confound register: the tool is part of the experiment Before a matrix result becomes a fabric verdict, check what the tool did to the path: | Tool | Path construction | Hop limit | Confound | |---|---|---|---| | `ibv_rc_pingpong` | hand-built AH, TCP out-of-band | source says **`grh.hop_limit = 1`** | 1 would permit **zero** router hops, which alone could reproduce a perfect diagonal with no fabric involvement. But the kernel overwrites it from the route, so in practice it is usually *not* in force — measured: pingpong succeeds across a 3-router path. Settle it by hop count (below), not by reading the source. | | `ib_write_bw` / `ib_write_lat` (perftest) | hand-built AH, TCP out-of-band | sets a high hop limit | Not subject to the above, but still never consults RDMA CM. Good second opinion. | | `ib_write_bw -R`, `rping`, `ucmatose` | **RDMA CM** (`rdma_resolve_addr`/`rdma_resolve_route`) | from the kernel route | Uses the same resolution path a real library uses. If cross-rail works here and not with a hand-built AH, the fabric is fine and your address handle was wrong. | | `ping -I` | kernel route + `SO_BINDTODEVICE` | kernel default (64) | Egress forced, **return path unobserved**. | **Closing the hop-limit confound cheaply.** You do not need a hop-limit sweep, and a sweep is hard to run correctly anyway (a failed run leaves the peer's server mid-timeout, so the next cell fails for harness reasons and the column fills with noise). Instead **count the router hops on the path the tool already succeeded on**. If the *same-rail* run passes and `mtr`/`traceroute` shows that same-rail crosses one or more routers, then a hop limit of 1 was demonstrably never in force — whatever the source says — and it cannot explain the cross-rail failures. One `mtr` run retires the whole question. **Minimum bar for a "cross-rail is dead" verdict:** the failure reproduces under at least two tools that build the path differently, each with a same-rail positive control in the same job, and `ip route` has been captured so the on-link-vs-routed question is settled. Prefer a second hand-built-AH tool over an RDMA CM tool if the CM tools cannot be made to source-bind — CM will not pick a cross-rail path on its own anyway (see 3c). ```bash # RDMA CM re-test — cheap, one 2-node job, and it is the test most often skipped B$ rping -s -a <B rail1 addr> A$ rping -c -a <B rail1 addr> -C 10 A$ ib_write_bw -R <B rail1 addr> # and a same-rail run as the control ``` ### 2d — Running multi-node tests under a scheduler (Slurm / Spur) You need coordinated processes on **two different** nodes. Pitfalls learned the hard way: - **Interactive `srun --overlap` from the login node lands on the first node only** (even with `-w` / `--nodelist` / `-N2 -n2`), and **direct SSH to compute nodes is often blocked** (publickey-only, key not installed). So drive it from a **batch job**, whose steps *do* spread across the allocation. This is also why `mori check`, which drives its mesh over SSH, cannot be used directly on such clusters. - **Spur-like schedulers:** `srun` may be **non-blocking** (returns on dispatch, so the batch script must wait on its own sentinel files or it falls off the end and kills the workers); `srun` may **not propagate the submitter's environment** (pass knobs via files in a shared run dir); `scontrol show hostnames` may be unsupported and `--ntasks-per-node` rejected. Don't expand the nodelist — where `srun` exists, launch `srun -N2 -n2 --overlap`; either way have the workers **self-organize by `$(hostname)`** (sort the names; lower = tester A, higher = target B), which is what makes the same worker correct under both fan-out models below. The batch script may run from a **spool copy**, so use `$SLURM_SUBMIT_DIR`, not `$BASH_SOURCE`.