Cluster Network Topology

Cluster network topology

Companion to mori check (tools/env_check.sh) and the deploy-mori skill: those answer "is this host configured correctly?", this answers "what is the fabric, and which tier broke?". The scripts referenced throughout ship beside this file — rail_detect.sh, probe_topology.sh, xrail_worker.sh, xrail_worker.sbatch, xrail_matrix.sh, xrail_matrix.sbatch, make_diagrams.py, make_report.py.

rail_detect.sh is the shared rail-discovery library the other scripts source: it auto-detects each device's global RoCEv2 GID index and address (IPv4-mapped or IPv6 ULA/GUA, skipping fe80:: link-local), so the tools never disagree about the same machine. Run it alone to see what a node looks like: bash rail_detect.sh --dump.

A "rail-optimized" GPU cluster gives each GPU its own NIC, and each NIC lives on its own isolated L2/L3 domain (a "rail"). Rails may or may not be routable to each other:

• Full-mesh fabric: any NIC can reach any NIC (cross-rail works) at both IP and RDMA.
• Rail-only fabric: a NIC can only reach the same rail on other nodes; cross-rail is unroutable. Symptom: same-rail RDMA works, cross-rail QP INIT->RTR fails (ibv_modify_qp returns ETIMEDOUT/110, or ping shows 100% loss cross-rail).
• IP-routable but RDMA rail-only (seen in the wild): cross-rail ICMP/IP works (the rails are routed), but cross-rail RoCEv2 RDMA still fails. So do not classify from ping alone — confirm at the RDMA layer (Step 2).

The goal of this skill is to (1) discover the layout, (2) classify the fabric at both layers, (3) localize a failure to a tier and say what could change it, and (4) produce a diagram + report + guidance for RDMA workloads.

Measurement discipline (read before running anything)

Three rules, each learned by getting it wrong. They cost more time to re-do than to follow.

1. Every negative needs a positive control from the same tool, in the same job, at the same instant. A cross-rail FAIL means nothing on its own — the peer might be down, the server might not have bound, the GID index might be wrong. It means something next to a same-rail OK from the same binary, same allocation, same minute. Sweeps (service level, DSCP/traffic class, MTU) are the usual offenders: a sweep where every value fails and no control was run is indistinguishable from a broken harness and carries zero evidential weight. Build the control into the harness, not into a follow-up run.

2. Never let one tool be the sole source of a verdict. Tools embed assumptions in their address handles, and a fabric-shaped conclusion drawn from one binary is really a conclusion about that binary. See the confound register in Step 2c — the canonical case is ibv_rc_pingpong, which hardcodes hop_limit = 1 and never uses RDMA CM. Before reporting "the fabric cannot do X", reproduce X's failure with at least two tools that build their path differently (one hand-built AH, one RDMA CM).

3. Capture the cheap artifacts every run, whether or not you think you need them. ip -6 route show table all, ip -6 neigh, ip -o addr, /sys/class/net/*/statistics, and the tool inventory cost milliseconds and are the first things you will wish you had when the verdict is questioned weeks later. ip route is the single most often forgotten and most load-bearing artifact — it decides whether a rail plane is on-link (pure L2) or routed, which in turn decides whether the hop_limit confound is fatal or harmless. If you get one allocation, get the routes.

Corollary: record the confounds you did not close in the report itself. A verdict with a named, dated open question is usable; a verdict that quietly rests on an unexamined assumption is a liability.

Step 1 — Discover the layout

Prefer the bundled probe_topology.sh (auto-detects NICs, GIDs incl. IPv6-ULA rails, GPUs, GPU model, same-PCI-domain affinity; emits topo_report.<host>.txt plus a raw topology_<host>_node.auto.{mmd,dot}). Run it inside the container/host that owns the devices:

./probe_topology.sh                 # local node
PEER_IPS='<rail0_ip> <rail1_ip> ...' ./probe_topology.sh --peer <host>   # + IP reachability matrix
GID_INDEX=1 ./probe_topology.sh     # force a RoCE GID index (else auto)

--peer needs PEER_IPS (the peer's addresses in rail order) because deriving them would mean guessing the site's addressing plan; without it the section says so rather than printing an empty pass. It is IP only — prefer xrail_worker.sbatch, which discovers both sides itself and tests the RDMA layer too.

If doing it by hand, collect these facts per node:

GPU↔NIC affinity. PCI addresses are domain:bus:dev.func (e.g. 0002:00:01.0). First isolate the rail NICs — drop the management/front-end NIC(s): the one on the default route (ip route show default) and any RDMA device with no global address. This matters on single-PCI-domain boxes where mgmt NICs are interspersed among the rails (e.g. mlx5 eth0/eth1 sitting among rdma0..7); if you don't drop them the pairing shifts. Then pair within the same PCI domain by ordinal: sort the rail NICs and the GPUs in each domain by full PCI address and zip them (k-th GPU ↔ k-th NIC). A naive "nearest bus number" instead collapses on multi-domain boxes (the bus field is 00 for everything). On a rail-optimized box the two halves land on the two NUMA nodes / PCI domains (first-half GPUs with first-half NICs, second half with second half); on single-domain boxes take the ordinal over the rails.

Note whether the mgmt NIC is cross-socket for half the GPUs — it usually is, and it matters if you later fall back to it (Step 5).

Script portability. The probe must run where the devices live. Two gotchas the bundled script already handles, but watch for if you hand-roll it:

• No /dev/fd in some container / scheduler-step namespaces → bash process substitution (while read … < <(cmd)) fails with /dev/fd/63: No such file. Read from a temp file instead.
• Parse the full PCI address (domain included), per the affinity note above.

Step 2 — Classify the fabric (both layers)

Pick two nodes A and B. Test per rail, both layers, and as a full N×N matrix rather than a couple of spot checks — the shape of the matrix is the finding.

2a — The N×N matrix

For an N-rail node, run all N² source-rail × destination-rail combinations at both layers. What you are looking for is the pattern:

IP layer (quick proxy). Bind to the rail's netdev. ping -I <ifname> sets SO_BINDTODEVICE, which genuinely forces egress out that device — so a cross-rail ping success is real evidence that L3 crosses planes. Caveat: -I only forces the outbound path. The return path is chosen by the far node and is not observed; a success therefore proves "there is a route out and some route back", not "the reverse traversed the same rail".

# rail-aligned: same subnet on both nodes  -> expect 0% loss
ping    -c2 -I <ethX_railN> <B_ip4_railN>
ping -6 -c2 -I <ethX_railN> <B_ip6_railN>
# cross-rail: source railN device to B's railM addr (M != N) -> loss if rail-only
ping -6 -c2 -I <ethX_railN> <B_ip6_railM>

Confirm the packets really left the rail NIC rather than leaking to the default route:

cat /sys/class/net/<ndev>/statistics/tx_packets     # before
ping -6 -c 20 -I <ndev> <peer cross-rail addr>
cat /sys/class/net/<ndev>/statistics/tx_packets     # after -> must rise by ~20

RDMA layer (authoritative). ICMP passing cross-rail does not prove RDMA works. Server on B, client on A, using the chosen RoCEv2 GID index (-g / -x):

# same-rail:  server & client both on rail 0
B$ ibv_rc_pingpong -d <dev0> -g <gid> -p 18500          # server
A$ ibv_rc_pingpong -d <dev0> -g <gid> -p 18500 <B_host> # client  -> expect success
# cross-rail: client on rail 0, server on rail 1
B$ ibv_rc_pingpong -d <dev1> -g <gid> -p 18501          # server
A$ ibv_rc_pingpong -d <dev0> -g <gid> -p 18501 <B_host> # client  -> hangs/times out if rail-only

The out-of-band handshake goes over the management network by hostname; the RDMA path follows -d/-g. Note ibv_rc_pingpong exits after one connection, so the server side must re-listen in a loop for a matrix run; the destination rail is selected by which server port you connect to, the source rail by the client's -d.

Classification:

• Same-rail RDMA OK and cross-rail RDMA fails ⇒ rail-only for RDMA (the important case) — regardless of whether ICMP crossed. But see 2c before writing it down.
• Cross-rail RDMA also OK ⇒ full-mesh.
• To see the real errno when a library asserts, enable the provider's debug (e.g. AMD ionic: IONIC_DEBUG=1 IONIC_DEBUG_FILE=/tmp/ionic_dbg; look for modify qp ... state 1 -> 2 rc <errno>, where state 1->2 = INIT->RTR).

2b — The paired IP-vs-RDMA control

Run both layers against the same node pair, in the same job, back to back. This is the single highest-value experiment in the whole skill, because the interesting fabrics are the ones where the two layers disagree, and a disagreement measured hours apart on different node pairs is not a disagreement — it is two unrelated observations.

Emit one row per (src rail, dst rail) with both verdicts side by side:

rail0 -> rail0 :  RDMA OK    IP OK
rail0 -> rail1 :  RDMA FAIL  IP OK      <- this row is the finding

2c — Confound register: the tool is part of the experiment

Before a matrix result becomes a fabric verdict, check what the tool did to the path:

Closing the hop-limit confound cheaply. You do not need a hop-limit sweep, and a sweep is hard to run correctly anyway (a failed run leaves the peer's server mid-timeout, so the next cell fails for harness reasons and the column fills with noise). Instead count the router hops on the path the tool already succeeded on. If the same-rail run passes and mtr/traceroute shows that same-rail crosses one or more routers, then a hop limit of 1 was demonstrably never in force — whatever the source says — and it cannot explain the cross-rail failures. One mtr run retires the whole question.

Minimum bar for a "cross-rail is dead" verdict: the failure reproduces under at least two tools that build the path differently, each with a same-rail positive control in the same job, and ip route has been captured so the on-link-vs-routed question is settled. Prefer a second hand-built-AH tool over an RDMA CM tool if the CM tools cannot be made to source-bind — CM will not pick a cross-rail path on its own anyway (see 3c).

# RDMA CM re-test — cheap, one 2-node job, and it is the test most often skipped
B$ rping -s -a <B rail1 addr>
A$ rping -c -a <B rail1 addr> -C 10
A$ ib_write_bw -R <B rail1 addr>        # and a same-rail run as the control

2d — Running multi-node tests under a scheduler (Slurm / Spur)

You need coordinated processes on two different nodes. Pitfalls learned the hard way:

• Interactive srun --overlap from the login node lands on the first node only (even with -w / --nodelist / -N2 -n2), and direct SSH to compute nodes is often blocked (publickey-only, key not installed). So drive it from a batch job, whose steps do spread across the allocation. This is also why mori check, which drives its mesh over SSH, cannot be used directly on such clusters.

• Spur-like schedulers: srun may be non-blocking (returns on dispatch, so the batch script must wait on its own sentinel files or it falls off the end and kills the workers); srun may not propagate the submitter's environment (pass knobs via files in a shared run dir); scontrol show hostnames may be unsupported and --ntasks-per-node rejected. Don't expand the nodelist — where srun exists, launch srun -N2 -n2 --overlap; either way have the workers self-organize by $(hostname) (sort the names; lower = tester A, higher = target B), which is what makes the same worker correct under both fan-out models below. The batch script may run from a spool copy, so use $SLURM_SUBMIT_DIR, not $BASH_SOURCE.

• On Spur, srun often does not exist on the compute nodes at all — measured absent on two unrelated Spur clusters (0.7.0 and 0.10.0), where it is a login-node client only and /opt/spur holds nothing but hooks/prolog.sh/epilog.sh. A batch body doing srun -N2 -n2 --overlap there dies with command not found. Instead the batch body itself is dispatched to every allocated node, which is the exact inverse of Slurm, where it runs on the first node only. The bundled wrappers branch on command -v srun and run the worker in place when it is missing, so one file covers both models.

  Determine the fan-out model empirically — do not infer it from the Spur version. Measured: Spur 0.7.0 fanned a plain sbatch <script> out to both nodes, while Spur 0.10.0 ran the same submission on one node and needed --wrap to spread. One 10-second job settles it, and the failure mode it prevents (a silent single-node run) is expensive:

  sbatch -N2 -n2 -t 00:02:00 --wrap='echo $(hostname) >> ~/fanout.txt'
  # 2 hostnames -> the wrapped command fans out. Now repeat with your real script as a
  # FILE: if that yields only 1 hostname, deliver the per-node body via --wrap instead:
  sbatch -N2 -n2 --gres=gpu:8 --wrap="bash \$HOME/railcheck/xrail_worker.sbatch"
  

  Two further traps in that same shell: the batch body is not a login shell, so the scheduler's PATH and controller address from /etc/profile.d are missing — yet sourcing those drop-ins under set -u aborts the job, because they are not written to be -u safe (wrap the sourcing in set +u / set -u). And squeue -j $JOB run inside its own job may return nothing forever, so a "wait until RUNNING" loop needs an escape hatch or it burns its entire budget before doing any work.

• Files on shared storage can read back empty on the peer node. Never silently default on an empty read — it desynchronizes the two sides asymmetrically and produces a plausible-looking wrong result. Retry, then fail loudly. The same rule covers peer discovery: under per-node dispatch the peer can start a minute or more after you, so wait generously (PEER_WAIT, default 180 s) and abort if only one host registered. Defaulting the target to your own hostname yields a loopback run that looks like a clean pass while answering an entirely different question.

• Avoid write races: only the tester writes results, guarded by an atomic mkdir <lock> (a scheduler may also spawn the same wrapper twice on one node); run the RDMA server in a re-listen loop and retry the client a few times (a first attempt can fire before the server is up). Retry only the "server not ready" race — a genuinely unreachable pair connects on TCP and then times out, and must not be retried away.

• Scheduler flakiness: jobs may land in JobHoldMaxRequeue (scontrol release <jid>) or hit transient JobLaunchFailure. If multi-node dispatch keeps failing, fall back to per-node jobs pinned with -w <node> rather than waiting it out. Check the QoS priority before blaming scarcity — a low-priority "burst"/scavenger QoS makes jobs sit PENDING with Reason=None for hours next to idle nodes, which looks exactly like a wedged scheduler.

Use the bundled xrail_matrix.sbatch + xrail_matrix.sh — these are cluster-agnostic: the worker auto-detects RoCE rail devices, each device's global GID index (IPv4-mapped or IPv6 ULA/GUA, skipping fe80::) and address, drops the management NIC (the one on the default route), and picks the ping family per destination automatically (ping -4 / ping -6, falling back to a separate ping6 binary on iputils too old to accept -6). Nothing site-specific is baked in — pass partition/account/qos/gres on the command line:

sbatch -p <partition> -A <acct> --qos=<qos> --gres=gpu:8 xrail_matrix.sbatch
# if the job is held: scontrol release <jid>

Two workers ship, both driven the same way and both writing the same output folder:

Start with xrail_worker.sbatch. Reach for xrail_matrix.sbatch once Step 2 says cross-rail fails, since the sweeps take much longer.

All artifacts land in a per-run output folder: <XRAIL_OUT>/job-<jid>/ (default XRAIL_OUT=<submit_dir>/xrail-output) — containing result.txt, topo_report.<host>.txt, routes.<host>, run.log (or run.log.<host> when every node runs the body), and the host.*/addrs.* coordination files. Optional env overrides (export before sbatch):

Rail index = position of the device in the sort -V order of /sys/class/infiniband (consistent across homogeneous nodes).

result.txt grammar. make_report.py parses this file, so a hand-written or site-modified worker must emit the same shape. Everything else in the file is free text and ignored; only these three line forms are read:

same-rail  A.<dev> -> B.rail<N> : REACHABLE|UNREACHABLE      # RDMA, one line
cross-rail A.<dev> -> B.rail<M> : REACHABLE|UNREACHABLE|SKIPPED
rail<N>(<ndev>)  rail<M>  OK|FAIL  same-rail|cross-rail      # IP, one line per pair

Rules the parser actually enforces:

• The RDMA lines are matched by the leading token (same-rail / cross-rail, case-insensitive) plus a : somewhere on the line. UNREACHABLE is tested before REACHABLE, so the substring overlap is safe. SKIPPED on the cross-rail line means "not attempted" and renders as -, not as a failure.
• The IP rows must match ^rail\d+\S*\s+rail\d+\s+(\S+)\s+(same-rail|cross-rail) — the (<ndev>) suffix on the source rail is optional, column padding is free, and any status token other than the literal OK counts as a failure.
• Same/cross tallies come only from the IP rows; the header, the rail map and the ### section titles are decorative.

Per-job artifact checklist — capture all of these every run, on both nodes:

addrs.<node>     idx dev ndev fam gididx addr, all rails (the addressing plan, Step 3a)
tools.<node>     local tool inventory, so a missing binary on the PEER is not
                 mistaken for a dead fabric
topo_report.<node>.txt   NIC/GPU/GID/PCI/NUMA inventory  (drives the diagrams, Step 1)
routes.<node>    ip -6/-4 route show table all; ip route show default; ip -6 neigh
env.<node>       ibv_devinfo, fw_ver, mtu, numa_node, PCI paths, tool inventory
result.txt       the paired IP-vs-RDMA verdicts for one pair (2b), from xrail_worker.sh
matrix.txt       the N x N RDMA matrix + sweeps, from xrail_matrix.sh
run.log[.<node>] everything, timestamped, including the failures

The addrs.<node> field order is fixed — make_diagrams.py parses it, and both workers must emit the same six fields. stats.<node> (the /sys/class/net/*/statistics before/after counters from Step 2a) is not produced by these scripts; capture it by hand when you need to prove packets really left the rail NIC.

Step 3 — Localize the failure to a tier

"Cross-rail RDMA fails" is a symptom. The actionable question is which tier decided that, and the host can get surprisingly far without any switch access.

3a — Decode the addressing plan

Rail addressing on these clusters is structured, not random, and the structure usually encodes physical grouping. This is the only handle on switch topology obtainable without root or vendor cooperation, and it is free — you already captured the addresses.

Method, in the abstract:

1. Collect all rail addresses from every node you can reach, tagged by rail index.
2. Split each address into its fields (IPv6 hextet groups, or IPv4 octets).
3. For each field position, ask: is it constant per rail across nodes (⇒ a rail code), constant per node across rails (⇒ a node id), or constant across a subset of nodes (⇒ a grouping: leaf, pod, or block)?
4. A field may be a sum of two of those — subtract the rail code out and see whether the remainder becomes node-invariant. That remainder is the grouping field.
5. Validate before believing: the derivation must hold for all rails of all nodes. Drop any node whose rails disagree rather than smoothing it over — an inconsistent node is evidence against the encoding, not noise.

A synthetic illustration of what step 4 looks like when it lands. Given ULAs of the form fdXX:A:B:C::/64, one row per rail per node:

node   rail   A       B       C        B>>8   (B>>8) - (A>>8)
n1     0      0x0300  0x4300  0x0012   0x43   0x40
n1     1      0x0100  0x4100  0x0012   0x41   0x40
n1     2      0x0200  0x4200  0x0012   0x42   0x40     <- constant per node
n2     0      0x0300  0x4300  0x0059   0x43   0x40     <- and shared with n1
n3     0      0x0300  0x6300  0x0027   0x63   0x60     <- different group

Read off: A = rail code (a fixed permutation, not necessarily rail << 8), C = node id, B = (group + rail) << 8 — so subtracting the rail out of B yields the grouping field. Here n1 and n2 share a group and n3 does not.

What you get: for each node, a (rail, group, node_id) triple. Then the payoff — check whether any experiment you already ran straddled two groups. If a same-rail RDMA test between two different groups passed, you have measured that a rail plane spans more than one switch, i.e. the upper tier already forwards RoCE, and the block is plane-to-plane rather than switch-to-switch. That single observation reframes the whole problem, and it usually falls out of data you collected for another purpose.

Sanity check the grouping against any vendor documentation: if the doc says N leaves per pod and you derive N distinct group values, the group is the pod. Do not assume a group field is a small within-group index — check the value range (widely separated values in one group rule that out).

3b — What the host can and cannot see

Expect to be asked "the cluster has spine switches, why can't we see them?" The answer is structural, not a permissions problem:

The one unprivileged probe that can reveal a router hop is a rail-local traceroute, and it is almost always the thing nobody ran. Prefer mtr -6 -n -c 3 -r <peer rail addr>: it is more often installed than traceroute on minimal images, and — the part that matters — it prints every next-hop it observed at a given TTL, not just the first.

Enumerating the upper tier. ECMP is usually described as hiding the spine. It does hide it from ip route, which shows one next-hop. But each traceroute probe hashes to a different uplink, so repeated runs sample the tier. Send traffic on rail t to the peer's rail t address (no source binding needed — the route picks the matching rail), collect the hop-2 addresses, repeat per rail, and compare the sets:

for t in 0 1 2 3 4 5 6 7; do
  printf 'r%s: ' "$t"
  for k in 1 2 3 4; do mtr -6 -n -c 3 -r "<peer rail $t addr>"; done \
    | grep -oE '<upper-tier prefix>[0-9a-f:]+' | sort -u | tr '\n' ' '
  echo
done

This answers, from an unprivileged shell, the question everyone assumes only the network operator can answer:

• Sets overlap (a rail sees spines that other rails also see) ⇒ the upper tier is shared. The rails are not disjoint planes; cross-rail is a routing/forwarding policy decision, and asking the operator to change it is a legitimate request.
• Sets are disjoint, each rail seeing about spines ÷ rails addresses ⇒ the tier is partitioned per rail, the planes really are separate end to end, and no host-side change will ever bridge them.

Sanity-check the union against the vendor's stated spine count and against the address range: a contiguous block roughly the size of the union is a good sign you sampled one pool rather than several. Beware the converse error — a small sample can look disjoint by chance, so run enough repetitions that each rail yields well more than spines ÷ rails addresses before concluding "partitioned".

3c — The unprivileged probe set

Run all of it in one allocation. Total cost: seconds.

# --- L3 shape: is a plane on-link, or routed? ---
ip -6 route show table all | grep -E '<rail prefix>|<rail ifname pattern>'
ip -6 route get <peer rail0 addr> oif <rail0 netdev>    # same-rail
ip -6 route get <peer rail1 addr> oif <rail0 netdev>    # cross-rail: same answer?
ip -6 neigh show dev <rail0 netdev>

# --- hop count: leaf-only, or leaf-spine-leaf? ---
# traceroute is often absent on minimal images; check for mtr before assuming.
mtr -6 -n -c 3 -r <peer rail0 addr>                     # same-rail
mtr -6 -n -c 3 -r <peer rail1 addr>                     # cross-rail
traceroute -6 -n -i <rail0 netdev> -s <my rail0> <peer rail0 addr>   # if present

# --- did the cross-rail ping actually use the rail NIC? ---
cat /sys/class/net/<rail0 netdev>/statistics/tx_packets   # before
ping -6 -c 20 -I <rail0 netdev> <peer rail1 addr>
cat /sys/class/net/<rail0 netdev>/statistics/tx_packets   # after

# --- cross-rail RDMA through RDMA CM, not a hand-built address handle ---
rping -s -a <peer rail1 addr>                             # on the peer
rping -c -a <peer rail1 addr> -C 10                       # here
ib_write_bw -R <peer rail1 addr>                          # and a same-rail run as control

Reading it:

• On-link plane (a short prefix such as fdXX:800::/32 dev <rail0>, no via): same-rail is pure L2, cross-rail is the first case needing a router — and a tool with hop_limit = 1 fails it for reasons that have nothing to do with the fabric.
• Routed plane (via <gw> even for same-rail): the same-rail success already proves a router hop works, so hop_limit cannot explain the cross-rail failure.
• route get returns the same oif/via for cross-rail as same-rail: routing is configured; the block is below L3.
• traceroute/mtr cross-rail dies at the first hop where same-rail completes: the block is at the leaf. Completes with more hops: the upper tier forwards it and the block is RoCE-specific (a policy/ACL/class decision, not reachability).

Two traps that will silently invalidate the cross-rail cells:

1. Your "cross-rail" IP test is probably not cross-rail. If each rail plane has a route of its own, then aiming an unbound tool at the peer's rail-1 address makes the kernel egress rail 1 — a same-rail flow with a cross-rail-looking destination. Check with ip -6 route get <peer rail1 addr> from <my rail0 addr> and read the dev and src it returns. Only -I/SO_BINDTODEVICE (or an explicit source bind) forces the flow onto rail 0. Much of the folklore that "IP crosses rails but RoCE doesn't" is this artifact.
2. Source binding often breaks the RDMA CM tools outright. rping -I <src> and ib_write_bw -R --bind_source_ip may hang or fail same-rail. That is a tool limitation, not a fabric result — and with no working positive control those cells must be discarded, not reported as failures. Note also that ib_write_bw -R needs an explicit IPv6 flag (e.g. --ipv6-addr) or it fails address parsing before it ever touches the fabric, on both the client and the server.

Because RDMA CM derives its path from the routing table, it will essentially never choose a cross-rail path. That makes the device-pinned test (-d <rail0 device> against the peer's rail-1 GID) the one that actually matters — it is also what MoRI, UCX and NCCL do.

3d — Questions only the operator can answer

Everything above narrows it; these decide it. Ask them explicitly:

1. Confirm the shared-vs-partitioned upper tier. You should already have a strong answer from the spine enumeration in 3b — bring it, rather than asking cold, and ask them to confirm or correct it. (If you skipped 3b, this is the question that decides whether cross-rail is policy or physics.)
2. Is there an ACL, VRF, or route-policy that permits ICMP but drops or fails to route RoCE (UDP/4791) between planes?
3. What is the oversubscription ratio — how many uplinks per leaf, at what rate? A plane-crossing flow that works but contends 8:1 at the spine is not a performance path even if enabled.
4. Is a lossless class (PFC/DSCP) configured on the plane-crossing path, or only within a plane?
5. What would enabling it cost, and is it supported configuration or a one-off?

Note that host-side PFC state is often unreadable as a normal user (dcb pfc show may return "Operation not supported" if the driver has no DCB netlink support; vendor tools require root), so question 4 genuinely cannot be answered from the node.

Step 4 — Rule out the tunables (with controls)

If cross-rail fails, the reflexive hypothesis is a QoS misconfiguration — a missing lossless class or an unmarked DSCP. Test it properly and it is usually wrong; test it without controls and you will chase it for a week.

On a cross-rail pair, sweep each knob while keeping a same-rail run of the identical command as the control:

If SL and MTU sweeps both fail flat against a working diagonal, the "missing lossless class" hypothesis is dead and you should stop spending time on QoS environment variables and vendor NIC-setup scripts for the cross-rail case. Say so explicitly in the report — it is the hypothesis everyone reaches for, and killing it is a real result.

But: a sweep only exonerates the knob for the tool that ran it. If all sweeps used one binary, Step 2c still applies to all of them at once.

Step 5 — Fallback paths

When cross-rail is genuinely unavailable, there are two escape hatches. Both are worth measuring before designing around the limitation.

5a — The management / default-route NIC as a full mesh

The mgmt NIC is usually on one flat routed subnet with no rail partitioning, and if it is RoCE-capable it is a full node-to-node RDMA mesh. This can unblock an all-to-all workload with no code change — just point the library's device selection at it.

Measure the price before recommending it:

ib_write_bw -d <mgmt dev> -x <mgmt gid> -s 65536 -D 5 <peer>   # and the same for a rail dev

Account for all of the taxes, because only the first is obvious:

• Line rate — mgmt is typically half the rail rate.
• Aggregate width — one mgmt NIC shared by all GPUs versus one rail NIC per GPU. With 8 GPUs this is often ~16x less aggregate bandwidth, which is the number that matters, not the per-link one.
• MTU — mgmt is often 1500 where rails are 9000, costing another chunk of achievable fraction of line rate.
• Contention — the mgmt network also carries NFS, the scheduler, and SSH.
• NUMA — the mgmt NIC is cross-socket for half the GPUs.
• A different GID index than the rails. Check it; do not reuse the rail's.

Cross-fabric (mgmt ↔ rail) will fail — they are separate fabrics. That is expected, not a bug.

How to use it: to get a distributed workload running for functional or CI purposes. Never quote its numbers as performance.

5b — Rail-affine peer NIC selection

This needs nothing from the fabric and is the durable answer to a rail-only fabric. For each remote peer, pick the local NIC whose rail matches that peer's rail, so every QP stays inside one plane. The peer's rail is readable from its GID prefix (Step 3a), so the mapping is derivable at connection setup with no new information from anyone.

The cost is structural, not incidental: a QP now belongs to a rail rather than to a device, so anything the connection advertises per-device — memory registration keys in particular — becomes per-peer. That is the change a library has to absorb, and it is the honest version of "why doesn't the library just use the right NIC?".

This is in the library's control and is a real gap when it is missing. Contrast:

• All-to-all / expert-parallel (GPU-initiated RDMA) genuinely needs GPUi(rail i) ↔ GPUj(rail j), so on a rail-only fabric it hangs or asserts at QP setup unless the library does 5b.
• Point-to-point / KV transfer can usually be pinned to one rail per connection — look for an existing rail-affinity option before writing one.
• Single-rail funneling is not a general workaround: forcing every rank onto one NIC makes connectivity same-subnet but overloads and can deadlock all-to-all transports that assume one NIC per GPU. Prefer per-connection rail affinity over a global single-rail setting.

Step 6 — Draw it

The diagrams are generated, not drawn by hand (no AI image tools — topology must be exact). make_diagrams.py reads the run folder and emits both diagrams; make_report.py calls it automatically for anything missing, so in the normal flow Step 6 is Step 8.

python3 make_diagrams.py <output_folder>          # or just run make_report.py
python3 make_diagrams.py <output_folder> --force  # redraw after re-measuring

It consumes topo_report*.txt (single-node) and *result*.txt + addrs.<hostB> (cross-rail), and writes names that make_report.py's globs pick up: topology_<host>_node.{dot,png} and topology_<A>__<B>_crossrail.{dot,png}. Rendering shells out to graphviz dot; without it you still get the .dot files and the report shows their source plus the render

…(truncated)