# AI Networking Collectives

> AI Networking & Collective Communication

- Skill: `sanjeevrg89/ai-networking-collectives` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add sanjeevrg89/ai-networking-collectives`
- Raw SKILL.md: https://api.skillmd.com/api/skills/sanjeevrg89/ai-networking-collectives/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: sanjeevrg89 (https://skillmd.com/u/sanjeevrg89)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/sanjeevrg89/ai-networking-collectives

---


# AI Networking & Collective Communication

Apply the judgment of an engineer who has brought up and tuned the collective stack on
tens-of-thousands-of-accelerator clusters for years: **communication overhead is a dominant bottleneck
in large distributed training**, so map the chattiest collectives onto the fastest interconnect domain,
keep them overlapped with compute, and treat a tuned, topology-aware comm stack as a prerequisite for
high MFU — not an afterthought.

## How to use this skill

1. **Read `ai-networking-collectives-guide.md`** in this directory — the full reference (collective
   operations and where they appear, ring/tree/hierarchical algorithms, NCCL/RCCL internals and
   tunables, interconnects/transports, topology awareness, comm/compute overlap, diagnosis,
   anti-patterns, troubleshooting). Apply it to the task.
2. For concrete artifacts to imitate — an `all_reduce_perf` / nccl-tests diagnostic run, a
   topology-aware placement note, and a comms-bottleneck triage checklist — read **`examples.md`**.
3. Match the cluster's existing fabric (IB vs RoCE vs TCPX/ICI), NCCL version, and placement scheme;
   apply the correctness rules (keep collectives in the high-bandwidth domain, overlap comm with
   compute, validate the topology NCCL actually chose) regardless. **Never invent an NCCL_* flag,
   benchmark number, or hardware bandwidth figure — the stack moves fast; verify against current docs.**

## Essentials (full detail in `ai-networking-collectives-guide.md`)

- **Comms can dominate at scale.** At frontier scale, a topology-blind placement or an untuned
  collective stack silently halves goodput. The fix is structural (placement + algorithm + overlap),
  not a single magic flag.
- **Know which collective each parallelism axis emits.** DP grad sync = all-reduce (or
  reduce-scatter + all-gather); FSDP/ZeRO-3 = all-gather (params, fwd+bwd) + reduce-scatter (grads);
  TP = all-reduce (or reduce-scatter/all-gather with sequence parallel); MoE = all-to-all (dispatch +
  combine); pipeline = point-to-point send/recv. Tune the one that's on your critical path.
- **Match the collective to the interconnect.** Put the highest-volume, most-frequent collectives
  (TP all-reduce, MoE all-to-all) inside the NVLink/NVSwitch domain (intra-node); let DP/FSDP cross
  nodes over IB/RoCE. A TP group spanning nodes over Ethernet is a classic self-inflicted bottleneck.
- **Ring is bandwidth-optimal, tree is latency-optimal.** Large messages → ring/hierarchical
  (bandwidth-bound, ~2(N−1)/N of data moved); small messages/many ranks → tree (log-depth latency).
  NCCL auto-selects per size/topology; you mostly verify its choice rather than override it.
- **NCCL/RCCL is the engine.** It discovers topology, builds rings/trees across NVLink/PCIe/IB/RoCE,
  splits each collective across multiple **channels** for parallelism, and picks algo+protocol
  (LL/LL128/Simple) by message size. `NCCL_DEBUG=INFO` (plus `NCCL_DEBUG_SUBSYS=INIT,GRAPH`) prints the
  rings/trees and transports it chose — read it before touching any knob.
- **GPUDirect RDMA / SHARP are the big wins.** GPUDirect RDMA moves data NIC↔GPU without staging in
  host memory; SHARP does the reduction *in the switch* (offloads all-reduce from GPUs). On cloud
  Ethernet without RDMA NICs, GPUDirect-TCPX/TCPXO provide a GPU-direct datapath over TCP — verify the
  exact plugin/env for your platform.
- **RoCEv2 needs a lossless fabric.** RoCE over Ethernet requires PFC and ECN/DCQCN tuned end-to-end;
  unconfigured, you get pause storms, drops, and mysterious slow all-reduces. IB gives this with less
  hand-tuning. This is the #1 RoCE foot-gun.
- **Topology-aware placement is mandatory.** On rail-optimized fat-trees, place ranks so same-rail GPUs
  talk leaf-local; keep gang-scheduled jobs within a high-bandwidth block; minimize cross-spine hops.
  Oversubscription and congestion at the spine show up as a comms tax on every step.
- **Overlap comm with compute or pay full price.** FSDP prefetch, bucketed gradient all-reduce
  overlapping backward, async TP, pipeline that hides send/recv behind compute — if the collective is
  on the critical path (GPU idle waiting on the network), you've lost the overlap.
- **Mind NUMA/PCIe.** A GPU talking to a NIC on the wrong NUMA node or across the PCIe root complex
  loses bandwidth and adds latency. Pin processes, match GPU↔NIC affinity, and check `nvidia-smi topo
  -m` / the NCCL graph.
- **Diagnose with nccl-tests first.** `all_reduce_perf` reports **busbw** (bus bandwidth) — compare it
  to the fabric's expected ceiling. Low busbw at size = bad algo/topology/transport; a cliff at node
  boundary = the inter-node fabric. Then nvtx/Nsight/PyTorch-profiler timelines to find stragglers.
- **One slow rank stalls the whole collective.** Collectives are barriers; a single straggler GPU,
  hot NIC, throttling link, or bad cable drags every rank. Hunt the outlier (per-rank kernel time,
  ECC errors, link flaps), don't average it away.

## Related skills

- `[[training-frameworks]]` — choosing DP/FSDP/TP/PP/EP degrees and the parallelism that *emits* these
  collectives; this skill is the comm layer beneath it.
- `[[ml-frameworks]]` — PyTorch/JAX/XLA the collectives run under (process groups, XLA SPMD, ICI).
- `[[slurm-hpc-on-kubernetes]]` — MPI/RDMA on Slurm/Volcano, the HPC-side launcher and fabric.
- `[[gke-master]]` — provisioning RDMA/TCPX(O) node pools, compact placement, the cloud fabric.
- `[[jobset-leaderworkerset]]` — gang/topology-aware placement of multi-host workers.
- `[[aiml-on-kubernetes]]` — umbrella for running training/inference on K8s/GKE.
- `[[kubernetes-internals-expert]]` — scheduler/topology plumbing under topology-aware placement.