Datacenter Fabric Standards

Data centre fabric standards — Clos, EVPN and when to do none of this

Criteria verified as of August 2026. Re-verify on the web before committing to anything (§8).

1. Scope and triggers

Applies to designing, sizing and operating a data centre's internal network: leaf-spine Clos topology, oversubscription and scaling, the fabric protocol (underlay), the VXLAN/EVPN overlay, integrated routing and the distributed gateway, VRF multi-tenancy, encapsulation MTU, server multihoming, lossless networking for storage and RDMA, data centre interconnect, and the criteria for not building an EVPN fabric.

Triggers: "leaf-spine", "Clos", "spine", "leaf", "border leaf", "superspine", "oversubscription", vxlan/vni/vtep/nve, evpn, l2vpn evpn, route-type 2/type-5, esi, anycast-gateway, irb, l3vni/l2vni, vrf, "BGP unnumbered", "underlay/overlay", mtu 9216/"jumbo", pfc, 802.1Qbb, dcbx, ets, ecn, wred, dcqcn, "lossless", "RoCE", "DCI", "VXLAN stretch", "MLAG", "vPC", "ESI-LAG".

Not applicable: each skill decides a different thing: networking-standards (the backbone, the parent: it decides addressing and IPAM, which VLAN exists, BGP/OSPF fundamentals, design MTU/MSS, proxies and host overlays, NetBox as the SoT; none of that is reopened here, here we decide the fabric's topology and control plane); routing-switching-standards (it decides campus and edge: STP, campus MLAG, VRRP, the site IGP, BGP policy towards the outside, RPKI, QoS, CoPP and the management plane — here only BGP as the internal fabric protocol, not Internet policy); network-troubleshooting-standards (it decides the reactive method when the fabric is already failing); network-automation-standards (it decides how this configuration is generated, tested and applied, and its telemetry); high-speed-interconnect-standards (already written: InfiniBand and RoCE as a compute interconnect are theirs; here only the Ethernet that carries them); datacenter-facilities-standards (the physical plant is theirs — power, cooling, racks, cabling); kubernetes-standards (it decides what runs on top: CNI, Service, Ingress, NetworkPolicy, service mesh); firewall-policy-standards (it decides which flow is permitted between tenants and zones); finops-standards (cost per port, optics and transceivers). Also boundaries: onprem-standards (the umbrella), iac-standards, observability-standards, sre-practice-standards, secrets-management-standards, linux-hardening-standards, vulnerability-management-standards, identity-access-management-standards, offensive-security-standards (this skill is defensive), vpn-standards, dns-standards, network-vendors-standards, telco-5g-standards and wan-legacy-standards.

Governing principle: the modern data centre is designed as a routed fabric, not as one big switched network. Layer 2 is reduced to a minimum and carried encapsulated over routing; the throughput scales horizontally; and predictable latency is worth more than peak bandwidth.

2. Default decisions

Verify status, version and RFC on the web before committing to anything (§8). The RFCs in this table are verified one by one against the rfc-editor.org JSON API.

3. Design decisions

EVPN: what it actually solves VXLAN without a control plane floods and learns, like a switch. EVPN replaces that with BGP advertising what each VTEP knows, and with it the dependency on multicast in the underlay, the flooding of unknown unicast and much of the ARP/ND on the wire all disappear.

• Type 1 — Ethernet Auto-Discovery: discovery by ESI; it underpins multihoming (fast convergence when a link drops, aliasing to balance towards a multihomed server).
• Type 2 — MAC/IP Advertisement: the host's MAC and, optionally, its IP. It is what makes ARP/ND suppression possible (proxy ARP/ND operational aspects in RFC 9161).
• Type 3 — IMET: it builds the replication tree for BUM traffic. Procedures updated by RFC 9572 (May-2024).
• Type 4 — Ethernet Segment: it discovers who shares a segment and elects the designated forwarder that forwards BUM towards it (avoiding duplicates and loops).
• Type 5 — IP Prefix (RFC 9136): IP prefixes with no MAC; external connectivity, summarisation, VRFs.
• Reading rule: an L2 reachability problem within a VNI → types 2 and 3; multihoming or duplicated traffic → types 1 and 4; connectivity between VRFs or towards the outside → type 5.

MTU: the non-negotiable requirement

• The encapsulation adds a header (VXLAN over UDP/IP adds tens of bytes; Geneve more, and variable by options). If the fabric does not carry the tenant's frame plus that header, it breaks.
• What breaks if it is missing: ping works, the TCP handshake works and the transfer hangs. Large packets with DF disappear and, without ICMP fragmentation needed / packet too big coming back, PMTUD dies silently. The canonical symptom: "it connects but hangs on transfer" — the demonstration belongs to network-troubleshooting-standards; the fix belongs here and is a design one.
• Uniform means uniform: a single link with a lower MTU makes the problem intermittent and dependent on the ECMP path. It is verified as a gate (§4), not by visual inspection.

Server multihoming

• EVPN-MH (ESI-LAG) is the default: standard, with no peer link between leaves, supporting more than two leaves, with no proprietary shared state. The ESI's value and uniqueness, the DF election algorithm (RFC 8584) and the behaviour when a leaf loses its only uplink are decided explicitly.
• MLAG/vPC remains valid where the hardware dictates, but it drags the campus's failure modes into the fabric (split brain, keepalive, peer upgrades) which do not exist in EVPN-MH. If you choose it, those three cases are tested in a lab (criteria in routing-switching-standards).

Storage and RDMA: badly done lossless networking propagates congestion

• PFC (IEEE 802.1Qbb, today incorporated into 802.1Q) pauses per priority on the link, and its effect is to push congestion backwards: if the receiver does not drain, the pause propagates hop by hop and stops traffic that had nothing to do with it (congestion spreading). With buffer dependency cycles it can produce PFC deadlock, from which the network does not recover on its own.
• That is why the order is ECN first: ECN marking (RFC 3168) with per-queue thresholds and end-to-end rate control (DCQCN in RoCEv2, which combines ECN and feedback to the sender). PFC remains as a last-resort safety net, not as the primary mechanism.
• Requirements if it is deployed: a single lossless class, ETS (IEEE 802.1Qaz) sharing out the rest, priority mapping consistent across every hop (one mismatch breaks the guarantee), sized buffer headroom, and monitoring of pause frames and ECN marking: if nobody looks at the PFC counters, you do not know you are pausing.
• InfiniBand versus RoCE belongs to high-speed-interconnect-standards (already written).

DCI: the danger of stretching layer 2

• By default, DCs are interconnected at layer 3. Stretching layer 2 creates a single failure domain with latency in the middle: a storm, a loop or a control plane failure propagates to both sites at once, and the high availability that motivated the stretch disappears precisely in the scenario that justified it.
• If the requirement is real, it is bounded: only the necessary VNIs, with BUM and MAC control, separate failure domains on both sides and a written retirement date. A "temporary" stretch with no date is permanent. The correct alternative is almost always to fix the application that assumes L2 adjacency and treats the IP as an identity.

When an EVPN fabric is NOT needed — the section that adds the most value

• One rack, two ToRs and plain routing are enough. With two switches, a couple of VLANs, LACP towards the servers and a redundant gateway, EVPN contributes nothing and adds a whole control plane that has to be known how to operate, diagnose and upgrade.
• Honest thresholds for considering it: more than a handful of racks; a real need to move a subnet between racks; multi-tenancy with overlapping addresses; standard multihoming to more than two leaves. None of them is "we want VXLAN because it's modern".
• The cost that must be accepted first: staff capable of diagnosing BGP with L2VPN families, a NOS with mature support, a lab to test changes, and automation (it is not operated by hand). If any of the four is missing, an EVPN fabric is a source of incidents, not an improvement.
• Legitimate alternatives: pure routing with ECMP and no overlay when each rack is a subnet; an overlay on the host (Kubernetes or the hypervisor already do it, kubernetes-standards) leaving the physical network as dumb, fast transport.
• Rule: the complexity of the control plane is justified by a written requirement, not by the vendor's catalogue.

4. Quality gates

• End-to-end MTU verification as an automatic gate after every change: a test with a large packet and the DF bit between VTEPs and between hosts on different VNIs, not configuration inspection.
• Failure testing with real traffic and measurement before production: the loss of a spine (must be transparent), of a leaf uplink, of a leaf on a multihomed server, and the recovery from each of them (coming back fails more often than going).
• A negative tenant isolation test: verify that the VRFs cannot reach each other. Multi-tenancy tested only along the happy path is not tested.
• Consistency between peers of the same role: the fabric is regular by design, so a leaf that differs from the rest is a finding. Automatic diff.
• A virtual lab with the same NOS versions before any change to EVPN, the underlay or VRF policy (tooling in network-automation-standards).
• With lossless networking: validate that the class is mapped identically on every hop and run a load test watching the PFC and ECN counters.

5. Operations and security

• An OOB management plane for the whole fabric, separate from the data traffic, with centralised AAA and SSH/SNMPv3 (the full criteria in routing-switching-standards). It is what lets you fix the underlay change that isolated a leaf.
• The fabric is not a trust zone. Sharing a fabric does not authorise traffic: east-west policy belongs to firewall-policy-standards. A VRF segregates routing, it does not apply policy.
• VXLAN encrypts and authenticates nothing: whoever can inject into the underlay can inject into a VNI. The underlay must be physically bounded and sensitive traffic protected above it (mTLS, IPsec, MACsec on links that leave the premises).
• Signals that are always watched: underlay and overlay BGP sessions, EVPN routes by type, MAC and ARP/ND entries per VTEP against the ASIC's table limit (exhausting it is a silent and brutal failure), per-interface discards and errors, the real ECMP balance (an unbalanced hash saturates one link with the fabric at 40%), PFC/ECN counters, and optics.
• Upgrades: one at a time, by role, draining the node first (removing it from the ECMP) and verifying between steps. Upgrading spines and leaves at the same time is how a data centre is lost.
• Fabric-specific performance: the value of a Clos is that every leaf-leaf pair is at the same distance and the high percentile of latency is stable — a design that improves the peak at the cost of unequal paths is worse for distributed applications, which wait for the slowest one. ECMP with a per-flow hash, never per packet; watch the real entropy (a few large flows unbalance ECMP even with spare capacity). The ASIC buffer is finite and shared: most of the unexplainable discards are incast, which is not fixed with more bandwidth.
• Capacity: the oversubscription ratio is reviewed with real per-rack data, not with the one from the original design. Cost per port and optics in finops-standards.

6. Performance

Section deliberately omitted: telemetry, its thresholds and its alerts belong to observability-standards; reactive diagnosis, to network-troubleshooting-standards; and what is specific to fabric performance is integrated into §5, where it is operated. Duplicating it here would be filler.

7. Sustainability and prohibitions

• Cadence: the NOS reviewed quarterly and in the face of an exploitable CVE; a stable branch with mature EVPN support over the latest feature. An EVPN bug is an incident for the whole DC.
• Deprecation: retired VNIs, VRFs and ESIs are removed from the configuration, the SoT and the documentation.
• End of support for the inventoried hardware: the fabric is replaced by generations, and the plan starts before the vendor announces it.

FORBIDDEN

• ❌ Designing a new DC as one big switched network with stretched VLANs and STP as the convergence mechanism.
• ❌ Leaf-leaf or spine-spine links in a Clos.
• ❌ VXLAN flood-and-learn (without EVPN) in a new deployment.
• ❌ A fabric with MTU 1500, or with a different MTU on some link.
• ❌ A centralised gateway instead of a distributed anycast one, with no written requirement.
• ❌ Asymmetric IRB by default in a fabric that will grow.
• ❌ Stretching layer 2 between data centres with no written requirement, without bounding the failure domain and without a retirement date.
• ❌ PFC across the whole fabric "just in case"; or lossless networking without ECN, without consistent mapping on every hop and without monitoring pause counters.
• ❌ Treating the fabric as a trust zone and skipping east-west policy.
• ❌ Building an EVPN fabric for one rack, or without the staff, lab and automation to operate it.
• ❌ Upgrading several leaves or spines at once, or without draining the node first.
• ❌ Divergent configuration between devices of the same role, or made by hand outside the SoT.
• ❌ Not declaring the oversubscription ratio per rack role.
• ❌ ECMP with a per-packet hash.
• ❌ Putting a fabric into production without having tested and measured the loss of a spine, an uplink and a leaf with real traffic.

8. Mandatory web verification

Methodology: the RFCs were verified one by one against the rfc-editor.org JSON API (title, status, date, obsoletes/obsoleted_by/updated_by), not against HTML summaries.

RFCs verified Aug-2026. VXLAN = RFC 7348 (Aug-2014, Informational — it is not Standards Track; citing it as "a standard" is incorrect); Geneve = RFC 8926 (Nov-2020, Proposed Standard). EVPN = RFC 7432 (Feb-2015), updated by 8584, 9161, 9572, 9573 and 9746; EVPN over NVO/VXLAN = RFC 8365 (Mar-2018), updated by 9746. Symmetric/asymmetric IRB = RFC 9135 and IP prefixes / type-5 = RFC 9136 (both Oct-2021). DF election framework = RFC 8584 (Apr-2019, updated by 9722 and 9785). Split-horizon in EVPN multihoming = RFC 9746 (Mar-2025), which updates 7432 and 8365. Proxy ARP/ND = RFC 9161 (Jan-2022); BUM = RFC 9572 (May-2024). BGP in large-scale DCs = RFC 7938 (Aug-2016, Informational). IPv6 next hop for IPv4 NLRI ("BGP unnumbered") = RFC 8950 (Nov-2020), which obsoletes RFC 5549 — citing 5549 today is a factual error. ECN = RFC 3168, updated by 4301, 6040, 8311 and 9768. IEEE: PFC = 802.1Qbb and QCN = 802.1Qau, both incorporated into the 802.1Q base (the current revision is 802.1Q-2022); ETS = 802.1Qaz; DCBX is an extension of LLDP (802.1AB).

Declared discrepancy: RFC 7432 has a successor in progress. draft-ietf-bess-rfc7432bis is at revision -14 with activity on 2-Mar-2026 and is not yet an RFC (rfc: null in the datatracker, with no intended std level declared in the register consulted). The current normative reference is still RFC 7432, but it is being replaced: verify whether it has been published before citing it in an architecture document.

Declared gaps — do NOT fill from memory:

1. The exact overhead bytes of VXLAN and Geneve and the resulting concrete minimum MTU: not verified byte by byte. Geneve is variable because of its options. Calculate it and test it.
2. Geneve support in ASICs by platform and generation: not verified, and it is the criterion that decides VXLAN versus Geneve.
3. ASIC table limits (MAC, ARP/ND, routes, VTEPs, VNIs): not verified. They are the fabric's real ceiling and they vary by model and forwarding profile.
4. EVPN versions and support status per NOS (including open NOSes such as SONiC and FRR-based ones): not verified. Check the version, maintenance and raw licence before setting any of them as the default.
5. ECN thresholds, PFC headroom and DCQCN parameters: not verified and highly dependent on the hardware and the traffic profile. They start from the vendor's guidance and are validated under load.
6. PFC deadlock in specific Clos topologies and its current mitigations: general criteria here, not cross-checked against recent literature.
7. Typical oversubscription ratios by workload (AI versus general compute): they are engineering criteria, not verified measurements.
8. high-speed-interconnect-standards already exists: InfiniBand, RoCE v2, iWARP, the subnet manager, PFC deadlock as seen from the interconnect and NVMe over Fabrics are theirs. datacenter-facilities-standards also exists: the physical plant — power, cooling, cabling — is theirs. Do not improvise it here.

If the web contradicts this document, the web wins — flag the discrepancy.