Lesson Contents
In this lesson, we’ll look at the evolution of Data Center fabrics, from early tree-like L2 networks to modern L3-based architectures. A fabric is a network topology that connects network devices and servers, designed for high bandwidth, low latency, and redundancy.
We’ll explore the progression from Spanning Tree Protocol (STP) and its workarounds, such as vPC, to full STP replacements like TRILL and FabricPath, and finally to the current industry standard: VXLAN overlays.
Early networks (since the 1990s), including data centers, were primarily based on Layer 2 (L2) networks, where we created tree-like topologies and blocked redundant links using the Spanning Tree Protocol (STP). A single blocked link in a traditional STP network with two uplinks could waste up to 50% of your available bandwidth, yet we accepted this as “normal” for decades. Adding a redundant link meant it would sit idle, doing nothing until a failure occurred.
L2 networks have other issues, such as broadcast storms. A single broadcast storm, caused by a loop or malfunctioning device, could propagate throughout the entire L2 domain and take down the entire network.
Another critical issue was the 4094 usable VLAN limit because of the 12-bit VLAN ID field. When virtualization and multi-tenancy took off, this limit quickly became a bottleneck. Modern data centers require way more segmentation than the 4094 VLANs could offer.
Understanding the limitations of legacy solutions and protocols helps to know why they were replaced with newer, more scalable solutions.
Architecture Evolution
The three-tier architecture has been used in network designs for a long time:
- Access Layer: Connects to end devices (servers).
- Distribution/Aggregation Layer: Aggregates connections from the access layer and provides policy enforcement.
- Core Layer: Provides high-speed transport between distribution layer blocks.
This model was optimized for north-south traffic. However, with the rise of virtualization and modern applications, traffic between servers (east-west) increased; this design became a bottleneck, as communication between two servers in different access blocks had to travel up to the distribution or core layer and back down, increasing latency.
To solve this, we moved to a flattened, two-tier spine-and-leaf design.
- Leaf Switches: Connect to servers and other end devices.
- Spine Switches: Connect only to leaf switches.
In this topology, every leaf is connected to every spine. Any server can talk to any other server with a maximum of two hops (server > leaf > spine > leaf > server). This design reduces latency, provides predictable performance, and increases bandwidth by allowing all links to be active.
Spanning Tree
Standardized as IEEE 802.1D, the primary job of STP was to ensure a single, stable, loop-free path by logically blocking redundant physical links.
While revolutionary at the time, the original STP was slow, with convergence times often taking 30 to 50 seconds. This is unacceptable for modern networks, and newer versions of STP introduced some improvements:
- Rapid Spanning Tree Protocol (RSTP, 802.1w): Introduced in 2001, RSTP improved convergence times, reducing recovery times from tens of seconds to a couple of seconds after a topology change.
- Per-VLAN Spanning Tree Plus (PVST+): A Cisco proprietary enhancement that allowed running a separate STP instance for each VLAN. By selecting different root bridges for different VLANs, it is possible to use some redundant links (for different VLANs).
- Multiple Spanning Tree Protocol (MSTP, 802.1s): This standard was created to reduce the CPU and memory burden of running thousands of PVST+ instances. It allows grouping multiple VLANs into a single MST instance.
Even though these different STP versions offer improvements, the core issue remains: STP blocks links to prevent loops. You can read more about the problems of STP in the STP limitations lesson.
L2 Enhancements and Workarounds
To overcome the limitations of STP, several L2 technologies were developed in the mid-to-late 2000s to enable active-active links and enhance scalability. Let’s take a look.
Link Aggregation and Device Virtualization
These technologies “trick” STP by making multiple links or multiple physical devices appear as a single logical entity.
Link Aggregation (EtherChannel)
The first step was bundling multiple physical links between a single pair of switches into a single logical channel, called EtherChannel. Standardized as LACP (802.3ad, superseded by 802.1AX), this increased bandwidth and provided redundancy without STP blocking the links. If one link in the bundle failed, traffic would continue flowing over the remaining links.
Chassis Virtualization (VSS, StackWise)
The next evolution was to make two physical switches appear as a single logical switch. Technologies like Cisco’s Virtual Switching System (VSS) and StackWise, or Juniper’s Virtual Chassis, allowed downstream devices to connect to two separate physical chassis using a standard EtherChannel. From STP’s perspective, there was only one logical switch, so no loops were formed.
Multi-Chassis Link Aggregation
Technologies like Cisco’s Virtual PortChannel (vPC) or Multi-chassis Link Aggregation (MLAG) allowed a downstream device (like a server) to connect to two independent upstream switches. Using standard LACP, the server forms a single logical PortChannel, but its physical links connect to two separate switches. The two upstream switches work together to make the links appear as a single PortChannel. This eliminated STP-blocked ports at the access layer and provided active-active, loop-free uplinks with device-level redundancy, without full chassis virtualization.
All these technologies were workarounds for the limitations of STP. However, they still operated within an L2 domain and didn’t solve the scaling issues, such as the VLAN limit and broadcast storm issues.
L2 Fabrics
Around the early 2010s, the next logical step was to replace STP altogether with solutions that could use a routing-protocol-like control plane for L2 forwarding. Technologies like TRILL, FabricPath, and SPB emerged, all sharing a similar core design:
- Control Plane: They used the IS-IS link-state protocol to learn the topology, enabling loop-free, multi-path forwarding.
- Load Balancing: This allowed all links to be active using Equal-Cost Multi-Path (ECMP).
- Encapsulation: They forwarded frames by encapsulating the original Ethernet frame in a new header (often called MAC-in-MAC).
Technologies
TRILL (Transparent Interconnection of Lots of Links)
TRILL is an IETF standard (RFC 6325) that implements this model.
To forward frames, TRILL encapsulates the original Ethernet frame inside a new TRILL header.
Although TRILL is a standard, it has never really gained widespread adoption. Vendors didn’t promote it or developed proprietary fabric solutions.
FabricPath
FabricPath is Cisco’s proprietary implementation, functionally similar to TRILL. Launched in 2010, it introduced concepts like conversational MAC learning but ultimately suffered from vendor lock-in and limited hardware support as Cisco began shifting focus to VXLAN.
FabricPath introduced key concepts, such as conversational MAC learning, where switches only learn the MAC addresses of endpoints with which they are actively communicating, rather than flooding and learning all MAC addresses. It also uses a hierarchical MAC address format where the outer destination MAC address includes the Switch ID of the destination switch, making forwarding more like routing.
Like TRILL, it never really took off. FabricPath is Cisco’s proprietary, and many customers didn’t want vendor lock-in. It wasn’t supported on all Cisco Nexus switches. Cisco also shifted its focus more to VXLAN.
Shortest Path Bridging (SPB)
SPB (IEEE 802.1aq) was the IEEE’s standardized competitor to TRILL. Like TRILL and FabricPath, it uses IS-IS to build a loop-free topology. SPB had two different modes:
- 802.1aq (SPBM MAC-in-MAC)
- 802.1Qbp (SPBV – VLAN-based)
Like TRILL, it was largely overshadowed by the rapid rise of VXLAN. It didn’t help that IEEE was late with standardization.
Why these technologies failed
In the end, the L2 fabric technologies, such as TRILL, FabricPath, and SPB, failed to gain widespread adoption. These technologies all relied on building a “better” L2 domain by adding routing to the L2 domain. There were a number of issues:
- Complexity: It’s neither L2 nor L3. Network engineers had to learn new protocols.
- Scalability: Although they addressed the issues of STP, these technologies still operate within a large L2 domain with its flooding/learning behavior, scalability issues, and large failure domains.
- Fragmentation: Each vendor had different implementations.
IP Networking with Overlays
Overlay networking gained widespread adoption, making L2 fabrics obsolete. With overlay networking, we have two networks:
- Underlay network
- Overlay network
The underlay network is used for transport. It provides basic IP connectivity between switches (and hypervisors). On top of the underlay network, we run one or more virtual networks, which we call overlay networks.
There are a couple of reasons why overlay networking became so popular:
- Transport: Uses proven IP routing and routing protocols that network engineers are familiar with (BGP, IS-IS, OSPF).
- Simple: Uses simple, scalable underlay networks with ECMP, fast convergence, and small failure domains.
- Decoupling: It separates the physical network (L3 underlay) from the logical network (L2 overlay). This means you can create and extend L2 segments anywhere in or between data centers just by building a tunnel over your IP network.
- Works on any IP network.
Instead of building new L2 fabrics, we use an L3 underlay network with the routing protocols we already know, and we run L2 on top of it in overlay networks. There are different options for overlay networking. Here are some examples:
- NVGRE (Network Virtualization using Generic Routing Encapsulation)
- STT (Stateless Transport Tunneling)
- GENEVE (Generic Network Virtualization Encapsulation)
- LISP (Locator/ID Separation Protocol)
- VXLAN (Virtual Extensible LAN)
VXLAN
Among all overlay networking options, VXLAN has become the most popular in data centers due to its simplicity, high performance, and widespread industry support from multiple vendors.
VXLAN (standardized in 2014 in RFC 7348) is an encapsulation protocol that tunnels L2 Ethernet frames over an existing L3 IP network. This is often called MAC-in-IP/UDP encapsulation.
This offers multiple advantages over previous L2 fabric technologies like TRILL and FabricPath:
- Transport: Because VXLAN runs over IP, it can use any L3 transport. The underlay network can be built using well-understood routing protocols like OSPF, IS-IS, or BGP. Thanks to ECMP, all links can be used in an active-active fashion.
- Massive Scale: VXLAN uses a 24-bit identifier called the VXLAN Network Identifier (VNI). This allows for over 16 million unique logical segments, solving the issue of the 4094 VLAN limit.
Control Plane
The first VXLAN deployments used a flood-and-learn mechanism for host discovery, similar to traditional Ethernet. This relied on IP multicast in the underlay to handle Broadcast, Unknown Unicast, and Multicast (BUM) traffic, which was inefficient and difficult to scale.
To solve this issue, BGP EVPN (Ethernet VPN). EVPN (RFC 7432) was originally designed as a scalable control plane for MPLS, and now also used for VXLAN. Instead of flooding, switches use BGP to advertise endpoint reachability information (both MAC and IP addresses).
EVPN-VXLAN is the de facto standard for modern data center fabrics since 2016-2018. It provides:
- Control-plane-based MAC and IP learning ensures that there is no inefficient flooding.
- Integrated L2 and L3 forwarding.
- Standards-based multi-vendor interoperability.
- Advanced features like multi-tenancy, mobility, and policy enforcement.
Conclusion
The data center network has undergone a remarkable transformation. We moved from inefficient L2 networks to highly scalable and resilient L3 fabrics:
- We started with STP, which prevented loops but at the cost of slow convergence and blocking redundant links.
- We developed workarounds for STP, such as vPC and MLAG, to enable active-active links.
- We attempted to replace STP with L2 fabrics, such as TRILL and FabricPath, which introduced routing intelligence into L2.
- Finally, we settled on overlay networking, where we run L2 overlay networks on top of L3 underlay networks, with EVPN-VXLAN being the most popular option.
I hope you have enjoyed this lesson. If you have any questions, please leave a comment.