Introduction to Virtual Extensible LAN (VXLAN)

Virtual eXtensible Local Area Network (VXLAN) is a tunneling protocol that tunnels Ethernet (layer 2) traffic over an IP (layer 3) network.

Traditional layer 2 networks have issues because of three main reasons:

• Spanning tree.
• Limited amount of VLANs.
• Large MAC address tables.

Spanning tree blocks any redundant links to avoid loops. Blocking links to create a loop-free topology gets the job done, but it also means we pay for links we can’t use. We could switch to a layer 3 network, but some technology requires layer 2 networking.

The VLAN ID is 12-bit, which means we can create 4094 VLANs (0 and 4095 are reserved). Only 4094 available VLANs can be an issue for data centers. For example, imagine we have a service provider with 500 customers. With 4094 available VLANs, they can only offer 8 VLANs to each customer.

Because of server virtualization, the number of addresses in the MAC address tables of our switches has grown exponentially. Before server virtualization, a switch only had to learn one MAC address per switchport. With server virtualization, we run many virtual machines (VM) or containers on a single physical server. Each VM has a virtual NIC and a virtual MAC address. The switch has to learn many MAC addresses on a single switchport.

A Top of Rack (ToR) switch in a data center could connect to 24 or 48 physical servers. A data center could have many racks, so each switch has to store the MAC addresses of all VMs that communicate with each other. We require much larger MAC address tables compared to networks without server virtualization.

In this lesson, I’ll explain what VXLAN is, how it works, and how it solves the above layer 2 issues.

Overlay vs Underlay

VXLAN uses an overlay and underlay network:

An overlay network is a virtual network that runs on top of a physical underlay network. Even if you never heard about this terminology before, you have probably seen it. A GRE tunnel is a simple example of an overlay network. The GRE tunnel runs on top of a physical underlay network.

With VXLAN, the overlay is a layer 2 Ethernet network. The underlay network is a layer 3 IP network. Another name for the underlay network is a transport network.

The underlay network is simple; its only job is to get packets from A to B. We don’t use any layer 2 here, only layer 3. When we use layer 3, we can use an IGP like OSPF or EIGRP and load balance traffic on redundant links.

Another advantage is that the overlay and underlay network are independent. The overlay network is virtual and requires an underlay network, but whatever changes you make in the overlay network won’t affect the underlay network. You can add and remove links in the underlay network, and as long as your routing protocol can reach the destination, your overlay network will remain unchanged.

VNI

The VXLAN Network Identifier (VNI) identifies the VXLAN and has a similar function as the VLAN ID for regular VLANs. We use 24 bits for the VNI, which means we can create 16,777,215 ( ~16 million) VXLANs. That’s a lot, compared to those 4094 VLANs with a 12-bit VLAN ID. We can create plenty of VXLANs, which means a large service provider with even thousands of customers can use as many VXLANs per customer as needed.

VTEP

The VXLAN tunnel endpoint (VTEP) is the device that’s responsible for encapsulating and de-encapsulating layer 2 traffic. This device is the connection between the overlay and the underlay network. The VTEP comes in two forms:

• Software (host-based)
• Hardware (gateway)

Let’s look at these two options.

Software

When I’m talking about hosts, I mean hypervisors like VMWare’s ESXi or Microsoft’s Hyper-V. These hypervisors use virtual switches, and some of them support VXLAN. Here’s an illustration:

The VXLAN tunnels are between the virtual switches of the hypervisors. The underlay network is unaware of VXLAN.

Hardware

A hardware VTEP is a router, switch, or firewall which supports VXLAN. We also call a hardware VTEP a VXLAN gateway because it combines a regular VLAN and VXLAN segment into a single layer 2 domain. Some switches have VXLAN support with ASICs, offering better VXLAN performance than a software VTEP. Here’s what it looks like:

In the above picture, the VXLAN tunnels are between the physical switches. The devices that connect to the physical switches are unaware of VXLAN.

Interfaces

Each VTEP has two interfaces types:

• VTEP IP interface: Connects the VTEP to the underlay network with a unique IP address. This interface encapsulates and de-encapsulates Ethernet frames.
• VNI interface: A virtual interface that keeps network traffic separated on the physical interface. Similar to an SVI interface.

A VTEP can have multiple VNI interfaces, but they associate with the same VTEP IP interface. Here’s a picture to help you visualize this: