Lesson Contents
VXLAN requires an underlay network so that VTEPs can reach each other. There are different routing protocols to choose from, and OSPF is one of them.
OSPF is widely adopted in LANs, WANs, and data centers. Many network engineers are familiar with it. Convergence is fast, which makes OSPF suitable as the routing protocol for your VXLAN underlay.
There are some discussions about whether OSPF is suitable for very large Clos topologies. This is related to the effect of flooding, the size of the LSDB, and the speed of SPF calculations. One “issue” with OSPF is that it does a complete SPF calculation when a link changes. This isn’t much of an issue with modern hardware and smaller topologies, but it might be a scalability issue when you have very large (Clos) topologies. There are ways to mitigate this, such as using multiple areas and summarization. It’s something to be aware of when you decide on the routing protocol for your VXLAN underlay. With smaller topologies, OSPF is a great choice.
When configuring OSPF, you should always use the point-to-point network type for your interfaces. The default network type is broadcast, which means that OSPF will do a DR/BDR election. This is unnecessary because while all interfaces between the spine and leaf devices are Ethernet, they are point-to-point links. When you use the point-to-point network type, OSPF won’t create type 2 LSAs, which also reduces the size of the database.
In many VXLAN lessons, like the VXLAN MP-BGP EVPN L2 VNI lesson, I used OSPF on the underlay with /24 prefixes. I do this on purpose and to keep it simple so we can focus on the VXLAN parts. This works and is simple, but it’s not very effective. There are however two better options to configure OSPF:
- Use a /31 on all point-to-point links.
- Use ip unnumbered on point-to-point links.
We’ll look at these two options to see how you can configure OSPF for the VXLAN underlay, which is similar to how you might configure it in a production network. We’ll only focus on the underlay network. We won’t configure the overlay network.
Configuration
Let’s start with the physical topology:
We have two spine and four leaf switches. I’m using Cisco NX-OS 9000v version 9.3(9) on all switches. We should start by setting the correct MTU. The highest possible MTU can be different depending on your platform and OS version. On these Cisco NX-OS switches, the highest possible MTU value is 9216.
Let’s configure it on the spine switches:
SPINE1 & SPINE2
(config)# interface Ethernet 1/1-4
(config-if-range)# mtu 9216
And also on the leaf switches:
LEAF1-LEAF4
(config)# interface Ethernet 1/1-2
(config-if-range)# mtu 9216
Now, we can focus on OSPF.
/31 Prefixes
On “normal” point-to-point links, you would use a /30 prefix. This means you’ll have:
- Network address
- Two usable IP addresses for hosts
- Broadcast address
A /31 prefix only has two IP addresses and can be used for point-to-point links. This will save you a lot of IP addresses. Let’s look at an example.
Let’s say we have 4 spine switches and 20 leaf switches. This means we’ll have 4 x 20 = 80 point-to-point links:
- 80 point-to-point links x 4 IP addresses ( /30 prefixes) = 320 IP addresses required for point-to-point links.
- 80 point-to-point links x 2 IP addresses ( /31 prefixes) = 160 IP addresses required for point-to-point links.
Our switches also require loopback interfaces. We’ll need 4 for the spine switches and 20 for the leaf switches. That’s 24 IP addresses for loopbacks.
In total, we would need:
- /30 prefixes: 344 IP addresses
- /31 prefixes: 184 IP addresses
We’ll save quite some IP address space by using /31 prefixes. Here’s an example of what it could look like:
I took the 192.168.1.0/24 subnet and created many /31 prefixes for the physical interfaces. Let’s configure this on the switches.
Configurations
Want to take a look for yourself? Here, you will find the startup configuration of each device.
LEAF1
hostname LEAF1
interface Ethernet1/1
no switchport
mac-address 0050.c253.3001
ip address 192.168.1.1/31
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.3002
ip address 192.168.1.9/31
no shutdown
interface loopback0
ip address 3.3.3.3/32
LEAF2
hostname LEAF2
interface Ethernet1/1
no switchport
mac-address 0050.c253.4001
ip address 192.168.1.3/31
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.4002
ip address 192.168.1.11/31
no shutdown
interface loopback0
ip address 4.4.4.4/32
LEAF3
hostname LEAF3
interface Ethernet1/1
no switchport
mac-address 0050.c253.5001
ip address 192.168.1.5/31
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.5002
ip address 192.168.1.13/31
no shutdown
interface loopback0
ip address 5.5.5.5/32
LEAF4
hostname LEAF4
interface Ethernet1/1
no switchport
mac-address 0050.c253.6001
ip address 192.168.1.7/31
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.6002
ip address 192.168.1.15/31
no shutdown
interface loopback0
ip address 6.6.6.6/32
SPINE1
hostname SPINE1
interface Ethernet1/1
no switchport
mac-address 0050.c253.1001
ip address 192.168.1.0/31
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.1002
ip address 192.168.1.2/31
no shutdown
interface Ethernet1/3
no switchport
mac-address 0050.c253.1003
ip address 192.168.1.4/31
no shutdown
interface Ethernet1/4
no switchport
mac-address 0050.c253.1004
ip address 192.168.1.6/31
no shutdown
interface loopback0
ip address 1.1.1.1/32
SPINE2
hostname SPINE2
interface Ethernet1/1
no switchport
mac-address 0050.c253.2001
ip address 192.168.1.8/31
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.2002
ip address 192.168.1.10/31
no shutdown
interface Ethernet1/3
no switchport
mac-address 0050.c253.2003
ip address 192.168.1.12/31
no shutdown
interface Ethernet1/4
no switchport
mac-address 0050.c253.2004
ip address 192.168.1.14/31
no shutdown
interface loopback0
ip address 2.2.2.2/32
Now, let’s configure OSPF. We’ll enable the feature on all devices:
LEAF1-LEAF4, SPINE1 & SPINE2
(config)# feature ospf
Now, we can start the OSPF process and set the router ID:
SPINE1(config)# router ospf UNDERLAY
SPINE1(config-router)# router-id 1.1.1.1
SPINE2(config)# router ospf UNDERLAY
SPINE2(config-router)# router-id 2.2.2.2
LEAF1(config)# router ospf UNDERLAY
LEAF1(config-router)# router-id 3.3.3.3
LEAF2(config)# router ospf UNDERLAY
LEAF2(config-router)# router-id 4.4.4.4
LEAF3(config)# router ospf UNDERLAY
LEAF3(config-router)# router-id 5.5.5.5
LEAF4(config)# router ospf UNDERLAY
LEAF4(config-router)# router-id 6.6.6.6
We’ll use a named process and call it UNDERLAY. Now, let’s enable OSPF on the loopback interfaces:
LEAF1-LEAF4, SPINE1 & SPINE2
(config)# interface Loopback 0
(config-if)# ip router ospf UNDERLAY area 0.0.0.0
We’ll have to make sure we use the point-to-point network type on the physical interfaces. We’ll do this on the leaf switches:
LEAF1-LEAF4
(config)# interface Ethernet 1/1-2
(config-if-range)# ip ospf network point-to-point
(config-if-range)# ip router ospf UNDERLAY area 0.0.0.0
And the spine switches:
SPINE1 & SPINE2
(config)# interface Ethernet 1/1-4
(config-if-range)# ip ospf network point-to-point
(config-if-range)# ip router ospf UNDERLAY area 0.0.0.0
That completes the configuration.
Verification
Let’s check the OSPF process:
SPINE1# show ip ospf
Routing Process UNDERLAY with ID 1.1.1.1 VRF default
Routing Process Instance Number 1
Stateful High Availability enabled
Graceful-restart is configured
Grace period: 60 state: Inactive
Last graceful restart exit status: None
Supports only single TOS(TOS0) routes
Supports opaque LSA
Administrative distance 110
Reference Bandwidth is 40000 Mbps
SPF throttling delay time of 200.000 msecs,
SPF throttling hold time of 1000.000 msecs,
SPF throttling maximum wait time of 5000.000 msecs
LSA throttling start time of 0.000 msecs,
LSA throttling hold interval of 5000.000 msecs,
LSA throttling maximum wait time of 5000.000 msecs
Minimum LSA arrival 1000.000 msec
LSA group pacing timer 10 secs
Maximum paths to destination 8
Number of external LSAs 0, checksum sum 0
Number of opaque AS LSAs 0, checksum sum 0
Number of areas is 1, 1 normal, 0 stub, 0 nssa
Number of active areas is 1, 1 normal, 0 stub, 0 nssa
Install discard route for summarized external routes.
Install discard route for summarized internal routes.
Number of new LSAs originated : 4
Number of new LSAs received : 47
RFC1583: Disabled, IOS Compatibility: Disabled
Area BACKBONE(0.0.0.0)
Area has existed for 03:14:26
Interfaces in this area: 5 Active interfaces: 5
Passive interfaces: 0 Loopback interfaces: 1
No authentication available
SPF calculation has run 12 times
Last SPF ran for 0.000440s
Area ranges are
Number of LSAs: 6, checksum sum 0x20dd9
OSPF is up and running. We can also check each interface:
SPINE1# show ip ospf interface Ethernet 1/1
Ethernet1/1 is up, line protocol is up
IP address 192.168.1.0/31
Process ID UNDERLAY VRF default, area 0.0.0.0
Enabled by interface configuration
State P2P, Network type P2P, cost 40
Index 2, Transmit delay 1 sec
1 Neighbors, flooding to 1, adjacent with 1
Timer intervals: Hello 10, Dead 40, Wait 40, Retransmit 5
Hello timer due in 00:00:05
No authentication
Number of opaque link LSAs: 0, checksum sum 0
Interface ospf state change count: 3
This tells us that OSPF is enabled on the interface and that the network type is point-to-point. Let’s check the neighbor adjacencies:
SPINE1# show ip ospf neighbors
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 4
Neighbor ID Pri State Up Time Address Interface
3.3.3.3 1 FULL/ - 00:12:45 192.168.1.1 Eth1/1
4.4.4.4 1 FULL/ - 00:06:57 192.168.1.3 Eth1/2
5.5.5.5 1 FULL/ - 00:06:56 192.168.1.5 Eth1/3
6.6.6.6 1 FULL/ - 00:06:55 192.168.1.7 Eth1/4
SPINE2# show ip ospf neighbors
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 4
Neighbor ID Pri State Up Time Address Interface
3.3.3.3 1 FULL/ - 00:08:30 192.168.1.9 Eth1/1
4.4.4.4 1 FULL/ - 00:14:15 192.168.1.11 Eth1/2
5.5.5.5 1 FULL/ - 00:14:15 192.168.1.13 Eth1/3
6.6.6.6 1 FULL/ - 00:14:14 192.168.1.15 Eth1/4
Each spine switch has four neighbor adjacencies, so that’s looking good. Let’s take a look at some routes:
SPINE1# show ip route ospf
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
2.2.2.2/32, ubest/mbest: 4/0
*via 192.168.1.1, Eth1/1, [110/81], 00:08:52, ospf-UNDERLAY, intra
*via 192.168.1.3, Eth1/2, [110/81], 00:09:03, ospf-UNDERLAY, intra
*via 192.168.1.5, Eth1/3, [110/81], 00:08:59, ospf-UNDERLAY, intra
*via 192.168.1.7, Eth1/4, [110/81], 00:08:59, ospf-UNDERLAY, intra
3.3.3.3/32, ubest/mbest: 1/0
*via 192.168.1.1, Eth1/1, [110/41], 00:14:47, ospf-UNDERLAY, intra
4.4.4.4/32, ubest/mbest: 1/0
*via 192.168.1.3, Eth1/2, [110/41], 00:09:03, ospf-UNDERLAY, intra
5.5.5.5/32, ubest/mbest: 1/0
*via 192.168.1.5, Eth1/3, [110/41], 00:08:59, ospf-UNDERLAY, intra
6.6.6.6/32, ubest/mbest: 1/0
*via 192.168.1.7, Eth1/4, [110/41], 00:08:59, ospf-UNDERLAY, intra
192.168.1.8/31, ubest/mbest: 1/0
*via 192.168.1.1, Eth1/1, [110/80], 00:13:14, ospf-UNDERLAY, intra
192.168.1.10/31, ubest/mbest: 1/0
*via 192.168.1.3, Eth1/2, [110/80], 00:09:03, ospf-UNDERLAY, intra
192.168.1.12/31, ubest/mbest: 1/0
*via 192.168.1.5, Eth1/3, [110/80], 00:08:59, ospf-UNDERLAY, intra
192.168.1.14/31, ubest/mbest: 1/0
*via 192.168.1.7, Eth1/4, [110/80], 00:08:59, ospf-UNDERLAY, intra
SPINE1 knows how to reach all loopback interfaces. We see the same thing on SPINE2:
SPINE2# show ip route ospf
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
1.1.1.1/32, ubest/mbest: 4/0
*via 192.168.1.9, Eth1/1, [110/81], 00:09:12, ospf-UNDERLAY, intra
*via 192.168.1.11, Eth1/2, [110/81], 00:09:23, ospf-UNDERLAY, intra
*via 192.168.1.13, Eth1/3, [110/81], 00:09:19, ospf-UNDERLAY, intra
*via 192.168.1.15, Eth1/4, [110/81], 00:09:19, ospf-UNDERLAY, intra
3.3.3.3/32, ubest/mbest: 1/0
*via 192.168.1.9, Eth1/1, [110/41], 00:09:12, ospf-UNDERLAY, intra
4.4.4.4/32, ubest/mbest: 1/0
*via 192.168.1.11, Eth1/2, [110/41], 00:14:52, ospf-UNDERLAY, intra
5.5.5.5/32, ubest/mbest: 1/0
*via 192.168.1.13, Eth1/3, [110/41], 00:14:51, ospf-UNDERLAY, intra
6.6.6.6/32, ubest/mbest: 1/0
*via 192.168.1.15, Eth1/4, [110/41], 00:14:51, ospf-UNDERLAY, intra
192.168.1.0/31, ubest/mbest: 1/0
*via 192.168.1.9, Eth1/1, [110/80], 00:09:12, ospf-UNDERLAY, intra
192.168.1.2/31, ubest/mbest: 1/0
*via 192.168.1.11, Eth1/2, [110/80], 00:14:52, ospf-UNDERLAY, intra
192.168.1.4/31, ubest/mbest: 1/0
*via 192.168.1.13, Eth1/3, [110/80], 00:14:51, ospf-UNDERLAY, intra
192.168.1.6/31, ubest/mbest: 1/0
*via 192.168.1.15, Eth1/4, [110/80], 00:14:51, ospf-UNDERLAY, intra
This is looking good. The spine switches know about all the loopback interfaces in our topology and have all required neighbor adjacencies. Let’s check one of the leaf switches:
LEAF1# show ip route ospf
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
1.1.1.1/32, ubest/mbest: 1/0
*via 192.168.1.0, Eth1/1, [110/41], 00:08:16, ospf-UNDERLAY, intra
2.2.2.2/32, ubest/mbest: 1/0
*via 192.168.1.8, Eth1/2, [110/41], 00:07:22, ospf-UNDERLAY, intra
4.4.4.4/32, ubest/mbest: 2/0
*via 192.168.1.0, Eth1/1, [110/81], 00:07:32, ospf-UNDERLAY, intra
*via 192.168.1.8, Eth1/2, [110/81], 00:07:22, ospf-UNDERLAY, intra
5.5.5.5/32, ubest/mbest: 2/0
*via 192.168.1.0, Eth1/1, [110/81], 00:08:15, ospf-UNDERLAY, intra
*via 192.168.1.8, Eth1/2, [110/81], 00:07:20, ospf-UNDERLAY, intra
6.6.6.6/32, ubest/mbest: 2/0
*via 192.168.1.0, Eth1/1, [110/81], 00:07:25, ospf-UNDERLAY, intra
*via 192.168.1.8, Eth1/2, [110/81], 00:07:22, ospf-UNDERLAY, intra
LEAF1 knows how to reach the different loopback interfaces of the other leaf switches. The other leaf switches would show a similar output.
Configurations
Want to take a look for yourself? Here, you will find the final configuration of each device.
LEAF1
hostname LEAF1
feature ospf
interface Ethernet1/1
no switchport
mtu 9216
mac-address 0050.c253.3001
ip address 192.168.1.1/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/2
no switchport
mtu 9216
mac-address 0050.c253.3002
ip address 192.168.1.9/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/3
switchport access vlan 10
interface loopback0
ip address 3.3.3.3/32
ip router ospf UNDERLAY area 0.0.0.0
router ospf UNDERLAY
router-id 3.3.3.3
LEAF2
hostname LEAF2
feature ospf
interface Ethernet1/1
no switchport
mtu 9216
mac-address 0050.c253.4001
ip address 192.168.1.3/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/2
no switchport
mtu 9216
mac-address 0050.c253.4002
ip address 192.168.1.11/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/3
switchport access vlan 10
interface loopback0
ip address 4.4.4.4/32
ip router ospf UNDERLAY area 0.0.0.0
router ospf UNDERLAY
router-id 4.4.4.4
LEAF3
hostname LEAF3
feature ospf
interface Ethernet1/1
no switchport
mtu 9216
mac-address 0050.c253.5001
ip address 192.168.1.5/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/2
no switchport
mtu 9216
mac-address 0050.c253.5002
ip address 192.168.1.13/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/3
switchport access vlan 10
interface loopback0
ip address 5.5.5.5/32
ip router ospf UNDERLAY area 0.0.0.0
router ospf UNDERLAY
router-id 5.5.5.5
LEAF4
hostname LEAF4
feature ospf
interface Ethernet1/1
no switchport
mtu 9216
mac-address 0050.c253.6001
ip address 192.168.1.7/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/2
no switchport
mtu 9216
mac-address 0050.c253.6002
ip address 192.168.1.15/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/3
switchport access vlan 10
interface loopback0
ip address 6.6.6.6/32
ip router ospf UNDERLAY area 0.0.0.0
router ospf UNDERLAY
router-id 6.6.6.6
SPINE1
hostname SPINE1
feature ospf
interface Ethernet1/1
no switchport
mtu 9216
mac-address 0050.c253.1001
ip address 192.168.1.0/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/2
no switchport
mtu 9216
mac-address 0050.c253.1002
ip address 192.168.1.2/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/3
no switchport
mtu 9216
mac-address 0050.c253.1003
ip address 192.168.1.4/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/4
no switchport
mtu 9216
mac-address 0050.c253.1004
ip address 192.168.1.6/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface loopback0
ip address 1.1.1.1/32
ip router ospf UNDERLAY area 0.0.0.0
router ospf UNDERLAY
router-id 1.1.1.1
SPINE2
hostname SPINE2
feature ospf
interface Ethernet1/1
no switchport
mtu 9216
mac-address 0050.c253.2001
ip address 192.168.1.8/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/2
no switchport
mtu 9216
mac-address 0050.c253.2002
ip address 192.168.1.10/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/3
no switchport
mtu 9216
mac-address 0050.c253.2003
ip address 192.168.1.12/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface Ethernet1/4
no switchport
mtu 9216
mac-address 0050.c253.2004
ip address 192.168.1.14/31
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface loopback0
ip address 2.2.2.2/32
ip router ospf UNDERLAY area 0.0.0.0
router ospf UNDERLAY
router-id 2.2.2.2
Unnumbered Interfaces
Let’s look at our second example. Instead of configuring IP addresses on the physical links, we can “borrow” the IP address from a loopback interface and use this for the OSPF neighbor adjacency. This saves us a lot of required IP addresses. Let’s look at an example.
Imagine we have 4 spine switches and 20 leaf switches, that’s 24 devices in total.
We’ll only need 24 IP addresses for the loopback interfaces and that’s it. That’s a big improvement over using /30 or /31 prefixes.
Let’s see how to configure this. Here is the topology:
We use the same topology as before, but this time, we only have IP addresses configured on the loopback interfaces.
Configurations
Want to take a look for yourself? Here, you will find the startup configuration of each device.
LEAF1
hostname LEAF1
interface Ethernet1/1
no switchport
mac-address 0050.c253.3001
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.3002
no shutdown
interface loopback0
ip address 3.3.3.3/32
LEAF2
hostname LEAF2
interface Ethernet1/1
no switchport
mac-address 0050.c253.4001
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.4002
no shutdown
interface loopback0
ip address 4.4.4.4/32
LEAF3
hostname LEAF3
interface Ethernet1/1
no switchport
mac-address 0050.c253.5001
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.5002
no shutdown
interface loopback0
ip address 5.5.5.5/32
LEAF4
hostname LEAF4
interface Ethernet1/1
no switchport
mac-address 0050.c253.6001
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.6002
no shutdown
interface loopback0
ip address 6.6.6.6/32
SPINE1
hostname SPINE1
interface Ethernet1/1
no switchport
mac-address 0050.c253.1001
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.1002
no shutdown
interface Ethernet1/3
no switchport
mac-address 0050.c253.1003
no shutdown
interface Ethernet1/4
no switchport
mac-address 0050.c253.1004
no shutdown
interface loopback0
ip address 1.1.1.1/32
SPINE2
hostname SPINE2
interface Ethernet1/1
no switchport
mac-address 0050.c253.2001
no shutdown
interface Ethernet1/2
no switchport
mac-address 0050.c253.2002
no shutdown
interface Ethernet1/3
no switchport
mac-address 0050.c253.2003
no shutdown
interface Ethernet1/4
no switchport
mac-address 0050.c253.2004
no shutdown
interface loopback0
ip address 2.2.2.2/32
Let’s configure OSPF. We’ll enable the feature first: