BGP PIC (Prefix Independent Convergence) Core & Edge

PIC (Prefix Independent Convergence) is a feature to decrease the data plane convergence time. There are two flavors:

• BGP PIC Core: decreases convergence time when a core router fails and your IGP has to find a new best path to your PE router.
• BGP PIC Edge: decreases convergence time when a PE router fails and BGP has to switch to a different PE router.

PIC is best explained by showing it in action. We’ll use the following topology:

Above, we have a small provider network in AS 2 with routers running iBGP. P2 is our route reflector in this network. We have two customers, CE1 and CE2. CE2 is advertising two prefixes in BGP. Within AS 2, we use OSPF as the IGP. I increased the cost on the interfaces of the P2 router so that P1 is the preferred path.

hostname CE1
!
ip cef

!
interface GigabitEthernet0/1
 ip address 192.168.12.1 255.255.255.0
!
interface GigabitEthernet0/2
 ip address 192.168.13.1 255.255.255.0
!
router bgp 1
 bgp router-id 1.1.1.1
 neighbor 192.168.12.2 remote-as 2
 neighbor 192.168.13.3 remote-as 2
!
end

hostname CE2
!
ip cef
!
interface Loopback0
 ip address 8.8.8.8 255.255.255.255
!
interface Loopback1
 ip address 88.88.88.88 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 192.168.68.8 255.255.255.0
!
interface GigabitEthernet0/2
 ip address 192.168.78.8 255.255.255.0
!
router bgp 8
 bgp router-id 8.8.8.8
 network 8.8.8.8 mask 255.255.255.255
 network 88.88.88.88 mask 255.255.255.255
 neighbor 192.168.68.6 remote-as 2
 neighbor 192.168.78.7 remote-as 2
!
end

hostname P1
!
ip cef
!
interface Loopback0
 ip address 4.4.4.4 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 192.168.24.4 255.255.255.0
!
interface GigabitEthernet0/2
 ip address 192.168.46.4 255.255.255.0
!
interface GigabitEthernet0/3
 ip address 192.168.34.4 255.255.255.0
!
interface GigabitEthernet0/4
 ip address 192.168.47.4 255.255.255.0
!
router ospf 1
 network 4.4.4.4 0.0.0.0 area 0
 network 192.168.24.0 0.0.0.255 area 0
 network 192.168.34.0 0.0.0.255 area 0
 network 192.168.46.0 0.0.0.255 area 0
 network 192.168.47.0 0.0.0.255 area 0
!
router bgp 2
 bgp router-id 4.4.4.4
 neighbor 5.5.5.5 remote-as 2
 neighbor 5.5.5.5 update-source Loopback0
!
end

hostname P2
!
ip cef
!
interface Loopback0
 ip address 5.5.5.5 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 192.168.35.5 255.255.255.0
!
interface GigabitEthernet0/2
 ip address 192.168.57.5 255.255.255.0
!
interface GigabitEthernet0/3
 ip address 192.168.25.5 255.255.255.0
!
interface GigabitEthernet0/4
 ip address 192.168.56.5 255.255.255.0
!
router ospf 1
 network 5.5.5.5 0.0.0.0 area 0
 network 192.168.25.0 0.0.0.255 area 0
 network 192.168.35.0 0.0.0.255 area 0
 network 192.168.56.0 0.0.0.255 area 0
 network 192.168.57.0 0.0.0.255 area 0
!
router bgp 2
 bgp router-id 5.5.5.5
 neighbor 2.2.2.2 remote-as 2
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 2.2.2.2 route-reflector-client
 neighbor 3.3.3.3 remote-as 2
 neighbor 3.3.3.3 update-source Loopback0
 neighbor 3.3.3.3 route-reflector-client
 neighbor 4.4.4.4 remote-as 2
 neighbor 4.4.4.4 update-source Loopback0
 neighbor 4.4.4.4 route-reflector-client
 neighbor 6.6.6.6 remote-as 2
 neighbor 6.6.6.6 update-source Loopback0
 neighbor 6.6.6.6 route-reflector-client
 neighbor 7.7.7.7 remote-as 2
 neighbor 7.7.7.7 update-source Loopback0
 neighbor 7.7.7.7 route-reflector-client
!
end

hostname PE1
!
ip cef
!
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 192.168.24.2 255.255.255.0
!
interface GigabitEthernet0/2
 ip address 192.168.25.2 255.255.255.0
!
interface GigabitEthernet0/3
 ip address 192.168.12.2 255.255.255.0
!
router ospf 1
 network 2.2.2.2 0.0.0.0 area 0
 network 192.168.24.0 0.0.0.255 area 0
 network 192.168.25.0 0.0.0.255 area 0
!
router bgp 2
 bgp router-id 2.2.2.2
 neighbor 5.5.5.5 remote-as 2
 neighbor 5.5.5.5 update-source Loopback0
 neighbor 5.5.5.5 next-hop-self
 neighbor 192.168.12.1 remote-as 1
!
end

hostname PE2
!
ip cef
!
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 192.168.35.3 255.255.255.0
!
interface GigabitEthernet0/2
 ip address 192.168.34.3 255.255.255.0
!
interface GigabitEthernet0/3
 ip address 192.168.13.3 255.255.255.0
!
router ospf 1
 network 3.3.3.3 0.0.0.0 area 0
 network 192.168.34.0 0.0.0.255 area 0
 network 192.168.35.0 0.0.0.255 area 0
!
router bgp 2
 bgp router-id 3.3.3.3
 neighbor 5.5.5.5 remote-as 2
 neighbor 5.5.5.5 update-source Loopback0
 neighbor 192.168.13.1 remote-as 1
!
end

hostname PE3
!
ip cef
!
interface Loopback0
 ip address 6.6.6.6 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 192.168.46.6 255.255.255.0
!
interface GigabitEthernet0/2
 ip address 192.168.56.6 255.255.255.0
!
interface GigabitEthernet0/3
 ip address 192.168.68.6 255.255.255.0
!
router ospf 1
 network 6.6.6.6 0.0.0.0 area 0
 network 192.168.46.0 0.0.0.255 area 0
 network 192.168.56.0 0.0.0.255 area 0
!
router bgp 2
 bgp router-id 6.6.6.6
 neighbor 5.5.5.5 remote-as 2
 neighbor 5.5.5.5 update-source Loopback0
 neighbor 5.5.5.5 next-hop-self
 neighbor 192.168.68.8 remote-as 8
!
end

hostname PE4
!
ip cef
!
interface Loopback0
 ip address 7.7.7.7 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 192.168.57.7 255.255.255.0
!
interface GigabitEthernet0/2
 ip address 192.168.47.7 255.255.255.0
!
interface GigabitEthernet0/3
 ip address 192.168.78.7 255.255.255.0
!
router ospf 1
 network 7.7.7.7 0.0.0.0 area 0
 network 192.168.47.0 0.0.0.255 area 0
 network 192.168.57.0 0.0.0.255 area 0
!
router bgp 2
 bgp router-id 7.7.7.7
 neighbor 5.5.5.5 remote-as 2
 neighbor 5.5.5.5 update-source Loopback0
 neighbor 5.5.5.5 next-hop-self
 neighbor 192.168.78.8 remote-as 8
!
end

BGP PIC Core

PIC core helps to speed up convergence when there is a failure in the core of your network and the BGP next hop does not change.

Let’s have a look at PE1:

PE1#show ip bgp
BGP table version is 23, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  8.8.8.8/32       6.6.6.6                  0    100      0 8 i
 *>i  88.88.88.88/32   6.6.6.6                  0    100      0 8 i

Let’s focus on 8.8.8.8/32 and 88.88.88.88/32. These two prefixes were advertised by CE2 and PE1 learned them from P2 (our route reflector). The next hop for both prefixes is 6.6.6.6. When PE1 wants to reach CE2, our traffic path looks like this:

How does 6.6.6.6 get resolved? Let’s have a look:

PE1#show ip route 6.6.6.6
Routing entry for 6.6.6.6/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 192.168.24.4 on GigabitEthernet0/1, 00:44:32 ago
  Routing Descriptor Blocks:
  * 192.168.24.4, from 6.6.6.6, 00:44:32 ago, via GigabitEthernet0/1
      Route metric is 3, traffic share count is 1

We learn about 6.6.6.6 through OSPF and to get to this next hop, we use next hop 192.168.24.4 through interface GigabitEthernet0/1. This is what gets installed in the FIB (CEF table):

PE1#show ip cef 6.6.6.6
6.6.6.6/32
  nexthop 192.168.24.4 GigabitEthernet0/1

What about those two prefixes? This is how they show up in the FIB:

PE1#show ip cef | include 8.8
8.8.8.8/32           192.168.24.4         GigabitEthernet0/1
88.88.88.88/32       192.168.24.4         GigabitEthernet0/1

Here’s a quick summary of the recursive routing we saw:

• BGP: to get to 8.8.8.8/32 and 88.88.88.88/32 we use next hop 6.6.6.6.
• IGP: to get to 6.6.6.6/32 we use next hop 192.168.24.4 with interface GigabitEthernet0/1.
• FIB:
  • 8.8.8.8/32 via 192.168.24.4 interface GigabitEthernet0/1
  • 88.88.88.88/32 via 192.168.24.4 interface GigabitEthernet0/1

For each prefix you have learned, you find an entry like this in the FIB. We only have two prefixes but if you learned a million prefixes through BGP, you’ll see them here.

Disaster strikes our provider and suddenly P1 fails:

How does this impact PE1 and how it gets to CE2?

Let’s find out! To see what happens in real-time, I’ll create an access-list that matches the next hop (6.6.6.6) and our two prefixes (8.8.8.8/32 and 88.88.88.88/32). We attach the access-list to a debug:

PE1(config)#access-list 1 permit host 6.6.6.6
PE1(config)#access-list 1 permit host 8.8.8.8
PE1(config)#access-list 1 permit host 88.88.88.88

PE1#debug ip routing 1
IP routing debugging is on for access list 1

Let’s shut all interfaces of P1 now:

P1(config)#interface range GigabitEthernet 0/1 - 4
P1(config-if-range)#shutdown

Now it’s up to OSPF to figure out that P1 is gone and find a new path to 6.6.6.6:

PE1#
RT: updating ospf 6.6.6.6/32 (0x0)  :
    via 192.168.25.5 Gi0/2  0 1048578

RT: closer admin distance for 6.6.6.6, flushing 1 routes
RT: add 6.6.6.6/32 via 192.168.25.5, ospf metric [110/4]
RT: updating bgp 8.8.8.8/32 (0x0)  :
    via 6.6.6.6   0 1048577

RT: closer admin distance for 8.8.8.8, flushing 1 routes
RT: add 8.8.8.8/32 via 6.6.6.6, bgp metric [200/0]
RT: updating bgp 88.88.88.88/32 (0x0)  :
    via 6.6.6.6   0 1048577

RT: closer admin distance for 88.88.88.88, flushing 1 routes
RT: add 88.88.88.88/32 via 6.6.6.6, bgp metric [200/0]

We see that OSPF reconverges and installs 192.168.25.5 (P2) as the new next hop to get to 6.6.6.6/32. BGP is also updated. Let’s take a look at the BGP table:

PE1#show ip bgp
BGP table version is 33, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  8.8.8.8/32       6.6.6.6                  0    100      0 8 i
 *>i  88.88.88.88/32   6.6.6.6                  0    100      0 8 i

Nothing changed in the BGP table, this makes sense since our next hop did not change. Only the path to get to the next hop. Let’s take a look at OSPF:

PE1#show ip route 6.6.6.6
Routing entry for 6.6.6.6/32
  Known via "ospf 1", distance 110, metric 4, type intra area
  Last update from 192.168.25.5 on GigabitEthernet0/2, 00:02:06 ago
  Routing Descriptor Blocks:
  * 192.168.25.5, from 6.6.6.6, 00:02:06 ago, via GigabitEthernet0/2
      Route metric is 4, traffic share count is 1

We see that OSPF now uses 192.168.25.5 (P2) to get to 6.6.6.6/32 via the GigabitEthernet0/2 interface. Let’s take a look at the FIB for our next hop:

PE1#show ip cef 6.6.6.6
6.6.6.6/32
  nexthop 192.168.25.5 GigabitEthernet0/2

Above, we see the new next hop for 6.6.6.6/32 and interface in the FIB. Let’s take a look at the FIB entries for our two prefixes:

PE1#show ip cef | include 8.8    
8.8.8.8/32           192.168.25.5         GigabitEthernet0/2
88.88.88.88/32       192.168.25.5         GigabitEthernet0/2

The FIB entries for our two prefixes has changed as well to reflect the new next hop and outgoing interface. Think about this for a second…

Our router had to figure out a new path to get to 6.6.6.6/32 and update all entries in the FIB that used the old next hop (192.168.24.4) to the new next hop (192.168.25.5). With two prefixes, this is a piece of cake but what if we have one million prefixes in our FIB? They all have to get updated one-by-one and this takes a LONG time (minutes).

The problem here is how our FIB works. We use what we call a “flat” or “flattened” FIB structure: