Troubleshooting Spanning Tree

Out of the box spanning tree does a fine job creating a loop-free topology. There are a number of things that could go wrong, maybe you expect a certain output but your switches are telling you something different. Also because of misconfiguration some funky things can happen…let’s take a look at some scenarios. Here’s the topology we will use:

Incorrect Root Port

Three switches and between the switches we have two links for redundancy. SW1 has been elected as the root bridge for VLAN 1. When you are dealing with spanning tree it’s best to draw a small picture of the network and write down the interface roles for each switch (designated, non-designated/alternate or blocked). Note that one of the links between SW1 and SW3 is an Ethernet interface (10Mbit). All the other links are FastEthernet. Let’s take a closer loop at the spanning tree topology:

I used the show spanning-tree command to verify the interface roles for SW1 and SW3. As you can see SW3 has elected its Ethernet 0/13 interface as its root port and the FastEthernet 0/14 interface is elected as an alternate port. This is not a very good idea. It means we’ll send all traffic down the 10Mbit link while the 100Mbit is not used at all. When a switch has to elect a root port it will select one like this:

1. Choose the interface that has the lowest cost to the root bridge.
2. If the cost is equal, select the interface with the lowest port priority.

Normally the cost of an Ethernet interface is higher than FastEthernet so it should select the FastEthernet interface. Why did SW3 pick the Ethernet 0/13 interface?

SW3#show spanning-tree vlan 1           

VLAN0001
  Spanning tree enabled protocol ieee
  Root ID    Priority    24577
             Address     0011.bb0b.3600
             Cost        19
             Port        13 (FastEthernet0/13)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     000f.34ca.1000
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 15 

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
E0/13               Root FWD 19        128.13   P2p 
Fa0/14              Altn BLK 19        128.14   P2p 
Fa0/16              Desg FWD 19        128.16   P2p 
Fa0/17              Desg FWD 19        128.17   P2p

We can see that the Ethernet 0/13 and FastEthernet 0/14 interface have the same cost. SW3 will then select the interface with the lowest port-priority which is interface Ethernet 0/13. Let’s check the interface:

SW3#show run interface fa0/13
Building configuration...

Current configuration : 102 bytes
!
Interface Ethernet0/13
 switchport mode dynamic desirable
 spanning-tree cost 19

We’ll check the interface configuration and you can see that someone has changed the cost of the interface to 19 (the default for FastEthernet interfaces). Let’s get rid of this:

SW3(config)#interface Ethernet 0/13
SW3(config-if)#no spanning-tree cost 19

Let’s get rid of the cost command and check the result:

SW3#show spanning-tree vlan 1

VLAN0001
  Spanning tree enabled protocol ieee
  Root ID    Priority    24577
             Address     0011.bb0b.3600
             Cost        19
             Port        14 (FastEthernet0/14)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     000f.34ca.1000
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 15 

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
E0/13               Altn BLK 100       128.13   P2p 
Fa0/14              Root FWD 19        128.14   P2p 
Fa0/16              Desg FWD 19        128.16   P2p 
Fa0/17              Desg FWD 19        128.17   P2p

After we removed the cost command you can see that the port state has changed. FastEthernet 0/14 is now the root port and the cost of the Ethernet 0/13 interface is 100 (which is the default for Ethernet interfaces). Problem solved!

Lesson learned: Make sure the interface you want to be the root port has the lowest cost path to the root bridge!

Spanning Tree Disabled

Let’s look at a different scenario. We use the same topology:

All the interfaces are equal (FastEthernet). SW1 is the root bridge for VLAN 10 and after checking the interface roles this is what we find:

Hmm interesting…SW1 is the root bridge and FastEthernet 0/17 has been elected as a backup port. Now that’s something you don’t see every day. SW2 has elected a root port and all the other interfaces are alternate ports. I don’t see anything on SW3. Let’s take a look at the spanning tree of VLAN 10:

SW1#show spanning-tree vlan 10

VLAN0010
  Spanning tree enabled protocol ieee

SW2#show spanning-tree vlan 10

VLAN0010
  Spanning tree enabled protocol ieee

SW3#show spanning-tree vlan 10

Spanning tree instance(s) for vlan 10 does not exist.

We can see that SW1 and SW2 are running spanning tree for VLAN 10. SW3 however is not running spanning tree for VLAN 10. What could be the issue? Just in case, let’s check the interfaces of SW3:

SW3#show interfaces fa0/13 | include line protocol
FastEthernet0/13 is up, line protocol is up (connected)

SW3#show interfaces fa0/14 | include line protocol
FastEthernet0/14 is up, line protocol is up (connected)

SW3#show interfaces fa0/16 | include line protocol
FastEthernet0/16 is up, line protocol is up (connected)

SW3#show interfaces fa0/17 | include line protocol
FastEthernet0/16 is up, line protocol is up (connected)

All interfaces are up and running, what about the trunk configuration?

SW3#show interfaces trunk 

Port        Mode             Encapsulation  Status        Native vlan
Fa0/13      desirable        n-isl          trunking      1
Fa0/14      desirable        n-isl          trunking      1
Fa0/16      desirable        n-isl          trunking      1
Fa0/17      desirable        n-isl          trunking      1

Port        Vlans allowed on trunk
Fa0/13      1-4094
Fa0/14      1-4094
Fa0/16      1-4094
Fa0/17      1-4094

Port        Vlans allowed and active in management domain
Fa0/13      1,10
Fa0/14      1,10
Fa0/16      1,10
Fa0/17      1,10

The interfaces are looking good, we have trunks and VLAN 10 is active on all interfaces of SW3. This means that spanning-tree should be active for VLAN 10.

SW3#show spanning-tree vlan 10

Spanning tree instance(s) for vlan 10 does not exist.

Let’s take another look at this message. It says that spanning tree for VLAN 10 does not exist. There are two reasons why could see this message:

• There are no interfaces active for VLAN 10.
• Spanning tree has been disabled for VLAN 10.

We confirmed that VLAN 10 is active on all interfaces of SW3 so maybe spanning tree has been disabled globally? Let’s give it a try:

SW3(config)#spanning-tree vlan 10

Let’s give it a shot by typing in spanning tree vlan 10 and verify our work:

SW3#show spanning-tree vlan 10

VLAN0010
  Spanning tree enabled protocol ieee
  Root ID    Priority    24586
             Address     0011.bb0b.3600
             Cost        19
             Port        13 (FastEthernet0/13)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32778  (priority 32768 sys-id-ext 10)
             Address     000f.34ca.1000
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Fa0/13              Root FWD 19        128.13   P2p 
Fa0/14              Altn BLK 19        128.14   P2p 
Fa0/16              Desg FWD 19        128.16   P2p 
Fa0/17              Desg FWD 19        128.17   P2p

There we go…that’s looking better! Spanning-tree is now enabled for VLAN 10 and is working…problem solved! This issue might sound a bit lame but I do see it every now and then in the real world. A scenario I encountered before is a customer that was told by their wireless vendor to disable spanning-tree for the interfaces that connect to the wireless access point. This is what the customer typed in on the switch:

SW3(config)#interface fa0/1
SW3(config-if)#no spanning-tree vlan 10

On the interface they typed no spanning tree vlan 10 but you can see you end up in the global configuration mode. There is no command to disable spanning tree on the interface like this so the switch thinks you typed in the global command to disable spanning tree. The switch accepts the command, disabled spanning tree for VLAN 10 and kicks you back to global configuration mode…problem solved!

Lesson learned: Check if spanning-tree is enabled or disabled.

BPDUs Blocked

Onto the next exercise, same topology:

This time the customer is complaining about bad performance. Let’s start by verifying the spanning tree topology:

Take a look at the picture above. Do you see that Interface FastEthernet 0/16 on SW2 and SW3 are designated? On SW1 all interfaces are designated. What do you think happens once one of our switches forwards a broadcast or has to flood a frame? Bingo! We’ll have a loop…

Normally in this topology the FastEthernet 0/16 and 0/17 interfaces on SW3 should both be alternate ports because SW3 has the worst bridge ID. Since they are both designated we can assume that SW3 is not receiving BPDUs on these interfaces.

So why did spanning tree fail here? An important detail to remember here is that spanning tree requires BPDUs sent between the switches in order to create a loop-free topology. BPDUs can be filtered because of MAC access-lists, VLAN access-maps or maybe something from the spanning tree toolkit?

Let’s check all of these:

SW1#show vlan access-map

SW2#show vlan access-map

SW3#show vlan access-map

There are no VLAN access maps on any of the switches. Any access-lists perhaps?

SW1#show access-lists

SW2#show access-lists

SW3#show access-lists

There are no access-lists…port-security maybe?

SW1#show port-security 
Secure Port  MaxSecureAddr  CurrentAddr  SecurityViolation  Security Action
                (Count)       (Count)          (Count)
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Total Addresses in System (excluding one mac per port)     : 0
Max Addresses limit in System (excluding one mac per port) : 6144

SW2#show port-security 
Secure Port  MaxSecureAddr  CurrentAddr  SecurityViolation  Security Action
                (Count)       (Count)          (Count)
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Total Addresses in System (excluding one mac per port)     : 0
Max Addresses limit in System (excluding one mac per port) : 6144

SW3#show port-security 
Secure Port  MaxSecureAddr  CurrentAddr  SecurityViolation  Security Action
                (Count)       (Count)          (Count)
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Total Addresses in System (excluding one mac per port)     : 0
Max Addresses limit in System (excluding one mac per port) : 6144

There’s no port security…what about spanning tree related commands?

SW2#show spanning-tree interface fa0/16 detail | include filter
Bpdu filter is enabled

SW2#show spanning-tree interface fa0/17 detail | include filter
Bpdu filter is enabled

We found something! BPDU filter has been enabled on the FastEthernet 0/16 and 0/17 interfaces of SW2. Because of this SW3 doesn’t receive BPDUs from SW2. Let’s get rid of this: