Troubleshooting CCNP TSHOOT Lab

In this lesson I’m going to walk you through a full lab that is based on the official Cisco CCNP TSHOOT topology. This lab is “broken” so that we have something to fix.

The official Cisco CCNP TSHOOT topology can be found here:

https://learningnetwork.cisco.com/thread/10965

I recreated this topology in GNS3 as best as I could. Cisco released the topology pictures but without any configurations. I recreated the configurations so it will probably be 95% the same as what you might encounter on the exam. You can find my GNS3 topology right here:

http://gns3vault.com/Faq/ccnp-tshoot-gns3-topology.html

Let’s troubleshoot one of those full labs! Before you continue reading I would highly suggest first giving this full lab a try yourself. If you get stuck you can continue reading and see how and why I solve it. Here’s the lab we are going to fix:

http://gns3vault.com/Troubleshooting/ccnp-tshoot-troubleshooting-1.html

Enough talking, let’s get labbing!

Its Monday morning 8 AM. You slept great and there’s a lovely croissant and cup of coffee waiting for you. Suddenly the phone rings…it’s your colleague!

His voice sounds like he’s in panic. There are users complaining that they can’t access the network and he is unable to solve the problems by himself…he seriously needs your help.

You throw your croissant out of the window…it’s time to show what a network troubleshooting rockstar looks like!

The first thing you do is grab a copy of the network diagram. You want to know the layer2 and layer3 topology so you don’t have to do everything off top of your head.

 

 

Here’s what the network diagrams look like:

 

 

Then you open your helpdesk application and you’ll find the following troubleshooting tickets waiting for you:

• Ticket #1: One of the users was working on Client1 but he’s complaining that there is no connectivity. He left a comment that he saw a message on the windows taskbar that said something like “no network connectivity”. One of your colleagues looked into the problem and told you it had probably something to do with DHCP.
• Ticket #2: After fixing the issue with Client1 you receive another ticket that users from VLAN 10 are complaining that they are unable to connect to the FTP server.
• Ticket #3: Your users are happy that they can connect to the FTP server but they are still unable to reach the webserver.
• Ticket #4: The IPv6 team left a ticket for you that they are unable to reach 2026::12:/122 from DSW1 or DSW2.

You decide to start with ticket #1. “no network connectivity” probably has something to do with the client not receiving an IP address.

Ticket 1

To solve this problem there are a number of devices we might have to check:

• DHCP server: The pool could be misconfigured, DHCP might not be running, maybe some of the interfaces are not working.
• Client: Maybe the client is configured to use a static IP address.
• ASW1, DSW1 and/or DSW2: These are devices that are in between our client and the DHCP server.

The ticket told us that client1 has an issue but didn’t tell us anything about other clients in VLAN 10 like client2. We can take a quick look if client2 can get a DHCP lease or not. If this is the case then we at least know the DHCP server is fine.

Let’s check this:

Client2#show ip interface brief 
Interface                 IP-Address      OK? Method Status                Protocol
FastEthernet0/0           unassigned      YES DHCP   up                    up

It seems client2 has the same issue, lucky for us nobody reported this issue yet.

Client1#show ip interface brief 
Interface                 IP-Address      OK? Method Status                Protocol
FastEthernet0/0           unassigned      YES DHCP   up                    up  

Client1 is configured for DHCP so to me this machine is looking good. If this were a real computer you might want to check other settings like the firewall. We will move our focus to the DHCP server:

R4#show ip dhcp server statistics 
Memory usage         23235
Address pools        1
Database agents      0
Automatic bindings   0
Manual bindings      0
Expired bindings     0
Malformed messages   0
Secure arp entries   0

Message              Received
BOOTREQUEST          0
DHCPDISCOVER         0
DHCPREQUEST          0
DHCPDECLINE          0
DHCPRELEASE          0
DHCPINFORM           0

Message              Sent
BOOTREPLY            0
DHCPOFFER            0
DHCPACK              0
DHCPNAK              0

First I’ll check if the DHCP server has received and sent any DHCP related messages. This doesn’t seem to be the case but doesn’t have to be an issue.
Maybe the router was rebooted and hasn’t received any DHCP discover messages since then. I do see that there is a DHCP pool. Let’s check if the DHCP service is running:

R4#show processes | include DHCP
  42 Mwe 6155B92C           0         43       0 5496/6000   0 DHCPD Timer     
 131 Mwe 61D4112C        1948      10172     19110568/12000  0 DHCPD Receive   
 150 Msi 61563C34           4         91      43 5124/6000   0 DHCPD Database

This router doesn’t support the “show ip sockets” command so I’ll take a look at the running processes. The DHCP service is running so we know this is not the issue. Let’s take a look at the DHCP pool:

R4#show ip dhcp pool 

Pool VLAN10 :
 Utilization mark (high/low)    : 100 / 0
 Subnet size (first/next)       : 0 / 0 
 Total addresses                : 254
 Leased addresses               : 0
 Pending event                  : none
 1 subnet is currently in the pool :
 Current index        IP address range                    Leased addresses
 10.2.10.1            10.2.10.1        - 10.2.10.254       0

We’ll take a look at the pool. The first thing that I notice is that the address range is incorrect. This pool is configured to use 10.2.10.0 /24 but VLAN 10 is using 10.2.1.0 /24. Let’s fix this:

R4(config)#ip dhcp pool VLAN10
R4(dhcp-config)#network 10.2.1.0 /24

This is looking better. The DHCP configuration itself is looking good now to me. Still, the DHCP server didn’t receive any DHCP discover messages. Let’s check if there is connectivity between R4 and DSW1 / DSW2. We’ll use a quick ping:

R4#ping 10.1.4.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.4.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms

R4#ping 10.1.4.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.4.10, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms

This is working fine so to me R4 is 100% operational when it comes to DHCP. This means we’ll have to move our focus to the switches in between the client and R4. I’ll start with ASW1:

ASW1#show ip int brief | include FastEthernet1/4 
FastEthernet1/4           unassigned      YES unset  up    up

First we’ll check the interface that is connected to client1. It seems to be up/up. Let’s check its switchport configuration:

ASW1#show interfaces fa1/4 switchport            
Name: Fa1/4
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: Disabled
Access Mode VLAN: 200 (VLAN0200)
Trunking Native Mode VLAN: 1 (default)
Trunking VLANs Enabled: ALL
Trunking VLANs Active: 200
Protected: false
Priority for untagged frames: 0
Override vlan tag priority: FALSE
Voice VLAN: none 
Appliance trust: none

Here’s an error. The interface is in access mode which is good but it has been assigned to VLAN 200. Client1 should be in VLAN 10:

ASW1(config)#interface fa 1/4
ASW1(config-if)#switchport access vlan 10

We’ll put the interface in VLAN 10. Now this probably doesn’t fix this error since client1 isn’t the only device that didn’t get an IP address through DHCP. Client2 was also having issues. We need to verify connectivity from ASW1 to DSW1 and DSW2. Let’s take another look at the network diagram:

This is the area of the topology where I will focus on. Client1 is connected to ASW1 and is in VLAN 10. VLAN 10 is configured on ASW1, ASW2, DSW1 and DSW3. R4 is our DHCP server but it’s not directly connected to VLAN 10, it’s in another subnet. This means that DSW1 and DSW2 require the IP helper command so that DHCP discover messages can be relayed to the the DHCP server. This is what we’ll have to check:

• Is VLAN 10 configured and operational on all switches? Even if only ASW1 and DSW1 would work, I still want to have redundancy so we’ll check DSW2 too.
• Is IP helper configured on DSW1 and DSW2 for VLAN 10?

We’ll start with IP helper. It’s only a single command so if this is the issue we’ll save a lot of time by not having to check all the interfaces between the switches.

DSW1#show ip interface vlan 10 | include Helper
  Helper address is not set

DSW2#show ip interface vlan 10 | include Helper
  Helper address is not set

That’s not looking good. DSW1 and DSW2 don’t have IP helper configured.

Let’s fix it:

DSW1(config)#interface vlan 10
DSW1(config-if)#ip helper-address 10.1.4.5

DSW2(config)#interface vlan 10
DSW2(config-if)#ip helper-address 10.1.4.9

This is how we configure IP helper. This would be a good moment to check if the clients are receiving IP addresses through DHCP.

Client2#show ip interface brief 
Interface                 IP-Address      OK? Method Status                Protocol
FastEthernet0/0           unassigned      YES DHCP   up                    up

Client1#show ip interface brief 
Interface                 IP-Address      OK? Method Status                Protocol
FastEthernet0/0           unassigned      YES DHCP   up                    up

Still no DHCP action. This looks like one of those Mondays where everything goes wrong…we’ll have to check all the interfaces between the switches:

ASW1#show etherchannel summary 
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        R - Layer3      S - Layer2
        U - in use
Group Port-channel  Ports
-----+------------+-----------------------------------------------------------
1     Po1(SU)     Fa1/2(P)   Fa1/3(P)   
2     Po2(SU)     Fa1/10(P)  Fa1/11(P) 

In between the switches we are using etherchannels. We can see that port-channel 1 and 2 are up and running. This doesn’t tell me anything about the VLANS though. Let’s take a closer look:

ASW1#show vlan-switch id 10

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
10   VLAN0010                         active    Fa1/4, Fa1/5

I can see that VLAN 10 is operational. Interface Fa1/4 and Fa1/5 are assigned to it. Let’s see if this VLAN is active on the trunk:

ASW1#show interfaces po1 trunk 

Port      Mode         Encapsulation  Status        Native vlan
Po1       on           802.1q         trunking      1

Port      Vlans allowed on trunk
Po1       1-9,11-4094

Port      Vlans allowed and active in management domain
Po1       1,20,200

Port      Vlans in spanning tree forwarding state and not pruned
Po1       1,200

ASW1#show interfaces po2 trunk 

Port      Mode         Encapsulation  Status        Native vlan
Po2       on           802.1q         trunking      1

Port      Vlans allowed on trunk
Po2       1-9,11-4094

Port      Vlans allowed and active in management domain
Po2       1,20,200

Port      Vlans in spanning tree forwarding state and not pruned
Po2       1,20

Now we are onto something! When we look at the trunk information of the Etherchannels we can see that VLAN 10 is not active. Let’s add it to the trunks:

ASW1(config)#interface po1
ASW1(config-if)#switchport trunk allowed vlan add 10

ASW1(config)#interface po2
ASW1(config-if)#switchport trunk allowed vlan add 10

This will add VLAN 10 to the etherchannels. Let’s check the other switches as well:

DSW1#show etherchannel summary 
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        R - Layer3      S - Layer2
        U - in use
Group Port-channel  Ports
-----+------------+-----------------------------------------------------------
1     Po1(SU)     Fa1/2(P)   Fa1/3(P)   
3     Po3(SU)     Fa1/4(P)   Fa1/5(P)   
5     Po5(SU)     Fa1/8(P)   Fa1/9(P)

DSW2#show etherchannel summary 
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        R - Layer3      S - Layer2
        U - in use
Group Port-channel  Ports
-----+------------+-----------------------------------------------------------
2     Po2(SU)     Fa1/10(P)  Fa1/11(P)  
3     Po3(SU)     Fa1/4(P)   Fa1/5(P)   
4     Po4(SU)     Fa1/6(P)   Fa1/7(P)

The etherchannels on DSW1 and DSW2 are operational. Next step is to check if they are in trunk mode and if VLAN 10 is allowed or not:

DSW1#show interfaces po1 trunk      

Port      Mode         Encapsulation  Status        Native vlan
Po1       on           802.1q         trunking      1

Port      Vlans allowed on trunk
Po1       1-4094

Port      Vlans allowed and active in management domain
Po1       1,10,20,200

Port      Vlans in spanning tree forwarding state and not pruned
Po1       1,10,20,200

DSW1#show interfaces po3 trunk

Port      Mode         Encapsulation  Status        Native vlan
Po3       on           802.1q         trunking      1

Port      Vlans allowed on trunk
Po3       1-4094

Port      Vlans allowed and active in management domain
Po3       1,10,20,200

Port      Vlans in spanning tree forwarding state and not pruned
Po3       1,10,20,200

Interface port-channel 1 is the one that is connected between ASW1 and DSW1. VLAN 10 is allowed on it so this should normally be enough. To make sure redundancy is intact I’m going to check the other interfaces as well:

DSW2#show interfaces po2 trunk 

Port      Mode         Encapsulation  Status        Native vlan
Po2       on           802.1q         trunking      1

Port      Vlans allowed on trunk
Po2       1-4094

Port      Vlans allowed and active in management domain
Po2       1,10,20,200

Port      Vlans in spanning tree forwarding state and not pruned
Po2       1,10,20,200

DSW2#show interfaces po3 trunk 

Port      Mode         Encapsulation  Status        Native vlan
Po3       on           802.1q         trunking      1

Port      Vlans allowed on trunk
Po3       1-4094

Port      Vlans allowed and active in management domain
Po3       1,10,20,200

Port      Vlans in spanning tree forwarding state and not pruned
Po3       10,20,200

We can verify that VLAN 10 is now operational on ASW1, DSW1 and DSW2. I didn’t bother with the interfaces that are connected to ASW2 because VLAN 10 is not active there. Did our hard work pay off?

Client2#show ip interface brief 
Interface                 IP-Address      OK? Method Status                Protocol
FastEthernet0/0           unassigned      YES DHCP   up                    up

Client1#show ip interface brief 
Interface                 IP-Address      OK? Method Status                Protocol
FastEthernet0/0           unassigned      YES DHCP   up                    up 

We fixed all layer 2 and DHCP related issues but the clients are still not getting an IP address. What else could be wrong?

Let’s check if our DHCP server is receiving the DHCP messages from our clients:

R4#show ip dhcp server statistics 
Memory usage         31953
Address pools        1
Database agents      0
Automatic bindings   2
Manual bindings      0
Expired bindings     0
Malformed messages   0
Secure arp entries   0

Message              Received
BOOTREQUEST          0
DHCPDISCOVER         216
DHCPREQUEST          0
DHCPDECLINE          0
DHCPRELEASE          0
DHCPINFORM           0

Message              Sent
BOOTREPLY            0
DHCPOFFER            216
DHCPACK              0
DHCPNAK              0

Great! At least R4 is receiving the DHCP discover messages from our clients. I don’t see any DHCP ACK messages so something somewhere is going wrong. Let’s do a debug:

R4#debug ip dhcp server packet

We’ll enable the debug to see DHCP packets.

R4# DHCPD: DHCPDISCOVER received from client 0063.6973.636f.2d63.3230.382e.3066.6134.2e30.3030.302d.4661.302f.30 through relay 10.2.1.2.
DHCPD: Sending DHCPOFFER to client 0063.6973.636f.2d63.3230.382e.3066.6134.2e30.3030.302d.4661.302f.30 (10.2.1.11).
DHCPD: unicasting BOOTREPLY for client c208.0fa4.0000 to relay 10.2.1.2.
DHCPD: DHCPDISCOVER received from client 0063.6973.636f.2d63.3230.382e.3066.6134.2e30.3030.302d.4661.302f.30 through relay 10.2.1.1.
DHCPD: Sending DHCPOFFER to client 0063.6973.636f.2d63.3230.382e.3066.6134.2e30.3030.302d.4661.302f.30 (10.2.1.11).
R4#
DHCPD: unicasting BOOTREPLY for client c208.0fa4.0000 to relay 10.2.1.1.

R4 is receiving and sending DHCP messages for VLAN 10 and they are relayed to IP address 10.2.1.1 (DSW1) and 10.2.1.2 (DSW2). Is R4 able to reach these IP addresses?

R4#ping 10.2.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.1.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R4#ping 10.2.1.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.1.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R4 is unable to reach these IP addresses so that’s why DHCP is not working now. Back to the drawing board:

Our topology picture shows up that DSW1, DSW2 and R4 are configured for EIGRP AS 10. R4 should learn about the 10.2.1.0 /24 prefix through EIGRP. Let’s see if R4 has any neighbors:

R4#show ip eigrp neighbors 
IP-EIGRP neighbors for process 10

R4 doesn’t have any EIGRP neighbors. Let’s check its EIGRP configuration:

R4#show ip protocols 
Routing Protocol is "eigrp 10"
  Outgoing update filter list for all interfaces is not set
  Incoming update filter list for all interfaces is not set
  Default networks flagged in outgoing updates
  Default networks accepted from incoming updates
  EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0
  EIGRP maximum hopcount 100
  EIGRP maximum metric variance 1
  Redistributing: eigrp 10, ospf 1
  EIGRP NSF-aware route hold timer is 240s
  Automatic network summarization is not in effect
  Maximum path: 4
  Routing for Networks:
    10.1.4.4/30
  Passive Interface(s):
    FastEthernet1/0
  Routing Information Sources:
    Gateway         Distance      Last Update
  Distance: internal 90 external 170

We’ll do a quick show ip protocols to see if we can find anything. There are two interesting things here:

• The passive interface for FastEthernet1/0 (that’s the interface to DSW1).
• No network command for 10.1.4.8 /30 (that’s the interface to DSW2).

Let’s fix these two issues:

R4(config)#router eigrp 10
R4(config-router)#no passive-interface fastEthernet 1/0
R4(config-router)#network 10.1.4.8 0.0.0.3

Let’s see if this made any difference:

R4#show ip eigrp neighbors 
IP-EIGRP neighbors for process 10
H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq
                                            (sec)         (ms)       Cnt Num
1   10.1.4.10               Fa2/0             11 00:00:34   12   200  0  19
0   10.1.4.6                Fa1/0             13 00:00:43   10   200  0  21

Seems we now have EIGRP neighbors! Let’s see if this solves our DHCP problem:

R4#show ip dhcp server statistics 
Memory usage         31953
Address pools        1
Database agents      0
Automatic bindings   2
Manual bindings      0
Expired bindings     0
Malformed messages   0
Secure arp entries   0

Message              Received
BOOTREQUEST          0
DHCPDISCOVER         340
DHCPREQUEST          4
DHCPDECLINE          0
DHCPRELEASE          0
DHCPINFORM           0

Message              Sent
BOOTREPLY            0
DHCPOFFER            340
DHCPACK              4
DHCPNAK              0

This is looking better! R4 is now sending DHCP ACKs so we know that the DHCP Offer makes it to the client. Let’s see if they finally got an IP address:

Client1#show ip interface brief 
Interface                 IP-Address      OK? Method Status                Protocol
FastEthernet0/0           10.2.1.11       YES DHCP   up                    up

Client2#show ip interface brief 
Interface                 IP-Address      OK? Method Status                Protocol
FastEthernet0/0           10.2.1.12       YES DHCP   up                    up  

Finally the clients are receiving IP addresses through DHCP! This ticket is solved.

To solve this ticket I tried to get a “quick win” by looking at the DHCP configuration on R4. After fixing it, it still wasn’t working. When this occurs it’s best to jump to the bottom of the OSI-model and check everything layer-by-layer. Start with the cabling, connectors and then move on to layer 2. Check the VLANs, interfaces, trunks, spanning tree, etc. When this is OK, check layer 3….IP addresses, routing, etc.

Let’s solve the next ticket!

Ticket 2

Here’s the ticket again:

• Ticket #2: After fixing the issue with Client1 you receive another ticket that users from VLAN 10 are complaining that they are unable to connect to the FTP server.

DHCP is working but it seems our clients are unable to reach the FTP server. This FTP server is in another subnet so traffic has to be routed. The first thing I would check is to see if they can reach their default gateway or not.