Traceroute

Traceroute, like the ping command can be used to isolate problems in our network. The ping command is a bit limited sometimes. For example, take a look at the following topology:

When we send a ping from H1 (192.168.1.1) to S1 (192.168.3.1) and this ping doesn’t work, what does it mean? We’ll know something is not working but we don’t know whether the problem is in between H1-R1, R1-R2, R2-R3 or R3-S1.

If you know the IP addresses of all routers in the path then you could ping all of these routers one by one. What if you have no idea how many routers are in between? Or if you don’t know their IP addresses?

The traceroute command will help us with that. Here’s a quick example:

C:\Users\renemolenaar>tracert facebook.com

Tracing route to facebook.com [31.13.64.35]
over a maximum of 30 hops:

  1     9 ms     8 ms     8 ms  D5644061.static.ziggozakelijk.nl [215.99.12.17]
  2     9 ms    14 ms     8 ms  tb-rc0001-cr101-ae40-0.core.as9143.net [213.51.158.66]
  3    10 ms    12 ms    11 ms  asd-tr0042-cr101-ae5-0.core.as9143.net [213.51.158.18]
  4    14 ms    12 ms    12 ms  br02.ams1.tfbnw.net [80.249.209.164]
  5    12 ms    12 ms    13 ms  po131.asw01.ams3.tfbnw.net [157.240.35.46]
  6    12 ms    12 ms    13 ms  po232.psw01c.amt2.tfbnw.net [157.240.35.175]
  7    13 ms    12 ms    12 ms  msw1ad.01.amt2.tfbnw.net [173.252.64.255]
  8    12 ms    11 ms    13 ms  edge-star-mini-shv-01-amt2.facebook.com [31.13.64.35]

Trace complete.

Above I used the tracert (traceroute) command on a Windows computer to trace the path from my computer to facebook.com. You can see all the IP addresses and hostnames of the routers in between my computer and the server that responds to www.facebook.com. The response times that you see is the round trip time from my computer to the router. For each router, traceroute sends three probes.

So, how does traceroute work?

Traceroute uses the TTL (Time to Live) field in the IP packet header. Normally, TTL is used to prevent packets from being forwarded forever when there is a routing loop. Whenever an IP packet is forwarded by a router, the TTL is decreased by one. When the TTL is zero, the IP packet will be discarded.

How do we use this for traceroute? Let’s look at an example. Let’s say that from H1 (192.168.1.1) we send a trace to S1 (192.168.3.1). The first IP packet that H1 sends, will have a TTL of 1:

When R1 receives the IP packet, it will want to forward it to R2 but it has to decrease the TTL from one to zero, as a result, the IP packet will be dropped and R1 will respond to H1 with a TTL exceeded message. H1 will now send a second packet with a TTL of 2:

R1 will decrease the TTL from two to one, forwards it and now R2 has to drop it. R2 will respond with a TTL exceeded message. H1 will now send another IP packet with a TTL of 3:

R1 will decrease the TTL from three to two, R2 decrease it from two to one and R3 will have to drop it. R3 sends the TTL exceeded message to R1. The last IP packet that H1 will send will have a TTL of four:

Each router will decrease the TTL by one, our server on the other end will receive an IP packet with a TTL of one and replies with an ICMP reply to H1. We now know that the destination is reachable and we have learned all routers in our path.

Each IP packet that we send is called a probe. Traceroute can be used with ICMP, UDP and TCP, depending on your operating system.

Traceroute Command

Let’s see this in action. I will use the following topology for this demonstration:

Above we have two host computers. H1 is a Windows computer (192.168.1.1) and H2 a Linux computer (192.168.1.2) running Ubuntu. The routers have been configured with EIGRP so that we have full connectivity.

Windows

Let’s start with a windows trace:

C:\Users\vmware>tracert 192.168.3.1

Tracing route to 192.168.3.1 over a maximum of 30 hops

  1     1 ms     1 ms    <1 ms  192.168.1.254
  2     1 ms     1 ms     1 ms  192.168.12.2
  3     1 ms     1 ms     1 ms  192.168.23.3
  4     1 ms    <1 ms    <1 ms  192.168.3.1

Trace complete.

Above we see all routers and the destination. For each hop, traceroute will send three IP packets. We do this to get a good average of the round trip time for each hop. Let’s see what the IP packets look like in Wireshark. Here’s the first one:

Above we see the first IP packet with a TTL of one. We can see that this is an ICMP request. When R1 receives this IP packet, it will respond like this:

Above, we see that R1 responds with a TTL expired message to H1. In the capture file, you will see the two packets above three times. Let’s look at the next IP packet:

This one has a TTL of two which will make it to R2:

And the third packet with a TTL of three:

R3 will respond to it with a TTL exceeded message:

Finally, the last IP packet with a TTL of four will be sent by H1:

This one finally makes it to S1 so we get an ICMP reply:

That’s how traceroute works on Windows. You can take a look at the capture file yourself if you want:

Windows Traceroute

Linux

The traceroute commands on Linux works similar to Windows. One important difference is that it doesn’t use ICMP but UDP. It also allows you to specify the number of IP packets (probes) you want to send. To create a nice clean wireshark capture, I configured it to send only one probe for each hop:

$ traceroute -N 1 -q 1 192.168.3.1
traceroute to 192.168.3.1 (192.168.3.1), 30 hops max, 60 byte packets
 1  192.168.1.254 (192.168.1.254)  1.202 ms
 2  192.168.12.2 (192.168.12.2)  1.122 ms
 3  192.168.23.3 (192.168.23.3)  1.192 ms
 4  192.168.3.1 (192.168.3.1)  1.886 ms

Here’s what this traceroute looks like in Wireshark:

Above we can see that H2 is sending UDP packets instead of ICMP requests. Let’s take a closer look at these packets. Here’s the first one:

Above we see that H2 is sending the first IP packet with a TTL of one. We see we use UDP and the destination port number is 33434. R1 will reply to this:

Since the TTL is zero, R1 drops it and replies with a TTL exceeded message. Time for the second IP packet with a TTL of two:

The packet above is the same as the first one except you can see that the destination port number has increased from 33434 to 33435. R2 will drop this:

Time for packet with a TTL of three:

Above you can see the destination port number has increased from 33435 to 33436. R3 will drop this:

The last packet will have a TTL of four and a destination port number of 33437:

This one will make it all the way to S1. Since S1 is not listening on any of these UDP port numbers, it will reply that the destination / port is unreachable:

This tells H2 that we made it to the destination.

Linux Traceroute

Cisco IOS

We can also use the traceroute command on Cisco IOS. Like Linux, Cisco uses UDP for traceroute. Let’s do a trace from R1 to S1, I will use a single probe:

R1#traceroute 192.168.3.1 probe 1
Type escape sequence to abort.
Tracing the route to 192.168.3.1
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 0 msec
  2 192.168.23.3 4 msec
  3 192.168.3.1 0 msec

Above we see that this trace makes it to the destination.

Here’s what the capture looks like in Wireshark:

This capture is the same as the Linux capture we just looked at so I won’t add the screenshots again. You will find the capture file at the end of this lesson if you want to see it for yourself.

Let’s look at some of the options of the traceroute command instead:

R1#traceroute ?
  WORD       Trace route to destination address or hostname
  appletalk  AppleTalk Trace
  clns       ISO CLNS Trace
  ethernet   Ethernet Traceroute
  ip         IP Trace
  ipv6       IPv6 Trace
  ipx        IPX Trace
  mpls       MPLS LSP Trace
  <cr>

We can choose if we want to trace an IP address or a hostname. If you don’t specify anything, then you can choose some parameters that you want to use:

R1#traceroute 
Protocol [ip]: 
Target IP address: 192.168.3.1
Source address: 
Numeric display [n]: y
Timeout in seconds [3]: 
Probe count [3]: 
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Port Number [33434]: 
Loose, Strict, Record, Timestamp, Verbose[none]: 
Type escape sequence to abort.
Tracing the route to 192.168.3.1
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 0 msec 4 msec 0 msec
  2 192.168.23.3 4 msec 0 msec 4 msec
  3 192.168.3.1 0 msec 0 msec 0 msec

You can also add these parameters behind the IP address, like this:

R1#traceroute 192.168.3.1 ?       
  numeric  display numeric address
  port     specify port number
  probe    specify number of probes per hop
  source   specify source address or name
  timeout  specify time out
  ttl      specify minimum and maximum ttl
  <cr>

Like I did when I specified the number of probes:

R1#traceroute 192.168.3.1 probe 1 

Here you can find the capture file if you want to see it for yourself:

Cisco IOS Traceroute

Troubleshooting

You have now seen the output of the traceroute command when everything worked as it should. Let’s look at two examples where we don’t have end-to-end connectivity. I only need these three routers for this example:

Destination Unreachable

Let’s start with a destination that is unreachable. I will add a new loopback on R3 with an IP address:

R3(config)#interface loopback 0
R3(config-if)#ip address 3.3.3.3 255.255.255.255

Right now, none of our routers know how to reach 3.3.3.3. Let’s see what the traceroute output looks like:

R1#traceroute 3.3.3.3
Type escape sequence to abort.
Tracing the route to 3.3.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1  *  *  * 
  2  *  *  * 
  3  *  *  * 

Above we see that none of the probes make it to the destination. Let’s add a static route on R1 so that this traffic makes it to R2 at least:

R1(config)#ip route 3.3.3.3 255.255.255.255 192.168.12.2

And do another traceroute:

R1#traceroute 3.3.3.3
Type escape sequence to abort.
Tracing the route to 3.3.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 4 msec 0 msec 4 msec
  2 192.168.12.2 !H  *  !H 

Now we can see that R1 is able to forward it, R2 responds that it’s unreachable. Let’s add the static route on R2:

R2(config)#ip route 3.3.3.3 255.255.255.255 192.168.23.3

And try one more trace:

R1#traceroute 3.3.3.3
Type escape sequence to abort.
Tracing the route to 3.3.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 0 msec 4 msec 0 msec
  2 192.168.23.3 4 msec *  0 msec

Now we are able to make it to the destination.

Source Unreachable

We can use traceroute to figure out if we can reach a certain destination, but you can also use it to check if other routers know about your source. Let’s add another loopback interface, this time on R1:

R1(config)#interface loopback 0
R1(config-if)#ip address 1.1.1.1 255.255.255.255

I will now do a traceroute to 192.168.3.1 with 1.1.1.1 as our source. We know that 192.168.3.1 is reachable since we tried it before. Let’s take a look:

R1#traceroute 192.168.3.1 source loopback 0
Type escape sequence to abort.
Tracing the route to 192.168.3.1
VRF info: (vrf in name/id, vrf out name/id)
  1  *  *  * 
  2  *  *  * 
  3  *  *  * 

This trace is failing. Nobody knows how to reach 1.1.1.1. Let’s add a static route on R2:

R2(config)#ip route 1.1.1.1 255.255.255.255 192.168.12.1

R2 now knows how to reach 1.1.1.1. Let’s try that traceroute again:

R1#traceroute 192.168.3.1 source loopback 0
Type escape sequence to abort.
Tracing the route to 192.168.3.1
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 0 msec 4 msec 0 msec
  2  *  *  * 
  3  *  *  * 

R2 is now responding. This tells us that the problem is not between R1 and R2 but somewhere further down the line. Let’s add the last static route:

R3(config)#ip route 1.1.1.1 255.255.255.255 192.168.23.2

R3 now also knows how to get to 1.1.1.1. Let’s try that trace:

R1#traceroute 192.168.3.1 source loopback 0
Type escape sequence to abort.
Tracing the route to 192.168.3.1
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 0 msec 4 msec 0 msec
  2 192.168.23.3 4 msec 0 msec 0 msec
  3 192.168.3.1 0 msec 0 msec 4 msec

Which now completes successfully.

Conclusion

You have now learned how traceroute uses the TTL (Time to Live) field in the IP packet to send probes to the destination, allowing us to discover the path from the source to the destination. You have also seen how Windows uses ICMP and Linux uses UDP for traceroute. We have also looked at how you can use traceroute to troubleshoot network reachability issues where the source and/or destination is unknown.