MTU Troubleshooting on Cisco IOS

Maximum Transmission Unit (MTU) is the largest size in bytes that a certain layer can forward. The MTU is different for each protocol and medium that we use. Ethernet for example has a MTU of 1500 bytes by default.

This means that a single Ethernet frame can carry up to 1500 bytes of data. On top of this data we add the Ethernet header. Typical header sizes are 14 bytes for Ethernet (add another 4 bytes if you use 802.1Q tagging).

Below is a picture with the different MTU fields that you might encounter:

MTU Header Sizes

You can see that a typical Ethernet header is 14 bytes, IP is 20 bytes and TCP is also 20 bytes. The maximum amount of payload that TCP can use is called the TCP MSS (Maximum Segment Size). This MSS value is the largest amount of data that a host can receive in a single TCP segment. This value is used to set a limit on the payload in order to prevent fragmentation and is sent in the SYN packet during the 3 way handshake. The MSS value isn’t synchronized between hosts, it can be different for each direction.

So why is all of this important to know? Let’s imagine we have an IP packet that is sent on our LAN. The size of the of the Ethernet frame will be like this:

  • 1460 bytes of payload for TCP.
  • 20 bytes for the TCP header.
  • 20 bytes for the IP header.
  • 14 bytes for the Ethernet header.

1460 (PAYLOAD) + 20 (TCP)  + 20 (IP) = 1500 bytes + 14 (ETHERNET) = 1514 bytes in total.

Sending 1514 bytes is no problem for Ethernet but some other mediums might have issues with large MTU values. Often problems arise when you add additional headers because of PPPoE, GRE Tunneling or IPSEC since this reduces the available bytes for our MTU. To demonstrate this problem (and how to solve it!) I will use a simple network with a reduced MTU. Here’s what it looks like:

r1 r2 webserver host

The network above has two routers, a webserver (S1) behind R1 and a client (H1) behind R2. Here’s what the configuration looks like:

Configuration

First we’ll configure some IP addresses:

R1(config)#interface fastEthernet 0/0
R1(config-if)#ip address 192.168.12.1 255.255.255.0

R1(config)#interface fastEthernet 0/1           
R1(config-if)#ip address 192.168.1.254 255.255.255.0
R2(config)#interface fastEthernet 0/0
R2(config-if)#ip address 192.168.12.2 255.255.255.0
R2(config)#interface fastEthernet 0/1
R2(config-if)#ip address 192.168.2.254 255.255.255.0

I’ll add some static routes for connectivity:

R1(config)#ip route 192.168.2.0 255.255.255.0 192.168.12.2
R2(config)#ip route 192.168.1.0 255.255.255.0 192.168.12.1

Here’s what the default MTU values look like:

R2#show interfaces fastEthernet 0/0 | include MTU
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, 

R2#show ip interface fastEthernet 0/0 | include MTU
  MTU is 1500 bytes

The first MTU value is the interface MTU, it’s 1500 bytes by default for Ethernet. The second one is the IP MTU which is also 1500 bytes. Once you get above 1500 bytes your router will start fragmenting the IP packets.

Is this limit of 1500 bytes really working? There’s an easy way to find out. Let’s do a ping with the DF-bit (Don’t Fragment) between the routers:

R2#ping
Protocol [ip]:   
Target IP address: 192.168.12.1
Repeat count [5]: 1
Datagram size [100]: 
Timeout in seconds [2]: 
Extended commands [n]: y
Source address or interface: 
Type of service [0]: 
Set DF bit in IP header? [no]: y
Validate reply data? [no]: 
Data pattern [0xABCD]: 
Loose, Strict, Record, Timestamp, Verbose[none]: v
Loose, Strict, Record, Timestamp, Verbose[V]: 
Sweep range of sizes [n]: y
Sweep min size [36]: 1495
Sweep max size [18024]: 1505
Sweep interval [1]: 
Type escape sequence to abort.
Sending 11, [1495..1505]-byte ICMP Echos to 192.168.12.1, timeout is 2 seconds:
Packet sent with the DF bit set
Reply to request 0 (1 ms) (size 1495)
Reply to request 1 (4 ms) (size 1496)
Reply to request 2 (1 ms) (size 1497)
Reply to request 3 (4 ms) (size 1498)
Reply to request 4 (1 ms) (size 1499)
Reply to request 5 (4 ms) (size 1500)
Request 6 timed out (size 1501)
Request 7 timed out (size 1502)
Request 8 timed out (size 1503)
Request 9 timed out (size 1504)
Request 10 timed out (size 1505)
Success rate is 54 percent (6/11), round-trip min/avg/max = 1/2/4 ms

In the ping above you can see that the largest packet that I can send is 1500 bytes. The second packet with 1501 bytes can’t be sent because it is too large and we set the DF-bit.

Let’s look at an actual packet between the client and the webserver, see what these values look like in an actual frame:

Wireshark TCP Syn Packet Maximum Segment Size

Above you can see the TCP MSS which is 1460 bytes. What else can we see in Wireshark?

Wireshark IP TCP Ethernet Bytes Length

Above you see that the total length of the IP packet is 1500 bytes (1460 bytes for TCP MSS + 40 bytes for the TCP/IP header). Ethernet adds another 14 bytes which is how we get to 1514 bytes in total.

To simulate a network with a lower MTU I will reduce the MTU of the FastEthernet 0/0 interface of R2:

R2(config)#interface fastEthernet 0/0
R2(config-if)#mtu 1400

By reducing the MTU to 1400 bytes, the largest TCP segment size will be only 1360 bytes (1400 – 40 = 1360).  This is a scenario where users often complain that sending a ping is no problem but accessing a website or something else doesn’t work. Why? Let’s look at the example below:

Wireshark ICMP Echo Reply 74 bytes

Above you see a ping between the client and webserver. As you can see the total length is only 74 bytes…no problem to send this because our MTU allows 1400 bytes. Now we will try to connect to the webserver from the client using HTTP:

Wireshark HTTP IP Packet 1500 bytes

This is where things go wrong…as you can see above the total length of the IP Packet is 1500 bytes which is 100 bytes above our MTU of 1400 bytes. It’s impossible to communicate between the client and webserver now.

How do we fix this? There are two solutions for this:

  • Set the correct IP MTU value so the router knows when to fragment IP packets.
  • Reduce the TCP MSS value for outgoing connections so there is less payload.

Here’s how to configure the correct IP MTU value:

We're Sorry, Full Content Access is for Members Only...

If you like to keep on reading, Become a Member Now! Here is why:

  • Learn any CCNA, CCNP and CCIE R&S Topic. Explained As Simple As Possible.
  • Try for Just $1. The Best Dollar You've Ever Spent on Your Cisco Career!
  • Full Access to our 654 Lessons. More Lessons Added Every Week!
  • Content created by Rene Molenaar (CCIE #41726)

540 Sign Ups in the last 30 days

satisfaction-guaranteed
100% Satisfaction Guaranteed!
You may cancel your monthly membership at any time.
No Questions Asked!

Tags: , , ,


Forum Replies

  1. Rene,
    Hi. Great explanations. Couple quick clarifications.

    1. You mention ‘The MSS value isn’t synchronized between hosts, it can be different for each direction’ - I thought this gets negotiated as a part of the 3 way handshake? Are you saying in your example above if say set “ip tcp adjust-mss 1360” on the f0/1 of R2 and then set “ip tcp adjust-mss 1460” on the f0/1 of R1 then it wouldnt negotatiate down to the lowest common MSS and hence break?

    2. Assuming I am using GRE over IPSEC with transport mode I looked at a recent customer network in which the tunnel i

    ... Continue reading in our forum

  2. Hi Rene,

    As you mention the interface MTU for both end doesn’t need to be same.

    If let say R1 with interface MTU 9000 and R2 with interface MTU 1500. It will not cause any issue?

    Because the R1 sending 6 times more packet in one frame, R2 will able to receive it without any problem?

    Davis

  3. Hello Sameer.

    The quick answer is yes. Fragmentation will occur during encapsulation. If a PC is preparing an email to send, it will take the transport segment and encapsulate it into an IP packet. An IP packet can theoretically be up to 65535 bytes long. So during encapsulation from the Network to the Data Link layer on the PC, if an IP packet is larger than the allowable 1500 bytes

    ... Continue reading in our forum

  4. Got a few questions-

    1. You mention that it is no problem when the frame reaches 1514 bytes, but realistically how high can this go before it starts causing an issue on Ethernet?

    2. Why did it cause a problem communicating with the web server when you reduced the MTU to 1400? Should it not have just fragmented it? Is the DF bit used in HTTP? I mean it would slow things down a bit to fragment, but not stop them working altogether?

    3. What happens if one side has an MTU of 1400, and the other 1500? Does the side set to 1400 have an issue receiving packets over this

    ... Continue reading in our forum

  5. ... Continue reading in our forum

124 more replies! Ask a question or join the discussion by visiting our Community Forum