Bandwidth Delay Product

TCP is one of those protocols that we usually don’t think about too much. As network engineers we are busy working with network devices like routers or switches. TCP is one of those protocols that is used most between hosts or servers and it works without giving it much thought. It establishes connections, transmits data, sends acknowledgments and when something goes wrong…it retransmits it.

TCP uses a sliding window size that indicates how much the receiver is willing to receive from the sender. Depending on the receive buffer and network conditions, this window size will increase or decrease as needed. The larger the window size, the higher the throughput will be. With a window size of 1, the receiver would send an acknowledgment for each segment that it receives which results in a lot of overhead.

Host to server satellite connection

This “stop and go” mechanism of TCP works very well “out of the box” but on certain links, TCP might require some tuning. This is especially true on so called long fat networks (LFN).

A LFN is a network that offers a high bandwidth but also a very high delay. An example could be a satellite connection. These connections offer a high bandwidth but the delay is also quite high since you have to send your signal 22000 miles up to the satellite and another 22000 miles down to reach the receiver. You can expect a round trip time anywhere between about 500-1000 ms.

The problem here is that when the sender sends some data, it has to be wait a very long time for an acknowledgment of the receiver before it can send the next data. During the time we are waiting, nothing happens so we don’t utilize the full bandwidth of our link.

The throughput of TCP is limited by the round trip time of the link and the window size. We can’t change the round trip time but we can play with the window size. Take a look at the image below:

TCP Long Fat Network Low Window Size

Imagine we send some data from the host to the server, when this piece of data is on its way we have to wait a long time before it reaches the server and for the acknowledgment to come back. A lot of bandwidth is wasted. This is what happens with a large window size:

TCP long fat network high window size

With a large window size, we can fill the entire “pipeline” with data. We don’t waste anything.

When you are using a 5 Mbit satellite link and you have a transmission rate of 1 or 2 Mbit of TCP traffic, you probably have some TCP tuning to do.

The most optimal window size depends on the bandwidth and delay of the link, we call this the bandwidth delay product. We can calculate it with the following formula:

Bandwidth Delay Product = bandwidth (bits per sec) * round trip time (in seconds)

So for example, let’s calculate the bandwidth delay product of a satellite link that has a round trip time of 500 ms:

5000000 bits * 0.5 seconds = bandwidth delay product 2500000

So our bandwidth delay product is 2500000 bits. The window size is typically configured in bytes so 2500000 / 8 would be 312500 bytes.

Here are some other examples:

ADSL 2 Mbit with 50 ms round trip time:

2000000 bits * 0.05 seconds = bandwidth delay product 100000 bits (or 12500 bytes)
ADSL2 20 Mbit with 50 ms round trip time:

20000000 bits * 0.05 seconds = bandwidth delay product 1000000 bits (or 125000 bytes)
FastEthernet LAN Interface with 1 ms round trip time:

100000000 bits * 0.001 seconds = bandwidth delay product 100000 bits (or 12500 bytes)
Gigabit LAN Interface with 1 ms round trip time:

1000000000 bits * 0.001 seconds = bandwidth delay product 1000000 bits (or 125000 bytes)

Are there any downsides to increasing the TCP window size? One thing to consider is that by increasing the window size, you also need a large receive buffer but this
shouldn’t be much of a problem on any modern hardware. Also with a larger window size you will have a lot of data “in transit” so if you have any errors on the link,
there’s a lot of data to retransmit.

iPerf Demonstration

Once you have calculated the bandwidth delay product, you should test if it works. A nice way to test this is by using iPerf. This application allows you to generate TCP traffic with different window sizes. To demonstrate this, I’ll use two hosts:

Iperf Client Server Switch

These two hosts are connected through a gigabit link so this is a high bandwidth low delay link. Even though the round trip time is low, we still have to use a decent window size to get some decent performance.

A quick ping tells us the round trip time:

We're Sorry, Full Content Access is for Members Only...

If you like to keep on reading, Become a Member Now!

  • Learn any CCNA, CCNP and CCIE R&S Topic. Explained As Simple As Possible.
  • Try for Just $1. The Best Dollar You’ve Ever Spent on Your Cisco Career!
  • Full Access to our 800 Lessons. More Lessons Added Every Week!
  • Content created by Rene Molenaar (CCIE #41726)
518 Sign Ups in the last 30 days
satisfaction-guaranteed

  • 100% Satisfaction Guaranteed!
  • You may cancel your monthly membership at any time.
  • No Questions Asked!

Tags:


Forum Replies

  1. Hi Rene,
    Thank you for this article, I work for satellite broadband provider and i deal with high latency networks daily, i am currently troubleshooting an issue where customer is disputing his usage of 5Gig , Cisco netflow is reporting that 5Gig of Apple iOS updates, and customer has only updated his iPhone, i am planning to cap few iOS updates and your article will definitely help. I liked the way you wrote this article in sequence after TCP header and then window scaling.
    Do you mind giving an option sort your articles by date posted.

    Regards,
    Bandu

  2. Hi Rene,

    I saw that you double the window size to 128000 and 256000 etc, and I know that window size filed on TCP header with size of 16 byte which mean the maximum window size = 65536 bits ( maximum IP packet size ), so how is it possible to double up over than that value ??

  3. Hi Bandu,

    You are welcome, glad to hear you like it. Discussions like you have with your customer can be difficult, netflow doesn’t lie and Apple iOS updates can be pretty large. For the customer it’s just an update but they probably don’t realize that it’s a few GB. Sounds like a good idea to cap it or to warn them in time when there is a lot of traffic for a certain duration.

    I’ll see if I can create a “latest post” page. I also send regular updates when I have finished some new posts.

    Rene

  4. Hi Rene,

    Is this only applicable if the server and the client on the same network ? what if the client is behind a router and the server is also behind a router ? And finally how can we apply this to all traffic sourcing from a router at site A going via satellite link to a remote router at site B ? Thanks for this great article!

    Ahmad

31 more replies! Ask a question or join the discussion by visiting our Community Forum