TCP tuning
Encyclopedia
TCP tuning techniques adjust the network congestion avoidance parameters of TCP
connections over high-bandwidth, high-latency networks. Well-tuned networks can perform up to 10 times faster in some cases.
(BDP) is a term primarily used in conjunction with the TCP
to refer to the number of bytes necessary to fill a TCP "path", i.e. it is equal to the maximum number of simultaneous bits in transit between the transmitter and the receiver.
High performance networks have very large BDPs. To give a practical example, two nodes communicating over a geostationary satellite link with a round trip delay of 0.5 seconds and a bandwidth of 10 Gbit/s can have up to 0.5×1010 bits, i.e., 5 Gbit = 625 MB
of unacknowledged data in flight. Despite having much lower latencies than satellite links, even terrestrial fiber links can have very high BDPs because their link capacity is so large. Operating systems and protocols designed as recently as a few years ago when networks were slower were tuned for BDPs of orders of magnitude smaller, with implications for limited achievable performance.
s of up to 64 kBytes (64 KiB
), which was adequate for slow links or links with small round trip times (RTTs). Larger buffers are required by the high performance options described below.
Buffering is used throughout high performance network systems to handle delays in the system. In general, buffer size will need to be scaled proportional to the amount of data "in flight" at any time. For very high performance applications that are not sensitive to network delays, it is possible to interpose large end to end buffering delays by putting in intermediate data storage points in an end to end system, and then to use automated and scheduled non-real-time data transfers to get the data to their final endpoints.
Receive Window) is the amount of data
that a computer
can accept without acknowledging the sender. If the sender has not received acknowledgement for the first packet it sent, it will stop and wait and if this wait exceeds a certain limit, it may even retransmit
. This is how TCP achieves reliable data transmission
.
Even if there is no packet loss in the network, windowing can limit throughput. Because TCP transmits data up to the window size before waiting for the acknowledgements, the full bandwidth of the network may not always get used. The limitation caused by window size can be calculated as follows:
where RWIN is the TCP Receive Window and RTT is the round-trip time
for the path.
At any given time, the window advertised by the receive side of TCP corresponds to the amount of free receive memory it has allocated for this connection. Otherwise it would take the risk to have to drop received packets by lack of space.
Unrelated to the TCP receive window, the sending side should also allocate the same amount of memory as the receive side for good performance. That is because, even after data has been sent on the network, the sending side must hold it in memory until it has been acknowledged as successfully received, just in case it would have to be retransmitted. If the receiver is far away, acknowledgments will take a long time to arrive. If the send memory is small, it can saturate and block emission. A simple computation gives the same optimal send memory size as for the receive memory size given above.
where MSS is the maximum segment size and Ploss is the probability of packet loss.
TCP timestamps (RFC 1323) play a double role: they avoid ambiguities due to the 32-bit sequence number field wrapping around, and they allow more precise RTT estimation in the presence of multiple losses per RTT. With those improvements, it becomes reasonable to increase the TCP window beyond 64 kB, which can be done using the window scaling option (RFC 1323).
The TCP selective acknowledgment options (SACK, RFC 2018) allows a TCP receiver to precisely inform the TCP server about which segments have been lost. This increases performance on high-RTT links, when multiple losses per window are possible.
Path MTU discovery
avoids the need for in-network fragmentation, which increases performance in the presence of losses.
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
connections over high-bandwidth, high-latency networks. Well-tuned networks can perform up to 10 times faster in some cases.
Bandwidth-delay product (BDP)
Bandwidth-delay productBandwidth-delay product
In data communications, bandwidth-delay product refers to the product of a data link's capacity and its end-to-end delay . The result, an amount of data measured in bits , is equivalent to the maximum amount of data on the network circuit at any given time, i.e. data that has been transmitted but...
(BDP) is a term primarily used in conjunction with the TCP
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
to refer to the number of bytes necessary to fill a TCP "path", i.e. it is equal to the maximum number of simultaneous bits in transit between the transmitter and the receiver.
High performance networks have very large BDPs. To give a practical example, two nodes communicating over a geostationary satellite link with a round trip delay of 0.5 seconds and a bandwidth of 10 Gbit/s can have up to 0.5×1010 bits, i.e., 5 Gbit = 625 MB
Megabyte
The megabyte is a multiple of the unit byte for digital information storage or transmission with two different values depending on context: bytes generally for computer memory; and one million bytes generally for computer storage. The IEEE Standards Board has decided that "Mega will mean 1 000...
of unacknowledged data in flight. Despite having much lower latencies than satellite links, even terrestrial fiber links can have very high BDPs because their link capacity is so large. Operating systems and protocols designed as recently as a few years ago when networks were slower were tuned for BDPs of orders of magnitude smaller, with implications for limited achievable performance.
Buffers
The original TCP configurations supported bufferBuffer (computer science)
In computer science, a buffer is a region of a physical memory storage used to temporarily hold data while it is being moved from one place to another. Typically, the data is stored in a buffer as it is retrieved from an input device or just before it is sent to an output device...
s of up to 64 kBytes (64 KiB
Kibibyte
The kibibyte is a multiple of the unit byte for quantities of digital information. The binary prefix kibi means 1024; therefore, 1 kibibyte is . The unit symbol for the kibibyte is KiB. The unit was established by the International Electrotechnical Commission in 1999 and has been accepted for use...
), which was adequate for slow links or links with small round trip times (RTTs). Larger buffers are required by the high performance options described below.
Buffering is used throughout high performance network systems to handle delays in the system. In general, buffer size will need to be scaled proportional to the amount of data "in flight" at any time. For very high performance applications that are not sensitive to network delays, it is possible to interpose large end to end buffering delays by putting in intermediate data storage points in an end to end system, and then to use automated and scheduled non-real-time data transfers to get the data to their final endpoints.
TCP speed limits
Maximum achievable throughput for a single TCP connection is determined by different factors. One trivial limitation is the maximum bandwidth of the slowest link in the path. But there are also other, less obvious limits for TCP throughput. Bit errors can create a limitation for the connection as well as round-trip time.Window size
In computer networking, RWIN (TCPTransmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
Receive Window) is the amount of data
Data (computing)
In computer science, data is information in a form suitable for use with a computer. Data is often distinguished from programs. A program is a sequence of instructions that detail a task for the computer to perform...
that a computer
Computer
A computer is a programmable machine designed to sequentially and automatically carry out a sequence of arithmetic or logical operations. The particular sequence of operations can be changed readily, allowing the computer to solve more than one kind of problem...
can accept without acknowledging the sender. If the sender has not received acknowledgement for the first packet it sent, it will stop and wait and if this wait exceeds a certain limit, it may even retransmit
Retransmission (data networks)
Retransmission, essentially identical with Automatic repeat request , is the resending of packets which have been either damaged or lost. It is a term that refers to one of the basic mechanisms used by protocols operating over a packet switched computer network to provide reliable communication...
. This is how TCP achieves reliable data transmission
Data transmission
Data transmission, digital transmission, or digital communications is the physical transfer of data over a point-to-point or point-to-multipoint communication channel. Examples of such channels are copper wires, optical fibres, wireless communication channels, and storage media...
.
Even if there is no packet loss in the network, windowing can limit throughput. Because TCP transmits data up to the window size before waiting for the acknowledgements, the full bandwidth of the network may not always get used. The limitation caused by window size can be calculated as follows:
where RWIN is the TCP Receive Window and RTT is the round-trip time
Round-trip delay time
In telecommunications, the round-trip delay time or round-trip time is the length of time it takes for a signal to be sent plus the length of time it takes for an acknowledgment of that signal to be received...
for the path.
At any given time, the window advertised by the receive side of TCP corresponds to the amount of free receive memory it has allocated for this connection. Otherwise it would take the risk to have to drop received packets by lack of space.
Unrelated to the TCP receive window, the sending side should also allocate the same amount of memory as the receive side for good performance. That is because, even after data has been sent on the network, the sending side must hold it in memory until it has been acknowledged as successfully received, just in case it would have to be retransmitted. If the receiver is far away, acknowledgments will take a long time to arrive. If the send memory is small, it can saturate and block emission. A simple computation gives the same optimal send memory size as for the receive memory size given above.
Packet loss
When packet loss occurs in the network, an additional limit is imposed on the connection. The limit can be calculated according to the formula (Mathis et al.):where MSS is the maximum segment size and Ploss is the probability of packet loss.
TCP Options for High Performance
A number of extensions have been made to TCP over the years to increase its performance over fast high-RTT links ("long fat networks", or LFNs for short).TCP timestamps (RFC 1323) play a double role: they avoid ambiguities due to the 32-bit sequence number field wrapping around, and they allow more precise RTT estimation in the presence of multiple losses per RTT. With those improvements, it becomes reasonable to increase the TCP window beyond 64 kB, which can be done using the window scaling option (RFC 1323).
The TCP selective acknowledgment options (SACK, RFC 2018) allows a TCP receiver to precisely inform the TCP server about which segments have been lost. This increases performance on high-RTT links, when multiple losses per window are possible.
Path MTU discovery
Path MTU discovery
Path MTU Discovery is a standardized technique in computer networking for determining the maximum transmission unit size on the network path between two Internet Protocol hosts, usually with the goal of avoiding IP fragmentation...
avoids the need for in-network fragmentation, which increases performance in the presence of losses.
External links
- RFC 1323 - TCP Extensions for High Performance
- RFC 2018 - TCP Selective Acknowledgment Options
- RFC 2582 - The NewReno Modification to TCP's Fast Recovery Algorithm
- RFC 2883 - An Extension to the Selective Acknowledgment (SACK) Option for TCP
- RFC 3517 - A Conservative Selective Acknowledgment-based Loss Recovery Algorithm for TCP
- RFC 4138 - Forward RTO-Recovery (F-RTO): An Algorithm for Detecting Spurious Retransmission Timeouts with TCP and the Stream Control Transmission Protocol (SCTP)
- TCP Tuning Guide, ESnet
- The Cable Guy: TCP Receive Window Auto-Tuning
- The Web100 Data Bandwidth Testing
- DrTCP - a utility for Microsoft WindowsMicrosoft WindowsMicrosoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
(prior to VistaWindows VistaWindows Vista is an operating system released in several variations developed by Microsoft for use on personal computers, including home and business desktops, laptops, tablet PCs, and media center PCs...
) which can quickly alter TCPTransmission Control ProtocolThe Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
performance parameters in the registry. - Information on 'Tweaking' your TCP stack, Broadband Reports
- TCP/IP Analyzer, speedguide.net
- NTTTCP Network Performance Test Tool, Microsoft Windows Server Performance Team Blog