Jumbo Frames
Encyclopedia
In computer networking, jumbo frames are Ethernet frame
s with more than 1500 bytes of payload. Conventionally, jumbo frames can carry up to 9000 bytes of payload, but variations exist and some care must be taken when using the term. Many Gigabit Ethernet
switches and Gigabit Ethernet network interface cards support jumbo frames. Some Fast Ethernet
switches and Fast Ethernet network interface cards also support jumbo frames. Most national research and education networks (such as Internet2
/NLR, ESnet, GÉANT
, and AARNet
) support jumbo frames, but most commercial Internet service providers do not.
, although this does not reduce the receiver CPU load. Interrupt-combining Ethernet chipsets, however, do provide most of the same gain for the receiver, and work without special consideration and without requiring all stations to support jumbo frames. Zero-copy
NICs and device drivers, when combined with interrupt combining, can provide effectively all the gains of jumbo frames without the re-send costs, and without requiring any changes to other stations on the network.
Jumbo frames gained initial prominence when Alteon WebSystems
introduced them in their ACEnic Gigabit Ethernet
adapters.
Many other vendors also adopted the size; however, they did not become part of the official IEEE 802.3
Ethernet standard.
. The presence of Jumbo frames may have an adverse effect on network latency, especially on low bandwidth links.
The use of 9000 bytes as preferred size for jumbo frames arose from discussions within the Joint Engineering Team of Internet2
and the U.S. federal government networks. Their recommendation has been adopted by all other national research and education networks. In order to meet this mandatory purchasing criterion, manufacturers have in turn adopted 9000 bytes as the conventional jumbo frame size.
Internet Protocol
subnetworks require that all hosts in a subnet have an identical MTU. As a result, interfaces using the standard frame size and interfaces using the jumbo frame size should not be in the same subnet. To reduce interoperability issues, network interface cards capable of jumbo frames require explicit configuration to use jumbo frames.
IETF solutions for adopting Jumbo Frames avoids the data integrity reductions through use of the Castagnoli CRC polynomial being implemented within the SCTP transport (RFC 4960), and iSCSI (RFC 3720). Selection of this polynomial was based upon work documented in the paper "32-Bit Cyclic Redundancy Codes for Internet Applications". The Castagnoli polynomial 0x11EDC6F41 achieves the Hamming Distance HD=6 beyond one Ethernet MTU (to a 16,360 bit data word length) and HD=4 to 114,663 bits, which is more than 9 times the length of an Ethernet MTU. This gives two additional bits of error detection ability at MTU-sized data words compared to the Ethernet CRC standard polynomial while not sacrificing HD=4 capability for data word sizes up to and beyond 72k bits.
By using a CRC checksum rather than simple additive checksums as contained within the UDP and TCP transports, errors generated internal to NICs can be detected as well. Both TCP and UDP have proven ineffective at detecting bus specific bit errors, since these errors with simple summations tend to be self cancelling. Testing that led to adoption of RFC 3309 compiled evidence based upon simulated error injection against real data that demonstrated as much as 2% of these errors were not being detected.
One of the major impediments toward the adoption of Jumbo Frames has been the inability to upgrade existing Ethernet infrastructure that would be needed to avoid a reduction in the ability to detect errors. CRC calculations done in software have always resulted in slower performance than that achieved when using simple additive checksums, as found with TCP and UDP. To overcome the performance penalty, Intel now offers 1Gb NIC (82576) and 10Gb NIC (X520) that off-load SCTP checksum calculations and Core i7 processors support the CRC32c instruction as part of their new SSE4
vector math instruction set.
Support of Castagnoli CRC polynomial within a general purpose transport designed to handle data chunks, and within a TCP transport designed to carry SCSI data, both provide improved error detection rates despite the use of Jumbo Frames where increase of the Ethernet MTU would have otherwise resulted in a significant reduction in error detection.
packets which have a payload in excess of the tacitly accepted jumbo frame size of 9000 bytes. The relative scalability of network data throughput as a function of packet transfer rates is related in a complex manner to payload size per packet. Generally, as line bit rate increases, the packet payload size should increase in direct proportion to maintain equivalent timing parameters. This however implies the covariant scaling of numerous intermediating logic circuits along the network path, to accommodate the maximum transmission unit
(MTU), required. As it has been a relatively difficult, and somewhat lengthy, process to increase the path MTU of high performance national research and education networks from 1518 bytes to 9000 bytes or so, a subsequent increase, possibly to 64000 bytes for example, may take some time.
The main factor involved with an increase in the maximum segment size
(MSS) is an increase in the available memory buffer size in all of the intervening persistence mechanisms along the path. The main benefit of this is the reduction of the packet rate
, both at end nodes and intermediate transit nodes. As the nodes in general use reciprocating logic to handle the packets, the number of machine cycles
spent parsing
packet headers decreases as the average MSS per packet increases. This relationship becomes increasingly important as average network line bit rate increases to 10 gigabits per second, and above.
Ethernet frame
A data packet on an Ethernet link is called an Ethernet frame. A frame begins with Preamble and Start Frame Delimiter. Following which, each Ethernet frame continues with an Ethernet header featuring destination and source MAC addresses. The middle section of the frame is payload data including any...
s with more than 1500 bytes of payload. Conventionally, jumbo frames can carry up to 9000 bytes of payload, but variations exist and some care must be taken when using the term. Many Gigabit Ethernet
Gigabit Ethernet
Gigabit Ethernet is a term describing various technologies for transmitting Ethernet frames at a rate of a gigabit per second , as defined by the IEEE 802.3-2008 standard. It came into use beginning in 1999, gradually supplanting Fast Ethernet in wired local networks where it performed...
switches and Gigabit Ethernet network interface cards support jumbo frames. Some Fast Ethernet
Fast Ethernet
In computer networking, Fast Ethernet is a collective term for a number of Ethernet standards that carry traffic at the nominal rate of 100 Mbit/s, against the original Ethernet speed of 10 Mbit/s. Of the fast Ethernet standards 100BASE-TX is by far the most common and is supported by the...
switches and Fast Ethernet network interface cards also support jumbo frames. Most national research and education networks (such as Internet2
Internet2
Internet2 is an advanced not-for-profit US networking consortium led by members from the research and education communities, industry, and government....
/NLR, ESnet, GÉANT
GEANT
GÉANT is the main European multi-gigabit computer network for research and education purposes...
, and AARNet
AARNet
AARNet or Australian Academic and Research Network offers Internet services to the Australian education and research communities and their research partners.AARNet is a not-for-profit company limited by shares...
) support jumbo frames, but most commercial Internet service providers do not.
Inception
The original 1500-byte payload size for Ethernet frames was used because of the high error rates and low speed of communications. Thus, if one receives a corrupted packet, only 1500 bytes (plus 18 bytes for the frame header and other overhead) must be re-sent to correct the error. However, each frame requires that the network hardware and software process it. If the frame size is increased, the same amount of data can be transferred with less effort. This reduces CPU utilization (mostly due to interrupt reduction) and increases throughput by allowing the system to concentrate on the data in the frames, instead of the frames around the data. At the sender, a similar reduction in CPU utilization can be achieved by using TCP segmentation offloadingTCP segmentation offloading
In computer networking, large segment offload is a technique for increasing outbound throughput of high-bandwidth network connections by reducing CPU overhead. It works by queuing up large buffers and letting the network interface card split them into separate packets...
, although this does not reduce the receiver CPU load. Interrupt-combining Ethernet chipsets, however, do provide most of the same gain for the receiver, and work without special consideration and without requiring all stations to support jumbo frames. Zero-copy
Zero-copy
"Zero-copy" describes computer operations in which the CPU does not perform the task of copying data from one memory area to another. This is most often used to save on processing power and memory use when sending files over a network.- Principle :...
NICs and device drivers, when combined with interrupt combining, can provide effectively all the gains of jumbo frames without the re-send costs, and without requiring any changes to other stations on the network.
Jumbo frames gained initial prominence when Alteon WebSystems
Alteon WebSystems
Alteon WebSystems Incorporated , originally known as Alteon Networks, was an Internet infrastructure company based in San Jose, California.Alteon was acquired by Nortel Networks on October 4, 2000.-History:...
introduced them in their ACEnic Gigabit Ethernet
Gigabit Ethernet
Gigabit Ethernet is a term describing various technologies for transmitting Ethernet frames at a rate of a gigabit per second , as defined by the IEEE 802.3-2008 standard. It came into use beginning in 1999, gradually supplanting Fast Ethernet in wired local networks where it performed...
adapters.
Many other vendors also adopted the size; however, they did not become part of the official IEEE 802.3
IEEE 802.3
IEEE 802.3 is a working group and a collection of IEEE standards produced by the working group defining the physical layer and data link layer's media access control of wired Ethernet. This is generally a local area network technology with some wide area network applications...
Ethernet standard.
Adoption
The IEEE 802 standards committee does not recognize jumbo frames, as doing so would remove interoperability with existing Ethernet equipment and other 802 protocols, including 802.5 Token Ring and 802.11 Wireless LANIEEE 802.11
IEEE 802.11 is a set of standards for implementing wireless local area network computer communication in the 2.4, 3.6 and 5 GHz frequency bands. They are created and maintained by the IEEE LAN/MAN Standards Committee . The base version of the standard IEEE 802.11-2007 has had subsequent...
. The presence of Jumbo frames may have an adverse effect on network latency, especially on low bandwidth links.
The use of 9000 bytes as preferred size for jumbo frames arose from discussions within the Joint Engineering Team of Internet2
Internet2
Internet2 is an advanced not-for-profit US networking consortium led by members from the research and education communities, industry, and government....
and the U.S. federal government networks. Their recommendation has been adopted by all other national research and education networks. In order to meet this mandatory purchasing criterion, manufacturers have in turn adopted 9000 bytes as the conventional jumbo frame size.
Internet Protocol
Internet Protocol
The Internet Protocol is the principal communications protocol used for relaying datagrams across an internetwork using the Internet Protocol Suite...
subnetworks require that all hosts in a subnet have an identical MTU. As a result, interfaces using the standard frame size and interfaces using the jumbo frame size should not be in the same subnet. To reduce interoperability issues, network interface cards capable of jumbo frames require explicit configuration to use jumbo frames.
IETF solutions for adopting Jumbo Frames avoids the data integrity reductions through use of the Castagnoli CRC polynomial being implemented within the SCTP transport (RFC 4960), and iSCSI (RFC 3720). Selection of this polynomial was based upon work documented in the paper "32-Bit Cyclic Redundancy Codes for Internet Applications". The Castagnoli polynomial 0x11EDC6F41 achieves the Hamming Distance HD=6 beyond one Ethernet MTU (to a 16,360 bit data word length) and HD=4 to 114,663 bits, which is more than 9 times the length of an Ethernet MTU. This gives two additional bits of error detection ability at MTU-sized data words compared to the Ethernet CRC standard polynomial while not sacrificing HD=4 capability for data word sizes up to and beyond 72k bits.
By using a CRC checksum rather than simple additive checksums as contained within the UDP and TCP transports, errors generated internal to NICs can be detected as well. Both TCP and UDP have proven ineffective at detecting bus specific bit errors, since these errors with simple summations tend to be self cancelling. Testing that led to adoption of RFC 3309 compiled evidence based upon simulated error injection against real data that demonstrated as much as 2% of these errors were not being detected.
One of the major impediments toward the adoption of Jumbo Frames has been the inability to upgrade existing Ethernet infrastructure that would be needed to avoid a reduction in the ability to detect errors. CRC calculations done in software have always resulted in slower performance than that achieved when using simple additive checksums, as found with TCP and UDP. To overcome the performance penalty, Intel now offers 1Gb NIC (82576) and 10Gb NIC (X520) that off-load SCTP checksum calculations and Core i7 processors support the CRC32c instruction as part of their new SSE4
SSE4
SSE4 is a CPU instruction set used in the Intel Core microarchitecture and AMD K10 . It was announced on 27 September 2006 at the Fall 2006 Intel Developer Forum, with vague details in a white paper; more precise details of 47 instructions became available at the Spring 2007 Intel Developer Forum...
vector math instruction set.
Support of Castagnoli CRC polynomial within a general purpose transport designed to handle data chunks, and within a TCP transport designed to carry SCSI data, both provide improved error detection rates despite the use of Jumbo Frames where increase of the Ethernet MTU would have otherwise resulted in a significant reduction in error detection.
Baby giant frames
Baby giants are frames that are only slightly larger than allowed by the IEEE Ethernet standards.Super jumbo frames
Super jumbo frames (SJFs) are generally considered to be InternetInternet
The Internet is a global system of interconnected computer networks that use the standard Internet protocol suite to serve billions of users worldwide...
packets which have a payload in excess of the tacitly accepted jumbo frame size of 9000 bytes. The relative scalability of network data throughput as a function of packet transfer rates is related in a complex manner to payload size per packet. Generally, as line bit rate increases, the packet payload size should increase in direct proportion to maintain equivalent timing parameters. This however implies the covariant scaling of numerous intermediating logic circuits along the network path, to accommodate the maximum transmission unit
Maximum transmission unit
In computer networking, the maximum transmission unit of a communications protocol of a layer is the size of the largest protocol data unit that the layer can pass onwards. MTU parameters usually appear in association with a communications interface...
(MTU), required. As it has been a relatively difficult, and somewhat lengthy, process to increase the path MTU of high performance national research and education networks from 1518 bytes to 9000 bytes or so, a subsequent increase, possibly to 64000 bytes for example, may take some time.
The main factor involved with an increase in the maximum segment size
Maximum segment size
The maximum segment size is a parameter of the TCP protocol that specifies the largest amount of data, specified in octets, that a computer or communications device can receive in a single TCP segment, and therefore in a single IP datagram. It does not count the TCP header or the IP header...
(MSS) is an increase in the available memory buffer size in all of the intervening persistence mechanisms along the path. The main benefit of this is the reduction of the packet rate
Bit rate
In telecommunications and computing, bit rate is the number of bits that are conveyed or processed per unit of time....
, both at end nodes and intermediate transit nodes. As the nodes in general use reciprocating logic to handle the packets, the number of machine cycles
Instruction cycle
An instruction cycle is the basic operation cycle of a computer. It is the process by which a computer retrieves a program instruction from its memory, determines what actions the instruction requires, and carries out those actions...
spent parsing
Parsing
In computer science and linguistics, parsing, or, more formally, syntactic analysis, is the process of analyzing a text, made of a sequence of tokens , to determine its grammatical structure with respect to a given formal grammar...
packet headers decreases as the average MSS per packet increases. This relationship becomes increasingly important as average network line bit rate increases to 10 gigabits per second, and above.
External links
- Jumbo Frames - Where to use it?
- Jumbo frames? Yes!, by Selina Lo, Alteon Networks, 2/23/1998 in NetworkWorld
- SJF Supercomputing 2005 Demo
- Pushing up the Internet MTU
- IEEE 802.3as Frame Expansion Task Force
- 32-Bit Cyclic Redundancy Codes for Internet Applications
- Need To Know: Jumbo Frames in Small Networks
- How to Jumbo Frames in Archlinux