ATA over Ethernet
Encyclopedia
ATA over Ethernet is a network protocol developed by the Brantley Coile Company, designed for simple, high-performance access of SATA
storage devices over Ethernet
networks. It is used to build storage area network
s (SANs) with low-cost, standard technologies.
. AoE does not use internet protocol
(IP), it cannot be accessed over the Internet or other IP networks. In this regard it is more comparable to Fibre Channel over Ethernet than iSCSI
.
This approach makes AoE more lightweight (with less load on the host), makes it easier to implement, provides a layer of inherent security, and offers higher performance. The AoE specification is 12 pages compared with iSCSI's 257 pages..
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | Ethernet Destination Addr |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4 | Ethernet Destination Addr | Ethernet Source Addr |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
8 | Ethernet Source Addr |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
12 | Ethernet Type (0x88A2) | Ver | Flags | Error |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 | Major | Minor | Command |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
20 | Tag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
24 | Arg |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
AoE has the IEEE assigned EtherType
0x88A2.
frames and lets them travel over an Ethernet network instead of a SATA or 40-pin ribbon cable. By using an AoE driver, the host operating system is able to access a remote disk as if it were directly attached.
The encapsulation of ATA provided by AoE is simple and low-level, allowing the translation to happen either at high performance or inside a small, embedded device, or both.
One option provided by AoE is to use the storage device itself as the mechanism for determining specific host access. This is the AoE "config string" feature . The config string can record who is using the device, as well as other information. If more than one host tries to set the config string simultaneously, only one succeeds. The other host is informed of the conflict.
s provide ATA over Ethernet (AoE) support:
brand, along with diskless gateways that add network-attached storage
functionality, using the NFS or SMB
protocols, to one or more AoE appliances.
In 2007 LayerWalker
announced the world's first single-chip AoE hardware solution called miniSAN running at both Fast and Gigabit Ethernet grades. The miniSAN product family offers standard AoE server functions plus other management features that targets PC, consumer and SMB markets.
When using a SAN network to access storage, there are several potential advantages over a local disk:
Traditional filesystems (such as FAT
or ext3
) are designed to be accessed by a single host, and will cause unpredictable behavior if accessed by multiple machines. Such filesystems may be used, and AoE provides mechanisms whereby an AoE target can be guarded against simultaneous access (see: Config String).
Shared disk file systems allow multiple machines to use a single hard disk safely by coordinating simultaneous access to individual files. These filesystems can be used to allow multiple machines access to the same AoE target without an intermediate server or filesystem (and at higher performance).
Serial ATA
Serial ATA is a computer bus interface for connecting host bus adapters to mass storage devices such as hard disk drives and optical drives...
storage devices over Ethernet
Ethernet
Ethernet is a family of computer networking technologies for local area networks commercially introduced in 1980. Standardized in IEEE 802.3, Ethernet has largely replaced competing wired LAN technologies....
networks. It is used to build storage area network
Storage area network
A storage area network is a dedicated network that provides access to consolidated, block level data storage. SANs are primarily used to make storage devices, such as disk arrays, tape libraries, and optical jukeboxes, accessible to servers so that the devices appear like locally attached devices...
s (SANs) with low-cost, standard technologies.
Protocol description
AoE runs on layer 2 EthernetEthernet
Ethernet is a family of computer networking technologies for local area networks commercially introduced in 1980. Standardized in IEEE 802.3, Ethernet has largely replaced competing wired LAN technologies....
. AoE does not use internet protocol
Internet Protocol
The Internet Protocol is the principal communications protocol used for relaying datagrams across an internetwork using the Internet Protocol Suite...
(IP), it cannot be accessed over the Internet or other IP networks. In this regard it is more comparable to Fibre Channel over Ethernet than iSCSI
ISCSI
In computing, iSCSI , is an abbreviation of Internet Small Computer System Interface, an Internet Protocol -based storage networking standard for linking data storage facilities. By carrying SCSI commands over IP networks, iSCSI is used to facilitate data transfers over intranets and to manage...
.
This approach makes AoE more lightweight (with less load on the host), makes it easier to implement, provides a layer of inherent security, and offers higher performance. The AoE specification is 12 pages compared with iSCSI's 257 pages..
- AoE Header Format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | Ethernet Destination Addr |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4 | Ethernet Destination Addr | Ethernet Source Addr |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
8 | Ethernet Source Addr |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
12 | Ethernet Type (0x88A2) | Ver | Flags | Error |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 | Major | Minor | Command |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
20 | Tag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
24 | Arg |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
AoE has the IEEE assigned EtherType
EtherType
EtherType is a two-octet field in an Ethernet frame. It is used to indicate which protocol is encapsulated in the PayLoad of an Ethernet Frame. This field was first defined by the Ethernet II framing networking standard, and later adapted for the IEEE 802.3 Ethernet networking standard.EtherType...
0x88A2.
ATA encapsulation
SATA (and older PATA) hard drives use the Advanced Technology Attachment (ATA) protocol to issue commands, such as read, write, and status. AoE encapsulates those commands inside EthernetEthernet
Ethernet is a family of computer networking technologies for local area networks commercially introduced in 1980. Standardized in IEEE 802.3, Ethernet has largely replaced competing wired LAN technologies....
frames and lets them travel over an Ethernet network instead of a SATA or 40-pin ribbon cable. By using an AoE driver, the host operating system is able to access a remote disk as if it were directly attached.
The encapsulation of ATA provided by AoE is simple and low-level, allowing the translation to happen either at high performance or inside a small, embedded device, or both.
Routability
AoE runs directly on top of Ethernet, instead of an intermediate protocol such as TCP/IP. This reduces the significant CPU overhead of TCP/IP. However, this means that routers cannot be used to route a packet across disparate networks (such as the Internet). Instead, AoE packets can travel within a single local Ethernet storage area network (eg, a set of computers connected to the same switch or in the same VLAN).Security
The non-routability of AoE is the only security mechanism (ie, an intruder can't connect through a router—they must physically plug into the local Ethernet switch where Ethernet frame tunneling over routed networks is not in use). However, there are no AoE-specific mechanisms for password verification or encryption. The protocol provides for AoE targets such as Coraid Storage appliances, vblade and GGAOED to establish access lists ("masks") allowing connections only from specific MAC addresses (although these can be spoofed).Config string
The AoE protocol provides a mechanism for host-based cooperative locking. When more than one AoE initiator is using an AoE target they must communicate to avoid interfering with one another as they read and write the config string data on the shared AoE device. Without this cooperation file-system corruption and data loss is likely, unless access is strictly read-only or a cluster file system is used).One option provided by AoE is to use the storage device itself as the mechanism for determining specific host access. This is the AoE "config string" feature . The config string can record who is using the device, as well as other information. If more than one host tries to set the config string simultaneously, only one succeeds. The other host is informed of the conflict.
Operating system support
The following operating systemOperating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
s provide ATA over Ethernet (AoE) support:
| OS | Support | Third-party drivers |
|---|---|---|
| Linux Linux kernel The Linux kernel is an operating system kernel used by the Linux family of Unix-like operating systems. It is one of the most prominent examples of free and open source software.... |
Native (2.6.11+) | Coraid |
| Windows | Third-party | StarWind Software AoE Initiator , WinAoE , and WinVBlock |
| Mac OS X Mac OS X Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, has been included with all new Macintosh computer systems... 10.4 and up |
Third-party | 2DegreesFrost |
| Mac OS X Mac OS X Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, has been included with all new Macintosh computer systems... 10.5 and 10.6 |
Third-party | Small Tree Communications |
| Solaris | Third-party | Coraid |
| FreeBSD FreeBSD FreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant... |
Third-party | Coraid (outdated) |
| OpenBSD OpenBSD OpenBSD is a Unix-like computer operating system descended from Berkeley Software Distribution , a Unix derivative developed at the University of California, Berkeley. It was forked from NetBSD by project leader Theo de Raadt in late 1995... |
Native (4.5-current) | |
| VMware VMware VMware, Inc. is a company providing virtualization software founded in 1998 and based in Palo Alto, California, USA. The company was acquired by EMC Corporation in 2004, and operates as a separate software subsidiary .... |
Third-party | Coraid |
| Plan 9 from Bell Labs Plan 9 from Bell Labs Plan 9 from Bell Labs is a distributed operating system. It was developed primarily for research purposes as the successor to Unix by the Computing Sciences Research Center at Bell Labs between the mid-1980s and 2002... |
Native |
Linux target support
Linux can function as an AoE target using one of these independently-developed implementations:- vbladehttp://sourceforge.net/projects/aoetools/vblade, a userspace daemon that is part of the aoetools package.
- kvbladehttp://sourceforge.net/projects/aoetools/kvblade, a Linux kernel module.
- ggaoedhttp://code.google.com/p/ggaoed/, a userspace daemon that takes advantage of Linux-specific performance features.
- qaoedhttp://code.google.com/p/qaoed/, a multithreaded userspace daemon.
Hardware support
The Coraid company offers an array of AoE SAN appliances under the EtherDriveEtherdrive
EtherDrive is a brand name for a variety of storage area network devices based upon the ATA over Ethernet protocol. EtherDrive is a registered trademark of Coraid, Inc.. EtherDrive is an invented word, owned and used as a trademark by Coraid since 2002 and registered with the United States Patent...
brand, along with diskless gateways that add network-attached storage
Network-attached storage
Network-attached storage is file-level computer data storage connected to a computer network providing data access to heterogeneous clients. NAS not only operates as a file server, but is specialized for this task either by its hardware, software, or configuration of those elements...
functionality, using the NFS or SMB
Server Message Block
In computer networking, Server Message Block , also known as Common Internet File System operates as an application-layer network protocol mainly used to provide shared access to files, printers, serial ports, and miscellaneous communications between nodes on a network. It also provides an...
protocols, to one or more AoE appliances.
In 2007 LayerWalker
LayerWalker
LayerWalker Technology, Inc. is a fabless design company that specializes in network storage SoC solutions. Their products target digital home, small business and consumer electronics markets...
announced the world's first single-chip AoE hardware solution called miniSAN running at both Fast and Gigabit Ethernet grades. The miniSAN product family offers standard AoE server functions plus other management features that targets PC, consumer and SMB markets.
Related concepts
Although AoE is a simple network protocol, it opens up a complex realm of storage possibilities. To understand and evaluate these storage scenarios, it helps to be familiar with a few concepts.Storage area networks
A SAN allows the physical hard drive to be removed from the server that uses it, and placed on the network. A SAN interface is similar in principle to non-networked interfaces such as SATA or SCSI. Most users will not use a SAN interface directly. Instead, they will connect to a server that uses a SAN disk instead of a local disk. Direct connection, however, can also be used.When using a SAN network to access storage, there are several potential advantages over a local disk:
- It is easier to add storage capacity and the amount of storage is practically unlimited.
- It is easier to reallocate storage capacity.
- Data may be shared.
- Additionally, compared to other forms of networked storage, SANs are low-level and high performance
Utilizing storage area networks
To utilize a SAN disk, the host must format it with a filesystem. However, unlike a SATA or SCSI disk a SAN hard drive may be accessed by multiple machines. This is a source of both danger and opportunity.Traditional filesystems (such as FAT
File Allocation Table
File Allocation Table is a computer file system architecture now widely used on many computer systems and most memory cards, such as those used with digital cameras. FAT file systems are commonly found on floppy disks, flash memory cards, digital cameras, and many other portable devices because of...
or ext3
Ext3
The ext3 or third extended filesystem is a journaled file system that is commonly used by the Linux kernel. It is the default file system for many popular Linux distributions, including Debian...
) are designed to be accessed by a single host, and will cause unpredictable behavior if accessed by multiple machines. Such filesystems may be used, and AoE provides mechanisms whereby an AoE target can be guarded against simultaneous access (see: Config String).
Shared disk file systems allow multiple machines to use a single hard disk safely by coordinating simultaneous access to individual files. These filesystems can be used to allow multiple machines access to the same AoE target without an intermediate server or filesystem (and at higher performance).
External links
- Articles:
- ATA Over Ethernet: Putting Hard Drives on the LAN — Linux JournalLinux JournalLinux Journal is a monthly technology magazine published by Belltown Media, Inc. of Houston, Texas. The magazine focuses specifically on Linux, allowing the content to be a highly specialized source of information for open source enthusiasts.-History:...
(28 April 2005) - ATA-over-Ethernet enables low-cost Linux-oriented SAN — LinuxDevices.com (23 June 2004)
- The ATA over Ethernet (AoE) Protocol — Linux MagazineLinux MagazineLinux Magazine is a European professional journal. It addresses itself to readers who work professionally with operating systems based around the Linux kernel. Linux Magazine is published by Linux New Media AG and was born after the great success of Linux-Magazin...
(June 15, 2005) - The ATA over Ethernet Protocol Official Document from Coraid Coraid.com (01 June 2009)
- ATA Over Ethernet: Putting Hard Drives on the LAN — Linux Journal
- Protocol:
- AoE protocol specification (PDFPortable Document FormatPortable Document Format is an open standard for document exchange. This file format, created by Adobe Systems in 1993, is used for representing documents in a manner independent of application software, hardware, and operating systems....
file) - AoE protocol specification
- AoE protocol specification (PDF
- HowTos: