Construct (python library)
Encyclopedia
Construct is a python
library for the construction and deconstruction of data structures in a declarative
fashion. In this context, construction, or building, refers to the process of converting (serializing
) a programmatic object into a binary representation. Deconstruction, or parsing, refers to the opposite process of converting (deserializing) binary data into a programmatic object. Being declarative means user code defines the only the data structure, instead of the convention of writing procedural code to accomplish the goal. Construct can work seamlessly with bit
- and byte
-level data granularity and various byte-ordering
.
Using declarative code has many benefits. For example, the same code that can parse can also build (symmetrical), debugging and testing are much simpler (provable to some extent), creating new constructs is easy (wrapping components), and many more. If you are familiar with C (programming language)
, it would be easiest to think of constructs is casting
of
using Construct. Note that some code is omitted for brevity and complexity. Also note that the following code is just python code that creates objects.
First, the ethernet
header (layer 2):
ethernet = Struct("ethernet_header",
Bytes("destination", 6),
Bytes("source", 6),
Enum(UBInt16("type"),
IPv4 = 0x0800,
ARP = 0x0806,
RARP = 0x8035,
X25 = 0x0805,
IPX = 0x8137,
IPv6 = 0x86DD,
),
)
Next, the IP
header (layer 3):
ip = Struct("ip_header",
EmbeddedBitStruct(
Const(Nibble("version"), 4),
Nibble("header_length"),
),
BitStruct("tos",
Bits("precedence", 3),
Flag("minimize_delay"),
Flag("high_throuput"),
Flag("high_reliability"),
Flag("minimize_cost"),
Padding(1),
),
UBInt16("total_length"),
# ...
)
And finally, the TCP
header (layer 4):
tcp = Struct("tcp_header",
UBInt16("source"),
UBInt16("destination"),
UBInt32("seq"),
UBInt32("ack"),
# ...
)
Now define the hierarchy of the protocol stack. The following code "binds" each pair of adjacent protocols into a separate unit. Each such unit will "select" the proper next layer based on its contained protocol.
layer4tcp = Struct("layer4",
Embed(tcp),
# ... payload
)
layer3ip = Struct("layer3",
Embed(ip),
Switch("next", lambda ctx: ctx["protocol"],
{
"TCP" : layer4tcp,
}
),
)
layer2ethernet = Struct("layer2",
Embed(ethernet),
Switch("next", lambda ctx: ctx["type"],
{
"IP" : layer3ip,
}
),
)
At this point, the code can parse captured TCP/IP frames into "packet" objects and build these packet objects back into binary representation.
tcpip_stack = layer2ethernet
pkt = tcpip_stack.parse("...raw captured packet...")
raw_data = tcpip_stack.build(pkt)
module, which originated as a port of Construct to the Perl programming language
. (see its main POD document for its inspiration). Since the initial version, some parts of the original API have been deprecated.
Python (programming language)
Python is a general-purpose, high-level programming language whose design philosophy emphasizes code readability. Python claims to "[combine] remarkable power with very clear syntax", and its standard library is large and comprehensive...
library for the construction and deconstruction of data structures in a declarative
Declarative programming
In computer science, declarative programming is a programming paradigm that expresses the logic of a computation without describing its control flow. Many languages applying this style attempt to minimize or eliminate side effects by describing what the program should accomplish, rather than...
fashion. In this context, construction, or building, refers to the process of converting (serializing
Serialization
In computer science, in the context of data storage and transmission, serialization is the process of converting a data structure or object state into a format that can be stored and "resurrected" later in the same or another computer environment...
) a programmatic object into a binary representation. Deconstruction, or parsing, refers to the opposite process of converting (deserializing) binary data into a programmatic object. Being declarative means user code defines the only the data structure, instead of the convention of writing procedural code to accomplish the goal. Construct can work seamlessly with bit
Bit
A bit is the basic unit of information in computing and telecommunications; it is the amount of information stored by a digital device or other physical system that exists in one of two possible distinct states...
- and byte
Byte
The byte is a unit of digital information in computing and telecommunications that most commonly consists of eight bits. Historically, a byte was the number of bits used to encode a single character of text in a computer and for this reason it is the basic addressable element in many computer...
-level data granularity and various byte-ordering
Endianness
In computing, the term endian or endianness refers to the ordering of individually addressable sub-components within the representation of a larger data item as stored in external memory . Each sub-component in the representation has a unique degree of significance, like the place value of digits...
.
Using declarative code has many benefits. For example, the same code that can parse can also build (symmetrical), debugging and testing are much simpler (provable to some extent), creating new constructs is easy (wrapping components), and many more. If you are familiar with C (programming language)
C (programming language)
C is a general-purpose computer programming language developed between 1969 and 1973 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system....
, it would be easiest to think of constructs is casting
Type conversion
In computer science, type conversion, typecasting, and coercion are different ways of, implicitly or explicitly, changing an entity of one data type into another. This is done to take advantage of certain features of type hierarchies or type representations...
of
char * to struct foo * and vice versa, rather than writing code which unpacks the data.Example
The following example will show how one might define a TCP/IP protocol stackProtocol stack
The protocol stack is an implementation of a computer networking protocol suite. The terms are often used interchangeably. Strictly speaking, the suite is the definition of the protocols, and the stack is the software implementation of them....
using Construct. Note that some code is omitted for brevity and complexity. Also note that the following code is just python code that creates objects.
First, the ethernet
Ethernet
Ethernet is a family of computer networking technologies for local area networks commercially introduced in 1980. Standardized in IEEE 802.3, Ethernet has largely replaced competing wired LAN technologies....
header (layer 2):
ethernet = Struct("ethernet_header",
Bytes("destination", 6),
Bytes("source", 6),
Enum(UBInt16("type"),
IPv4 = 0x0800,
ARP = 0x0806,
RARP = 0x8035,
X25 = 0x0805,
IPX = 0x8137,
IPv6 = 0x86DD,
),
)
Next, the IP
Internet Protocol
The Internet Protocol is the principal communications protocol used for relaying datagrams across an internetwork using the Internet Protocol Suite...
header (layer 3):
ip = Struct("ip_header",
EmbeddedBitStruct(
Const(Nibble("version"), 4),
Nibble("header_length"),
),
BitStruct("tos",
Bits("precedence", 3),
Flag("minimize_delay"),
Flag("high_throuput"),
Flag("high_reliability"),
Flag("minimize_cost"),
Padding(1),
),
UBInt16("total_length"),
# ...
)
And finally, the TCP
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
header (layer 4):
tcp = Struct("tcp_header",
UBInt16("source"),
UBInt16("destination"),
UBInt32("seq"),
UBInt32("ack"),
# ...
)
Now define the hierarchy of the protocol stack. The following code "binds" each pair of adjacent protocols into a separate unit. Each such unit will "select" the proper next layer based on its contained protocol.
layer4tcp = Struct("layer4",
Embed(tcp),
# ... payload
)
layer3ip = Struct("layer3",
Embed(ip),
Switch("next", lambda ctx: ctx["protocol"],
{
"TCP" : layer4tcp,
}
),
)
layer2ethernet = Struct("layer2",
Embed(ethernet),
Switch("next", lambda ctx: ctx["type"],
{
"IP" : layer3ip,
}
),
)
At this point, the code can parse captured TCP/IP frames into "packet" objects and build these packet objects back into binary representation.
tcpip_stack = layer2ethernet
pkt = tcpip_stack.parse("...raw captured packet...")
raw_data = tcpip_stack.build(pkt)
Ports and spin-offs
Data-ParseBinary is a CPANCPAN
CPAN, the Comprehensive Perl Archive Network, is an archive of nearly 100,000 modules of software written in Perl, as well as documentation for it. It has a presence on the World Wide Web at and is mirrored worldwide at more than 200 locations...
module, which originated as a port of Construct to the Perl programming language
Perl
Perl is a high-level, general-purpose, interpreted, dynamic programming language. Perl was originally developed by Larry Wall in 1987 as a general-purpose Unix scripting language to make report processing easier. Since then, it has undergone many changes and revisions and become widely popular...
. (see its main POD document for its inspiration). Since the initial version, some parts of the original API have been deprecated.