EEMBC
Encyclopedia
EEMBC, the Embedded Microprocessor Benchmark Consortium, is a non-profit organization
formed in 1997 with the aim of developing meaningful performance benchmark
s for the hardware and software used in embedded systems. The goal of its members is to make EEMBC benchmarks an industry standard for evaluating the capabilities of embedded microprocessors, compilers, and the associated embedded system implementations, according to objective, clearly defined, application-based criteria.
EEMBC benchmarks, which are available to its members, aim to reflect real-world applications and the demands that embedded systems encounter in these environments. The consortium also licenses its benchmark suites to non-members, although with different usage terms.
The president of EEMBC is Markus Levy, who is also president of the Multicore Association
. The director of software engineering is Shay Gal-On.
and GrinderBench), but they must submit these scores and their entire benchmark platform to the EEMBC Technology Center (ETC) for official (and free) certification before making the scores public. During the certification process, the ETC rebuilds the benchmark code and verifies accuracy and repeatability.
compatible source code. These benchmark suites included AutoBench 1.1 (for automotive, industrial, and general-purpose applications), ConsumerBench 1.1 (for digital imaging
tasks), Networking 1.1
, OABench 1.1 (targeting printer-related applications), and TeleBench 1.1 (for Digital signal processors).
In 2004, the consortium released GrinderBench, a benchmark based on the Java language. GrinderBench represented EEMBC's foray into embedded system benchmarking, allowing the user to approximate the performance of Java 2 Micro Edition (J2ME™) applications in products such as mobile phones and Blu-ray Disc
players.
In 2005, in order to meet the demands of increasing processor performance and cache memory sizes, EEMBC released DENBench and Networking 2.0, supersets of ConsumerBench 1.1 and Networking 1.1, respectively. Both of these suites have significantly enhanced datasets to heavily tax the processor's memory subsystems.
' LabVIEW
graphical development environment and data acquisition hardware.
benchmark. CoreMark’s primary goals are simplicity and providing a method for testing only a processor’s core features. Each iteration of CoreMark performs the following algorithms: list processing (find and sort), matrix manipulation (common matrix operations), state machine (determine if an input stream contains valid numbers), and CRC
. Running CoreMark produces a single-number score, allowing users to make quick comparisons between processors.
(SoCs), EEMBC began to evolve its benchmark suites to target more advanced embedded systems such as networking systems and smartphones. ETCPBench measures TCP/IP performance and conformance and validates the Ethernet functionality of embedded systems. The standardized methodology of this benchmark suite uses tools and scripts from Ixia. Regarding conformance, ETCPBench measure the ability of a system to adhere to applicable Request for Comments
(RFC) standards; regarding performance, ETCPBench utilizes various networking scenarios to measure a system’s bandwidth and latency capabilities. This benchmark suite is applicable to a wide range of network systems from a low-end home Gateway (telecommunications)
to high-end network switches and routers, as well as a wide variety of semiconductors ranging from Ethernet-enabled microcontroller
s to high-end processors for data center equipment.
EEMBC is currently working on, DPIBench, a separate benchmark suite that targets network firewall appliances performing Deep Packet Inspection
. DPIBench will be used to evaluate throughput and latency to highlight the strengths and weaknesses of DPI systems, processors, and middleware
. This testing approach considers the various threat vectors used in attempting to transfer an infected payload (software)
into a network. Without a common standard by which to compare performance across all these variables, consumers of DPI technologies lack an objective means of selecting a solution from the myriad vendor offerings available today, and are often at a severe disadvantage when attempting to select the most suitable solution to protect their information systems. As a matter of fact, throughput numbers from various DPI system vendors have been known to be off by as much as 90% from those numbers claimed on datasheets and marketing collateral.
Non-profit organization
Nonprofit organization is neither a legal nor technical definition but generally refers to an organization that uses surplus revenues to achieve its goals, rather than distributing them as profit or dividends...
formed in 1997 with the aim of developing meaningful performance benchmark
Benchmark (computing)
In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it...
s for the hardware and software used in embedded systems. The goal of its members is to make EEMBC benchmarks an industry standard for evaluating the capabilities of embedded microprocessors, compilers, and the associated embedded system implementations, according to objective, clearly defined, application-based criteria.
EEMBC benchmarks, which are available to its members, aim to reflect real-world applications and the demands that embedded systems encounter in these environments. The consortium also licenses its benchmark suites to non-members, although with different usage terms.
The president of EEMBC is Markus Levy, who is also president of the Multicore Association
Multicore Association
The Multicore Association, founded in 2005, is a member-funded, non-profit, industry consortium focused on the creation of open standard APIs, specifications, and guidelines that will allow system developers and programmers to adopt multicore technology into their applications more readily .The...
. The director of software engineering is Shay Gal-On.
Score Certification Program
Only EEMBC members are entitled to publish their benchmark test results (except for CoreMarkCoremark
CoreMark is a benchmark that aims to measure the performance of central processing units used in embedded systems. It was developed in 2009 by Shay Gal-On at EEMBC and is intended to become an industry standard, replacing the antiquated Dhrystone benchmark...
and GrinderBench), but they must submit these scores and their entire benchmark platform to the EEMBC Technology Center (ETC) for official (and free) certification before making the scores public. During the certification process, the ETC rebuilds the benchmark code and verifies accuracy and repeatability.
Benchmark Chronology
Up until 2004, the EEMBC benchmarks targeted embedded processors and were exclusively built using C standard libraryC standard library
The C Standard Library is the standard library for the programming language C, as specified in the ANSI C standard.. It was developed at the same time as the C POSIX library, which is basically a superset of it...
compatible source code. These benchmark suites included AutoBench 1.1 (for automotive, industrial, and general-purpose applications), ConsumerBench 1.1 (for digital imaging
Digital imaging
Digital imaging or digital image acquisition is the creation of digital images, typically from a physical scene. The term is often assumed to imply or include the processing, compression, storage, printing, and display of such images...
tasks), Networking 1.1
Networking hardware
Networking hardware typically refers to equipment facilitating the use of a computer network. Typically, this includes routers, switches, hubs, gateways, access points, network interface cards, Networking cables, network bridges, modems, ISDN adapters, firewalls and other equipments.In the most...
, OABench 1.1 (targeting printer-related applications), and TeleBench 1.1 (for Digital signal processors).
In 2004, the consortium released GrinderBench, a benchmark based on the Java language. GrinderBench represented EEMBC's foray into embedded system benchmarking, allowing the user to approximate the performance of Java 2 Micro Edition (J2ME™) applications in products such as mobile phones and Blu-ray Disc
Blu-ray Disc
Blu-ray Disc is an optical disc storage medium designed to supersede the DVD format. The plastic disc is 120 mm in diameter and 1.2 mm thick, the same size as DVDs and CDs. Blu-ray Discs contain 25 GB per layer, with dual layer discs being the norm for feature-length video discs...
players.
In 2005, in order to meet the demands of increasing processor performance and cache memory sizes, EEMBC released DENBench and Networking 2.0, supersets of ConsumerBench 1.1 and Networking 1.1, respectively. Both of these suites have significantly enhanced datasets to heavily tax the processor's memory subsystems.
Energy and Power Become Leading Indicators
While energy consumption has always been an important factor in any system design, the embedded industry paid much more attention to it in the 2006 time-frame. About that time, EEMBC released its EnergyBench test to provide data on the amount of energy a processor consumes while running EEMBC's performance benchmarks. Every processor vendor typically has its own power measurement methods, making it nearly impossible to make accurate comparisons among competing vendors. Many processor vendors offer "typical" power specifications on product datasheets that are difficult to compare with one another. The problem of interpreting these values is exacerbated when designers attempt to compare processor cores for system-on-chip implementations. EnergyBench defines a standardized methodology, currently using National InstrumentsNational Instruments
National Instruments Corporation, or NI , is an American company with over 5,000 employees and direct operations in 41 countries. Headquartered in Austin, Texas, it is a producer of automated test equipment and virtual instrumentation software...
' LabVIEW
LabVIEW
LabVIEW is a system design platform and development environment for a visual programming language from National Instruments. LabVIEW provides engineers and scientists with the tools needed to create and deploy measurement and control systems.The graphical language is named "G"...
graphical development environment and data acquisition hardware.
The Infusion of Multicore Technology
Around the 2006 time-frame, processor vendors began to introduce their multicore devices. System developers had expectations of linearly increasing performance relative to the number of cores. While there are many software programming issues to consider in order to achieve optimal parallelism, there are also performance limitations related to the multicore devices themselves (such as memory bus speed and shared resources). In 2008, EEMBC released MultiBench to analyze architectures, memory bottlenecks, OS scheduling support, synchronization efficiency, and other related system functions. It measures the impact of parallelization and scalability across both data processing and computationally intensive tasks.CoreMark
Coremark is a non-free benchmark that targets the CPU core. It was developed by Shay Gal-On and released as an industry standard by EEMBC in 2009, and will ultimately replace the DhrystoneDhrystone
Dhrystone is a synthetic computing benchmark program developed in 1984 by Reinhold P. Weicker intended to be representative of system programming. The Dhrystone grew to become representative of general processor performance...
benchmark. CoreMark’s primary goals are simplicity and providing a method for testing only a processor’s core features. Each iteration of CoreMark performs the following algorithms: list processing (find and sort), matrix manipulation (common matrix operations), state machine (determine if an input stream contains valid numbers), and CRC
Cyclic redundancy check
A cyclic redundancy check is an error-detecting code commonly used in digital networks and storage devices to detect accidental changes to raw data...
. Running CoreMark produces a single-number score, allowing users to make quick comparisons between processors.
TCP/IP and Firewall Appliance Benchmarking
As processors transformed into System-on-a-chipSystem-on-a-chip
A system on a chip or system on chip is an integrated circuit that integrates all components of a computer or other electronic system into a single chip. It may contain digital, analog, mixed-signal, and often radio-frequency functions—all on a single chip substrate...
(SoCs), EEMBC began to evolve its benchmark suites to target more advanced embedded systems such as networking systems and smartphones. ETCPBench measures TCP/IP performance and conformance and validates the Ethernet functionality of embedded systems. The standardized methodology of this benchmark suite uses tools and scripts from Ixia. Regarding conformance, ETCPBench measure the ability of a system to adhere to applicable Request for Comments
Request for Comments
In computer network engineering, a Request for Comments is a memorandum published by the Internet Engineering Task Force describing methods, behaviors, research, or innovations applicable to the working of the Internet and Internet-connected systems.Through the Internet Society, engineers and...
(RFC) standards; regarding performance, ETCPBench utilizes various networking scenarios to measure a system’s bandwidth and latency capabilities. This benchmark suite is applicable to a wide range of network systems from a low-end home Gateway (telecommunications)
Gateway (telecommunications)
In telecommunications, the term gateway has the following meaning:*In a communications network, a network node equipped for interfacing with another network that uses different protocols....
to high-end network switches and routers, as well as a wide variety of semiconductors ranging from Ethernet-enabled microcontroller
Microcontroller
A microcontroller is a small computer on a single integrated circuit containing a processor core, memory, and programmable input/output peripherals. Program memory in the form of NOR flash or OTP ROM is also often included on chip, as well as a typically small amount of RAM...
s to high-end processors for data center equipment.
EEMBC is currently working on, DPIBench, a separate benchmark suite that targets network firewall appliances performing Deep Packet Inspection
Deep packet inspection
Deep Packet Inspection is a form of computer network packet filtering that examines the data part of a packet as it passes an inspection point, searching for protocol non-compliance, viruses, spam, intrusions or predefined criteria to decide if the packet can...
. DPIBench will be used to evaluate throughput and latency to highlight the strengths and weaknesses of DPI systems, processors, and middleware
Middleware
Middleware is computer software that connects software components or people and their applications. The software consists of a set of services that allows multiple processes running on one or more machines to interact...
. This testing approach considers the various threat vectors used in attempting to transfer an infected payload (software)
Payload (software)
Payload in computing is the cargo of a data transmission. It is the part of the transmitted data which is the fundamental purpose of the transmission, to the exclusion of information sent with it solely to facilitate delivery.In computer security, payload refers to the...
into a network. Without a common standard by which to compare performance across all these variables, consumers of DPI technologies lack an objective means of selecting a solution from the myriad vendor offerings available today, and are often at a severe disadvantage when attempting to select the most suitable solution to protect their information systems. As a matter of fact, throughput numbers from various DPI system vendors have been known to be off by as much as 90% from those numbers claimed on datasheets and marketing collateral.
Measuring the Web Browsing Experience
- BrowsingBench, available in July 2011, is used to evaluate the browsing experience on smartphones and other systems with Internet browsing capabilities. BrowsingBench measures the complete user-experience from the click/touch on a URL to final page rendered on the screen, and is not limited to measuring only JavaScriptJavaScriptJavaScript is a prototype-based scripting language that is dynamic, weakly typed and has first-class functions. It is a multi-paradigm language, supporting object-oriented, imperative, and functional programming styles....
execution. In addition, it measures page rendering speed and factors in Internet content diversity as well as various network profiles used to access the Internet. One other benchmark feature that is particularly important in the mobile platform is the ability to measure power consumption. Since a browser can trade higher performance for better battery life, BrowsingBench will include a methodology for determining the browsing effects on battery life.