Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
. It describes the degree to which the source code
Source code
In computer science, source code is text written using the format and syntax of the programming language that it is being written in. Such a language is specially designed to facilitate the work of computer programmers, who specify the actions to be performed by a computer mostly by writing source...
A computer program is a sequence of instructions written to perform a specified task with a computer. A computer requires programs to function, typically executing the program's instructions in a central processor. The program has an executable form that the computer can use directly to execute...
has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing
White box testing
White-box testing is a method of testing software that tests internal structures or workings of an application, as opposed to its functionality...
.
Code coverage was among the first methods invented for systematic software testing. The first published reference was by Miller and Maloney in Communications of the ACM
Communications of the ACM
Communications of the ACM is the flagship monthly journal of the Association for Computing Machinery . First published in 1957, CACM is sent to all ACM members, currently numbering about 80,000. The articles are intended for readers with backgrounds in all areas of computer science and information...
in 1963.
Code coverage is one consideration in the safety certification of avionics equipment. The standard by which avionics gear is certified by the Federal Aviation Administration
Federal Aviation Administration
The Federal Aviation Administration is the national aviation authority of the United States. An agency of the United States Department of Transportation, it has authority to regulate and oversee all aspects of civil aviation in the U.S...
DO-178B, Software Considerations in Airborne Systems and Equipment Certification is a document dealing with the safety of software used in airborne systems....
.
Coverage criteria
To measure how well the program is exercised by a test suite
Test suite
In software development, a test suite, less commonly known as a validation suite, is a collection of test cases that are intended to be used to test a software program to show that it has some specified set of behaviours. A test suite often contains detailed instructions or goals for each...
, one or more coverage criteria are used.
Basic coverage criteria
There are a number of coverage criteria, the main ones being:
Function coverage - Has each function (or subroutine
Subroutine
In computer science, a subroutine is a portion of code within a larger program that performs a specific task and is relatively independent of the remaining code....
In mathematics and computer science, graph theory is the study of graphs, mathematical structures used to model pairwise relations between objects from a certain collection. A "graph" in this context refers to a collection of vertices or 'nodes' and a collection of edges that connect pairs of...
in the program been executed?
Decision coverage (not the same as branch coverage.) - Has every edge
Graph theory
In mathematics and computer science, graph theory is the study of graphs, mathematical structures used to model pairwise relations between objects from a certain collection. A "graph" in this context refers to a collection of vertices or 'nodes' and a collection of edges that connect pairs of...
in the program been executed? For instance, have the requirements of each branch of each control structure (such as in IF and CASE statements) been met as well as not met?
Condition coverage (or predicate coverage) - Has each boolean sub-expression evaluated both to true and false? This does not necessarily imply decision coverage.
Condition/decision coverage - Both decision and condition coverage should be satisfied.
For example, consider the following C++ function:
int foo(int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Assume this function is a part of some bigger program and this program was run with some test suite.
If during this execution function 'foo' was called at least once, then function coverage for this function is satisfied.
Statement coverage for this function will be satisfied if it was called e.g. as foo(1,1), as in this case, every line in the function is executed including z = x;.
Tests calling foo(1,1) and foo(0,1) will satisfy decision coverage, as in the first case the if condition and the short circuit condition are satisfied and z = x; is executed, and in the second neither conditional is satisfied and x is not assigned to z.
Condition coverage can be satisfied with tests that call foo(1,1), foo(1,0) and foo(0,0). These are necessary as in the first two cases (x>0) evaluates to true while in the third it evaluates false. At the same time, the first case makes (y>0)true while the second and third make it false.
Pascal is an influential imperative and procedural programming language, designed in 1968/9 and published in 1970 by Niklaus Wirth as a small and efficient language intended to encourage good programming practices using structured programming and data structuring.A derivative known as Object Pascal...
, where standard boolean operations are not short circuited, condition coverage does not necessarily imply decision coverage. For example, consider the following fragment of code:
if a and b then
Condition coverage can be satisfied by two tests:
a=true, b=false
a=false, b=true
However, this set of tests does not satisfy decision coverage as in neither case will the if condition be met.
In software testing, fault injection is a technique for improving the coverage of a test by introducing faults to test code paths, in particular error handling code paths, that might otherwise rarely be followed. It is often used with stress testing and is widely considered to be an important part...
may be necessary to ensure that all conditions and branches of exception handling
Exception handling
Exception handling is a programming language construct or computer hardware mechanism designed to handle the occurrence of exceptions, special conditions that change the normal flow of program execution....
Modified condition/decision coverage , is used in the standard DO-178B to ensure that Level A software is tested adequately.To satisfy the MC/DC coverage criterion, during testing all of the below must be true at least once:...
is satisifed. This criteria extends condition/decision criteria with requirements that each condition should affect the decision outcome independently. For example, consider the following code:
if (a or b) and c then
The condition/decision criteria will be satisfied by the following set of tests:
a=true, b=true, c=true
a=false, b=false, c=false
However, the above tests set will not satisfy modified condition/decision coverage, since in the first test, the value of 'b' and in the second test the value of 'c' would not influence the output. So, the following test set is needed to satisfy MC/DC:
a=false, b=false, c=true
a=true, b=false, c=true
a=false, b=true, c=true
a=true, b=true, c=false
The bold values influence the output, each variable must be present as an influencing value at least once with false and once with true.
Multiple condition coverage
This criteria requires that all combinations of conditions inside each decision are tested. For example, the code fragment from the previous section will require eight tests:
a=false, b=false, c=false
a=false, b=false, c=true
a=false, b=true, c=false
a=false, b=true, c=true
a=true, b=false, c=false
a=true, b=false, c=true
a=true, b=true, c=false
a=true, b=true, c=true
Other coverage criteria
There are further coverage criteria, which are used less often:
Linear code sequence and jump is a software analysis method used to identify structural units in code under test. Its primary use is with dynamic software analysis to help answer the question "How much testing is enough?"...
(LCSAJ) coverage - has every LCSAJ been executed?
JJ-Path coverage - have all jump to jump paths (aka LCSAJs) been executed?
Path coverage - Has every possible route through a given part of the code been executed?
Entry/exit coverage - Has every possible call and return of the function been executed?
Loop coverage - Has every possible loop been executed zero times, once, and more than once?
Safety-critical applications are often required to demonstrate that testing achieves 100% of some form of code coverage.
Some of the coverage criteria above are connected. For instance, path coverage implies decision, statement and entry/exit coverage. Decision coverage implies statement coverage, because every statement is part of a branch.
Full path coverage, of the type described above, is usually impractical or impossible. Any module with a succession of decisions in it can have up to paths within it; loop constructs can result in an infinite number of paths. Many paths may also be infeasible, in that there is no input to the program under test that can cause that particular path to be executed. However, a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem
Halting problem
In computability theory, the halting problem can be stated as follows: Given a description of a computer program, decide whether the program finishes running or continues to run forever...
). Methods for practical path coverage testing instead attempt to identify classes of code paths that differ only in the number of loop executions, and to achieve "basis path" coverage the tester must cover all the path classes.
In practice
The target software is built with special options or libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) is mapped back to the function points in the source code. This process allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested. The resulting output is then analyzed to see what areas of code have not been exercised and the tests are updated to include these areas as necessary. Combined with other code coverage methods, the aim is to develop a rigorous, yet manageable, set of regression tests.
In implementing code coverage policies within a software development environment one must consider the following:
What are coverage requirements for the end product certification and if so what level of code coverage is required? The typical level of rigor progression is as follows: Statement, Branch/Decision, Modified Condition/Decision Coverage
Modified Condition/Decision Coverage
Modified condition/decision coverage , is used in the standard DO-178B to ensure that Level A software is tested adequately.To satisfy the MC/DC coverage criterion, during testing all of the below must be true at least once:...
Linear code sequence and jump is a software analysis method used to identify structural units in code under test. Its primary use is with dynamic software analysis to help answer the question "How much testing is enough?"...
)
Will code coverage be measured against tests that verify requirements levied on the system under test (DO-178B
DO-178B
DO-178B, Software Considerations in Airborne Systems and Equipment Certification is a document dealing with the safety of software used in airborne systems....
)?
Is the object code generated directly traceable to source code statements? Certain certifications, (i.e. DO-178B Level A) require coverage at the assembly level if this is not the case: "Then, additional verification should be performed on the object code to establish the correctness of such generated code sequences" (DO-178B
DO-178B
DO-178B, Software Considerations in Airborne Systems and Equipment Certification is a document dealing with the safety of software used in airborne systems....
) para-6.4.4.2.
Test engineers can look at code coverage test results to help them devise test cases and input or configuration sets that will increase the code coverage over vital functions. Two common forms of code coverage used by testers are statement (or line) coverage and path (or edge) coverage. Line coverage reports on the execution footprint of testing in terms of which lines of code were executed to complete the test. Edge coverage reports which branches or code decision points were executed to complete the test. They both report a coverage metric, measured as a percentage. The meaning of this depends on what form(s) of code coverage have been used, as 67% path coverage is more comprehensive than 67% statement coverage.
Generally, code coverage tools and libraries exact a performance and/or memory or other resource cost which is unacceptable to normal operations of the software. Thus, they are only used in the lab. As one might expect, there are classes of software that cannot be feasibly subjected to these coverage tests, though a degree of coverage mapping can be approximated through analysis rather than direct testing.
There are also some sorts of defects which are affected by such tools. In particular, some race condition
Race condition
A race condition or race hazard is a flaw in an electronic system or process whereby the output or result of the process is unexpectedly and critically dependent on the sequence or timing of other events...
In computer science, real-time computing , or reactive computing, is the study of hardware and software systems that are subject to a "real-time constraint"— e.g. operational deadlines from event to system response. Real-time programs must guarantee response within strict time constraints...
sensitive operations can be masked when run under code coverage environments; and conversely, some of these defects may become easier to find as a result of the additional overhead of the testing code.
C is a general-purpose computer programming language developed between 1969 and 1973 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system....
C++ is a statically typed, free-form, multi-paradigm, compiled, general-purpose programming language. It is regarded as an intermediate-level language, as it comprises a combination of both high-level and low-level language features. It was developed by Bjarne Stroustrup starting in 1979 at Bell...
DevPartner is a set of software development and testing tools developed by Nu-Mega Technologies, acquired by Compuware Corporation in 1997, which on June 1, 2009 sold it to Micro Focus International...
Gcov with graphical summaries LCOV and text/XML summaries gcovr
Insure++ is a memory debugger computer program, used by software developers to detect various errors in programs written in C and C++. It is made by Parasoft, and is functionally similar to other memory debuggers, such as Purify and Valgrind.-Overview:...
DevPartner is a set of software development and testing tools developed by Nu-Mega Technologies, acquired by Compuware Corporation in 1997, which on June 1, 2009 sold it to Micro Focus International...
Tessy is a tool, that automates module/unit testing of embedded software written in various embedded dialects of the C/C++ programming language. Tessy can be used to get the certification for product according to standards like IEC 61508, EN 50128/50129, DO-178B, Automotive SPiCE, or the General...
Trucov is an open source code coverage analysis tool for the GCC compiler that improves upon gcov. The tool performs decision coverage analysis over any C or C++ code compiled by the GCC compiler...
DevPartner is a set of software development and testing tools developed by Nu-Mega Technologies, acquired by Compuware Corporation in 1997, which on June 1, 2009 sold it to Micro Focus International...
Kalistick is a French based company editing a software radiography platform for Java and C# developments. Its goal is to help test teams improve their efficiency in software testing.-Overview:...
NCover is a .NET code coverage tool. There are two non-related NCover products that do .NET code coverage. There is an open source NCover that can be found on Sourceforge and there is a company called NCover, LLC...
Java is a programming language originally developed by James Gosling at Sun Microsystems and released in 1995 as a core component of Sun Microsystems' Java platform. The language derives much of its syntax from C and C++ but has a simpler object model and fewer low-level facilities...
DevPartner is a set of software development and testing tools developed by Nu-Mega Technologies, acquired by Compuware Corporation in 1997, which on June 1, 2009 sold it to Micro Focus International...
EMMA is an open source toolkit for measuring and reporting Java code coverage. EMMA is distributed under the terms of Common Public License v1.0.EMMA is not currently under active development; the last stable release took place in mid-2005....
Jtest is an automated Java testing and static code analysis product that is made by Parasoft. It aims to improve Java code reliability, functionality, security, performance, and maintainability. Basic functionality includes Unit test-case generation, static analysis, regression testing, runtime...
Kalistick is a French based company editing a software radiography platform for Java and C# developments. Its goal is to help test teams improve their efficiency in software testing.-Overview:...
JavaScript is a prototype-based scripting language that is dynamic, weakly typed and has first-class functions. It is a multi-paradigm language, supporting object-oriented, imperative, and functional programming styles....
Perl is a high-level, general-purpose, interpreted, dynamic programming language. Perl was originally developed by Larry Wall in 1987 as a general-purpose Unix scripting language to make report processing easier. Since then, it has undergone many changes and revisions and become widely popular...
Devel::Cover is a complete suite for generating code coverage reports in HTML and other formats.
Python is a general-purpose, high-level programming language whose design philosophy emphasizes code readability. Python claims to "[combine] remarkable power with very clear syntax", and its standard library is large and comprehensive...
- Overview :Aldec provides software and hardware used in creation and verification of digital designs targeting FPGA and ASIC technologies. Headquartered in Henderson, Nevada, Aldec also has offices/development centers in Europe, Japan, Israel, India, China, Taiwan, Poland and Ukraine.As a member...
Cadence Design Systems, Inc is an electronic design automation software and engineering services company, founded in 1988 by the merger of SDA Systems and ECAD, Inc...
JEDA Technologies, Inc. is a provider of software products in the front-end of electronic design automation industry, specifically in ESL verification automation. JEDA products are applied to design models written in C, C++, SystemC for quality measurement, checking and improvement...
Mentor Graphics, Inc is a US-based multinational corporation dealing in electronic design automation for electrical engineering and electronics, as of 2004, ranked third in the EDA industry it helped create...
Nusym Technology, Inc. is a company that produces intelligent verification software products, also known as intelligent testbench products, which are a form of functional verification that targets and maximizes test coverage of a logic design by automatically adapting the verification testbenches...
Synopsys, Inc. is one of the largest companies in the Electronic Design Automation industry. Synopsys' first and best-known product is Design Compiler, a logic-synthesis tool. Synopsys offers a wide range of other products used in the design of an application-specific integrated circuit...
Cyclomatic complexity is a software metric . It was developed by Thomas J. McCabe, Sr. in 1976 and is used to indicate the complexity of a program. It directly measures the number of linearly independent paths through a program's source code...
Intelligent Verification, also referred to as intelligent testbench automation, is a form of functional verification used to verify that an electronic hardware design conforms to specification before device fabrication...
Linear code sequence and jump is a software analysis method used to identify structural units in code under test. Its primary use is with dynamic software analysis to help answer the question "How much testing is enough?"...
Modified condition/decision coverage , is used in the standard DO-178B to ensure that Level A software is tested adequately.To satisfy the MC/DC coverage criterion, during testing all of the below must be true at least once:...
Regression testing is any type of software testing that seeks to uncover new errors, or regressions, in existing functionality after changes have been made to a system, such as functional enhancements, patches or configuration changes....
A software metric is a measure of some property of a piece of software or its specifications. Since quantitative measurements are essential in all sciences, there is a continuous effort by computer science practitioners and theoreticians to bring similar approaches to software development...
Static program analysis is the analysis of computer software that is performed without actually executing programs built from that software In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code...
decisions in it can have up to 
paths within it; loop constructs can result in an infinite number of paths. Many paths may also be infeasible, in that there is no input to the program under test that can cause that particular path to be executed. However, a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem