Precision (arithmetic)
Encyclopedia
The precision of a value describes the number of digit
s that are used to express that value. In a scientific setting this would be the total number of digits (sometimes called the significant figures
or significant digits) or, less commonly, the number of fractional digits or decimal places (the number of digits following the point
). This second definition is useful in financial and engineering applications where the number of digits in the fractional part has particular importance.
In both cases, the term precision can be used to describe the position at which an inexact result will be rounded. For example, in floating point
arithmetic, a result is rounded to a given or fixed precision, which is the length of the resulting significand
. In financial calculations, a number is often rounded to a given number of places (for example, to two places after the decimal separator
for many world currencies).
As an illustration, the decimal
quantity 12.345 can be expressed with various numbers of significant digits or decimal places. If insufficient precision is available then the number is rounded
in some manner to fit the available precision. The following table shows the results for various total precisions and decimal places rounded to the nearest value using the round-to-even method.
Note that it is often not appropriate to display a figure with more digits than that which can be measured. For instance, if a device measures to the nearest gram and gives a reading of 12.345 kg, it would create false precision
if you were to express this measurement as 12.34500 kg.
The representation of a positive number x to a precision of p significant digits has a numerical value that is given by the formula
For a negative number, the numerical value is minus that of the absolute value.
The number 0, to any precision, can be taken to be 0.
Numerical digit
A digit is a symbol used in combinations to represent numbers in positional numeral systems. The name "digit" comes from the fact that the 10 digits of the hands correspond to the 10 symbols of the common base 10 number system, i.e...
s that are used to express that value. In a scientific setting this would be the total number of digits (sometimes called the significant figures
Significant figures
The significant figures of a number are those digits that carry meaning contributing to its precision. This includes all digits except:...
or significant digits) or, less commonly, the number of fractional digits or decimal places (the number of digits following the point
Radix point
In mathematics and computing, a radix point is the symbol used in numerical representations to separate the integer part of a number from its fractional part . "Radix point" is a general term that applies to all number bases...
). This second definition is useful in financial and engineering applications where the number of digits in the fractional part has particular importance.
In both cases, the term precision can be used to describe the position at which an inexact result will be rounded. For example, in floating point
Floating point
In computing, floating point describes a method of representing real numbers in a way that can support a wide range of values. Numbers are, in general, represented approximately to a fixed number of significant digits and scaled using an exponent. The base for the scaling is normally 2, 10 or 16...
arithmetic, a result is rounded to a given or fixed precision, which is the length of the resulting significand
Significand
The significand is part of a floating-point number, consisting of its significant digits. Depending on the interpretation of the exponent, the significand may represent an integer or a fraction.-Examples:...
. In financial calculations, a number is often rounded to a given number of places (for example, to two places after the decimal separator
Decimal separator
Different symbols have been and are used for the decimal mark. The choice of symbol for the decimal mark affects the choice of symbol for the thousands separator used in digit grouping. Consequently the latter is treated in this article as well....
for many world currencies).
As an illustration, the decimal
Decimal
The decimal numeral system has ten as its base. It is the numerical base most widely used by modern civilizations....
quantity 12.345 can be expressed with various numbers of significant digits or decimal places. If insufficient precision is available then the number is rounded
Rounding
Rounding a numerical value means replacing it by another value that is approximately equal but has a shorter, simpler, or more explicit representation; for example, replacing $23.4476 with $23.45, or the fraction 312/937 with 1/3, or the expression √2 with 1.414.Rounding is often done on purpose to...
in some manner to fit the available precision. The following table shows the results for various total precisions and decimal places rounded to the nearest value using the round-to-even method.
| Precision | Rounded to significant digits | Rounded to decimal places |
|---|---|---|
| Five | 12.345 | 12.34500 |
| Four | 12.34 | 12.3450 |
| Three | 12.3 | 12.345 |
| Two | 12 | 12.34 |
| One | 1 × 101 | 12.3 |
| Zero | n/a | 12 |
Note that it is often not appropriate to display a figure with more digits than that which can be measured. For instance, if a device measures to the nearest gram and gives a reading of 12.345 kg, it would create false precision
False precision
False precision occurs when numerical data are presented in a manner that implies better precision than is actually the case; since precision is a limit to accuracy, this often leads to overconfidence in the accuracy as well.In science and engineering, convention dictates that...
if you were to express this measurement as 12.34500 kg.
The representation of a positive number x to a precision of p significant digits has a numerical value that is given by the formula
- roundRoundingRounding a numerical value means replacing it by another value that is approximately equal but has a shorter, simpler, or more explicit representation; for example, replacing $23.4476 with $23.45, or the fraction 312/937 with 1/3, or the expression √2 with 1.414.Rounding is often done on purpose to...
(10−n·x)·10n, where n = floor(log10 x) + 1 − p.
For a negative number, the numerical value is minus that of the absolute value.
The number 0, to any precision, can be taken to be 0.
See also
- False precisionFalse precisionFalse precision occurs when numerical data are presented in a manner that implies better precision than is actually the case; since precision is a limit to accuracy, this often leads to overconfidence in the accuracy as well.In science and engineering, convention dictates that...
- Accuracy and precisionAccuracy and precisionIn the fields of science, engineering, industry and statistics, the accuracy of a measurement system is the degree of closeness of measurements of a quantity to that quantity's actual value. The precision of a measurement system, also called reproducibility or repeatability, is the degree to which...
- Precision (computer science)Precision (computer science)In computer science, precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics, which describes the number of digits that are used to express a...
- Round-off errorRound-off errorA round-off error, also called rounding error, is the difference between the calculated approximation of a number and its exact mathematical value. Numerical analysis specifically tries to estimate this error when using approximation equations and/or algorithms, especially when using finitely many...
- IEEE754 (IEEE floating point standard)
- Computational Error and Complexity in Science and Engineering