All Topics  
Precision (arithmetic)

 

 
   Email Print
   Bookmark   Link
 

 

Precision (arithmetic)
 
 
  Topics Precision (arithmetic) Topic Home

The precision of a value describes the number of digit
Numerical digit

In mathematics and computer science, a digit is a symbol used in numerals , to represent numbers, in Positional notation numeral systems. The name "digit" comes from the fact that the 10 digits of the hands correspond to the 10 symbols of the common base 10 number system, i.e....
s that are used to express that value. In a scientific setting this would be the total number of digits (sometimes called the significant figures
Significant figures

The significant figures of a number are those Numerical digit that carry meaning contributing to its accuracy . This includes all digits except:...
 or significant digits) or, less commonly, the number of fractional digits or decimal places (the number of digits following the point
Radix point

In mathematics and computing, a radix point is the symbol used in numerical representations to separate the integer part of a number from its fraction part ....
). This second definition is useful in financial and engineering applications where the number of digits in the fractional part has particular importance.

In both cases, the term precision can be used to describe the position at which an inexact result will be rounded.






Discussion
Ask a question about 'Precision (arithmetic)'
Start a new discussion about 'Precision (arithmetic)'
Answer questions from other users
Full Discussion Forum



Encyclopedia


The precision of a value describes the number of digit
Numerical digit

In mathematics and computer science, a digit is a symbol used in numerals , to represent numbers, in Positional notation numeral systems. The name "digit" comes from the fact that the 10 digits of the hands correspond to the 10 symbols of the common base 10 number system, i.e....
s that are used to express that value. In a scientific setting this would be the total number of digits (sometimes called the significant figures
Significant figures

The significant figures of a number are those Numerical digit that carry meaning contributing to its accuracy . This includes all digits except:...
 or significant digits) or, less commonly, the number of fractional digits or decimal places (the number of digits following the point
Radix point

In mathematics and computing, a radix point is the symbol used in numerical representations to separate the integer part of a number from its fraction part ....
). This second definition is useful in financial and engineering applications where the number of digits in the fractional part has particular importance.

In both cases, the term precision can be used to describe the position at which an inexact result will be rounded. For example, in floating point
Floating point

In computing, floating point describes a system for numerical representation in which a String of digits represents a rational number.The term floating point refers to the fact that the radix point can "float": that is, it can be placed anywhere relative to the Significant figures of the number....
 arithmetic, a result is rounded to a given or fixed precision, which is the length of the resulting significand
Significand

The significand is the part of a floating point that contains its significant digits. Depending on the interpretation of the exponent, the significand may be considered to be an integer or a fraction ....
. In financial calculations, a number is often rounded to a given number of places (for example, to two places after the decimal separator
Decimal separator

In a Positional notation numeral system, the decimal separator is a symbol used to mark the boundary between the integer and the fraction parts of a decimal numeral....
 for many world currencies).

As an illustration, the decimal
Decimal

The decimal numeral system has 10 as its Base . It is the most widely used numeral system....
 quantity 12.345 can be expressed with various numbers of significant digits or decimal places. If insufficient precision is available then the number is rounded
Rounding

Rounding involves reducing the number of significant digits in a number. The result of rounding is a "shorter" number having fewer non-zero digits yet similar in magnitude....
 in some manner to fit the available precision. The following table shows the results for various total precisions and decimal places, with the results rounded to nearest where ties round up or to an even digit (the most common rounding modes).

Note that it is often not appropriate to display a figure with more digits than that which can be measured. For instance, if a device measures to the nearest gram
Gram

The gram , ; symbol g, is a Physical unit of mass.Originally defined as "the absolute weight of a volume of pure water equal to the cube of the hundredth part of a metre, and at the temperature of melting ice" , a gram is now defined as one one-thousandth of the SI base unit, the kilogram, or Scientific notation kg, which itself is...
 and gives a reading of 12.345 kg
Kilogram

The kilogram or kilogrammeThe spelling kilogram is used by the International Committee for Weights and Measures and the U.S....
, it would create false precision
False precision

False precision occurs when numerical data are presented in a manner that implies better Accuracy and precision than is actually the case; since precision is a limit to accuracy, this often leads to overconfidence in the accuracy as well....
 if you were to express this measurement as 12.34500 kg.









 
Precision
Rounded to
significant digits
Rounded to
decimal places
Five
12.345
12.34500
Four
12.35
12.3450
Three
12.3
12.345
Two
12
12.35
One
1 × 101
12.3
Zero
n/a
12


The representation of a positive number x to a precision of p significant digits has a numerical value that is given by the formula
round
Rounding

Rounding involves reducing the number of significant digits in a number. The result of rounding is a "shorter" number having fewer non-zero digits yet similar in magnitude....
(10-n·x)·10n, where n = floor(log10 x) + 1 - p.
For a negative number, the numerical value is minus that of the absolute value. The number 0, to any precision, can be taken to be 0.

See also

  • Round-off error
    Round-off error

    A round-off error, also called rounding error, is the difference between the calculated approximation of a number and its exact mathematical value....
  • Precision (computer science)
    Precision (computer science)

    In computer science, precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits....
  • False precision
    False precision

    False precision occurs when numerical data are presented in a manner that implies better Accuracy and precision than is actually the case; since precision is a limit to accuracy, this often leads to overconfidence in the accuracy as well....
  • IEEE754 (IEEE floating point standard)