Precision (computer science)
Encyclopedia
In computer science
, precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics
, which describes the number of digits that are used to express a value.
. The number of bits used to store a number will often cause some loss of accuracy. An example would be to store sin(0.1) in IEEE single precision floating point standard. The error is then often magnified as subsequent computations are made to the data (it can also be reduced).
Computer science
Computer science or computing science is the study of the theoretical foundations of information and computation and of practical techniques for their implementation and application in computer systems...
, precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics
Precision (arithmetic)
The precision of a value describes the number of digits that are used to express that value. In a scientific setting this would be the total number of digits or, less commonly, the number of fractional digits or decimal places...
, which describes the number of digits that are used to express a value.
Rounding error
Precision is often the source of rounding errors in computationComputation
Computation is defined as any type of calculation. Also defined as use of computer technology in Information processing.Computation is a process following a well-defined model understood and expressed in an algorithm, protocol, network topology, etc...
. The number of bits used to store a number will often cause some loss of accuracy. An example would be to store sin(0.1) in IEEE single precision floating point standard. The error is then often magnified as subsequent computations are made to the data (it can also be reduced).
See also
- Integer (computer science)Integer (computer science)In computer science, an integer is a datum of integral data type, a data type which represents some finite subset of the mathematical integers. Integral data types may be of different sizes and may or may not be allowed to contain negative values....
- Arbitrary-precision arithmeticArbitrary-precision arithmeticIn computer science, arbitrary-precision arithmetic indicates that calculations are performed on numbers whose digits of precision are limited only by the available memory of the host system. This contrasts with the faster fixed-precision arithmetic found in most ALU hardware, which typically...
- Precision (arithmetic)Precision (arithmetic)The precision of a value describes the number of digits that are used to express that value. In a scientific setting this would be the total number of digits or, less commonly, the number of fractional digits or decimal places...
- IEEE754 (IEEE floating point standard)