Surf Wiki
Save to docs
general/approximations

From Surf Wiki (app.surf) — the open knowledge base

Precision (computer science)

Measure of detail a quantity is expressed in

Measure of detail a quantity is expressed in

In computer science, the precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics, which describes the number of digits that are used to express a value.

Some of the standardized floating-point precision formats are:

  • Half-precision floating-point format
  • Single-precision floating-point format
  • Double-precision floating-point format
  • Quadruple-precision floating-point format
  • Octuple-precision floating-point format

Of these, octuple-precision format is rarely used. The names refer to the size of a computer word, commonly 32-bits on many current computer architectures. A single-precision value occupies one word, a half-precision half of a word, double-precision two words, etc. The single- and double-precision formats are most widely used and supported on nearly all platforms. The use of half-precision format and minifloat formats has been increasing especially in the field of machine learning since many machine learning algorithms are inherently error-tolerant.

Rounding errors

Precision is often the source of rounding errors in computation. The number of bits used to store a number will often cause some loss of accuracy. An example would be to store "sin(0.1)" in IEEE single-precision floating point standard. The error is then often magnified as subsequent computations are made using the data (although it can also be reduced).

References

References

  1. "Precision".
Info: Wikipedia Source

This article was imported from Wikipedia and is available under the Creative Commons Attribution-ShareAlike 4.0 License. Content has been adapted to SurfDoc format. Original contributors can be found on the article history page.

approximations computer-data
Want to explore this topic further?

Ask Mako anything about Precision (computer science) — get instant answers, deeper analysis, and related topics.

Research with Mako

Free with your Surf account

Content sourced from Wikipedia, available under CC BY-SA 4.0.

This content may have been generated or modified by AI. CloudSurf Software LLC is not responsible for the accuracy, completeness, or reliability of AI-generated content. Always verify important information from primary sources.

Report