4.5.4.5–4.5.4.6 - Rounding, absolute and relative errors

Finite binary precision

A computer field contains only finitely many bits, so it can represent only finitely many distinct values.

A fixed-point field with four fractional bits has values separated by $2^{-4}=1/16=0.0625$. It can represent $0.375=6/16=0.0110_2$ exactly, but cannot represent $0.3$ exactly at that resolution. A nearby available value has to be chosen even though 0.3 is an ordinary terminating decimal.

An exact finite binary fraction is an integer divided by a power of two. Reducing the fraction to its lowest terms provides a useful test: if the denominator is a power of two, a terminating binary expansion exists. It still needs enough bits in the chosen field. For example, $3/8$ terminates as $0.011_2$, but a field with only two fractional positions cannot hold it exactly.

In contrast, $0.1_{10}=1/10$ has denominator $2\times5$, not a power of two. Its binary expansion repeats: $0.0001100110011\ldots_2$. No finite number of binary fractional positions can store this exact value. Rational does not mean exactly representable in every base or width.

Floating point moves the effective binary point using a power-of-two exponent, but its mantissa is still finite. Moving the point does not turn a repeating binary expansion into a terminating one. Floating point also approximates values when the required significant digits exceed the mantissa allocation. An exactly representable input can produce a result that needs approximation after a calculation; stored approximations can affect later calculations as well.

Choosing a finite approximation

When a value lies between available representations, the approximation rule determines which one is stored.

Truncation discards digits beyond the retained positions. For a positive value this chooses the lower available non-negative binary fraction. Rounding to nearest chooses the available value with the smallest distance from the exact value. If two distances are equal, a tie rule is needed; use the rule supplied by the task rather than assuming a particular rule.

Consider $0.1$ stored with four fractional bits. Truncating its expansion to $0.0001_2$ gives 0.0625. Rounding to nearest chooses $0.0010_2$, or 0.125: the distances are $0.1-0.0625=0.0375$ and $0.125-0.1=0.025$. Rounding produces the closer approximation here, but it is still not exact.

[DIAGRAM: asset_slug: four_bit_rounding_grid; description: Exact four-fractional-bit grid showing 0.1 between 0.0625 and 0.125, with the smaller rounding distance to 0.125; caption: Four-bit fractional grid showing truncation to 0.0625 and nearest rounding to 0.125 for the exact value 0.1]

For a non-negative binary fraction, look at the first discarded bit when rounding to nearest. If it is 0, the retained value is nearer. If it is 1, increase the retained pattern by one unit in its least-significant position, subject to the stated tie rule when all subsequent discarded bits are zero. For example, $0.10111_2$ rounds to three fractional positions as $0.110_2$: the discarded .11 means the value is beyond the midpoint above $0.101_2$.

For signed values, decide by numerical distance to the representable values. Do not apply a positive digit rule blindly to a negative two's-complement pattern. This lesson's worked truncations are explicitly non-negative; “truncate towards zero” is a separate rule a task may specify for signed values.

The fixed-point spacing is constant. Within the format's range, nearest rounding has absolute error at most half that spacing. Four fractional bits give a maximum of $1/32=0.03125$. This bound assumes the target is in range; rounding cannot solve an out-of-range value. Floating-point spacing depends on the exponent: with the same mantissa precision, larger magnitudes generally have larger gaps between representable values.

Absolute and relative error

Absolute error gives the distance from the exact value, while relative error compares that distance with the exact quantity's magnitude.

Absolute error is non-negative and has the same units as the quantity. Relative error is dimensionless and uses the exact value's magnitude in the denominator, not the approximation. For exact $0.1$ stored as 0.125, absolute error is 0.025 and relative error is $0.025/0.1=0.25$, or 25%. For exact $-2.5$ stored as $-2.46875$, absolute error is 0.03125 and relative error is $0.03125/2.5=0.0125$, or 1.25%. The absolute signs prevent a negative error for a negative exact value.

The same absolute error may be negligible for a large number and substantial for a small one. Suppose the absolute error is 0.001 in each case:

For magnitudes near one, the numerical sizes of absolute and relative error are similar because division by a number near one changes little. That comparison uses the stated units: absolute error changes if the quantity is converted from metres to millimetres; relative error does not, because numerator and denominator acquire the same scale factor.

A smaller absolute error need not mean a smaller relative error. Storing an exact 1000 as 1000.1 gives absolute error 0.1 but relative error 0.0001. Storing an exact 0.01 as 0.011 gives the smaller absolute error 0.001 but the larger relative error 0.1. Choose the measure that addresses the application: an absolute tolerance bounds a physical distance, while a relative tolerance bounds an error as a fraction of the magnitude.

At an exact value of zero, the usual relative-error formula divides by zero and is undefined. An absolute error can still be reported. Near zero, even a small absolute error can give a very large relative error, so a system may need an absolute tolerance rather than an unqualified percentage.