NU4–NU6 - Rounding, chopping and precision

NU4–NU6 - Rounding, chopping and precision

NU4 explains limited numerical representation. NU5 investigates loss of relative precision when nearly equal quantities are subtracted. NU6 uses rounding and chopping to calculate individual, maximum, average and expected errors, while keeping the arithmetic model and its precision explicit.

A displayed number need not be exact

A calculation needs 10001−100\sqrt{10001}-100, a small positive number. In an arithmetic system retaining 55 significant figures, the square root becomes 100.00100.00. Subtracting 100100 then produces zero. How can an exact algebraic rearrangement recover a useful answer from the same precision?

The difficulty starts with representation. A finite store of digits cannot represent every real number exactly. For example, 1/3=0.333…1/3=0.333\ldots has no terminating decimal representation. A stored 0.33330.3333 is an approximation, even if it is displayed without a warning.

You already know

Signed absolute error is approximate minus exact. Relative error divides by the exact non-zero value. Rounding intermediate stages changes a calculation; retaining more digits at those stages can reduce avoidable error.

Display precision and stored precision can differ: a calculator or spreadsheet may show fewer digits than it retains internally. Changing the display alone is different from deliberately replacing a value by its rounded or chopped form before the next operation.

Limited precision

A numerical representation has limited precision when only a fixed amount of digit information is retained. Later arithmetic uses that retained approximation, so small representation errors may propagate or grow.

A decimal terminating fraction need not terminate in another base: 1/101/10 has no finite binary expansion, because a denominator containing a factor 55 cannot divide a power of 22.

The examples below use explicitly stated decimal arithmetic models, not a claim that every calculator stores exactly those decimal digits.

Key idea

A long-looking answer is not evidence that all its digits are accurate. Separate the exact input, the stored approximation and the displayed result.

The loss in the opening subtraction will become clearer once the rounding step and the small difference are tracked separately.

Round to the nearest value; chop by discarding digits

Rounding chooses a nearest representable value at the specified precision. Chopping simply removes the digits beyond that precision. For a negative number, chopping moves towards zero; it is not the same as rounding down towards negative infinity.

Count decimal places from the decimal point. Count significant figures from the first non-zero digit: 0.00478620.0047862, for example, becomes 0.004790.00479 to 33 significant figures, not to 33 decimal places. At an exact halfway value, use the stated midpoint convention; the signed examples below avoid halfway cases.

Error bounds for a fixed decimal step

Suppose the retained values are spaced by h>0h>0. Rounding to the nearest value gives

∣eround∣≤h2|e_{\mathrm{round}}|\le\frac h2

Chopping towards zero gives

−h<echop≤0for a positive exact value-h<e_{\mathrm{chop}}\le0\quad\text{for a positive exact value} 0≤echop<hfor a negative exact value0\le e_{\mathrm{chop}}<h\quad\text{for a negative exact value}

These use signed error e=X−xe=X-x. A fixed decimal step applies within the specified precision range.

Not in the formula booklet: learn it.

For 22 decimal places, h=0.01h=0.01. Significant-figure spacing also depends on the number's size, so identify the retained place before applying a bound.

Worked example 1: a positive input

Question. Round and chop the exact value x=4.7862x=4.7862 to 33 significant figures. Find the signed absolute error in each result.

Plan. The third significant figure is in the hundredths place; inspect or discard the later digits.

Xround=4.79nearest hundredtheround=4.79−4.7862=0.0038signed discrepancyXchop=4.78discard later digitsechop=4.78−4.7862=−0.0062signed discrepancy\begin{aligned} X_{\mathrm{round}}&=4.79&&\text{nearest hundredth}\\ e_{\mathrm{round}}&=4.79-4.7862=0.0038&&\text{signed discrepancy}\\ X_{\mathrm{chop}}&=4.78&&\text{discard later digits}\\ e_{\mathrm{chop}}&=4.78-4.7862=-0.0062&&\text{signed discrepancy} \end{aligned}

Why this step? Chopping does not inspect the next digit and round up. It leaves the retained digits unchanged.

Answer. Rounded value 4.794.79, error 0.00380.0038; chopped value 4.784.78, error −0.0062-0.0062.

Check. 0.0038<0.005=h/20.0038<0.005=h/2, and the positive-value chopping error lies between −0.01-0.01 and zero.

Change only the sign of the exact input. Nearest rounding reflects the previous answer, while chopping still discards digits towards zero.

Worked example 2: a negative input

Question. Round and chop x=−4.7862x=-4.7862 to 33 significant figures, and find both signed absolute errors.

Plan. Apply the precision rule to the magnitude, retain the minus sign, then calculate signed errors.

Xround=−4.79nearest hundredtheround=−4.79−(−4.7862)=−0.0038signed errorXchop=−4.78discard digits towards zeroechop=−4.78−(−4.7862)=0.0062signed error\begin{aligned} X_{\mathrm{round}}&=-4.79&&\text{nearest hundredth}\\ e_{\mathrm{round}}&=-4.79-(-4.7862)=-0.0038&&\text{signed error}\\ X_{\mathrm{chop}}&=-4.78&&\text{discard digits towards zero}\\ e_{\mathrm{chop}}&=-4.78-(-4.7862)=0.0062&&\text{signed error} \end{aligned}

Why this step? Chopping to −4.79-4.79 would move away from zero and change a retained digit. That is not digit truncation.

Answer. Rounded value −4.79-4.79, error −0.0038-0.0038; chopped value −4.78-4.78, error 0.00620.0062.

Check. The magnitudes match the positive-input errors, but their signs reverse.

A negative signed error means the approximation lies below the exact value on the number line. It does not necessarily mean that its magnitude is too small.

Watch out

Wrong: chop −4.7862-4.7862 to −4.79-4.79 because chopping “goes down”. Right: chopping discards digits, giving −4.78-4.78. It differs from the floor operation on negative numbers.

Change the size of the negative input, so the retained decimal place changes too.

Unknown exact values need interval bounds

If you know only a rounded or chopped approximation, you do not know the actual error. You know an interval containing the exact value. Use that interval to obtain a bound, rather than substitute the approximate value into a denominator defined to be exact.

For a fixed approximation PP and a positive exact value pp,

r=P−pp=Pp−1r=\frac{P-p}{p}=\frac Pp-1

This decreases as pp increases. The endpoints therefore identify the extreme signed relative errors. This endpoint argument is not a separate booklet formula; it follows from the definition of relative error.

Worked example 3: a rounded integer

Question. The approximation P=320P=320 was obtained by rounding a positive exact value pp to the nearest integer. In halfway cases, positive values are rounded upwards. Find the exact-value interval and the maximum possible magnitude of the relative error.

Plan. Write the rounding interval, then compare the relative errors at its two bounding endpoints.

319.5≤p<320.5rounding intervalrlower endpoint=0.5319.5=1639positive errorrupper endpoint ⟶ −0.5320.5=−1641limiting negative error1639>1641compare magnitudes\begin{aligned} 319.5&\le p<320.5&&\text{rounding interval}\\ r_{\mathrm{lower\ endpoint}}&=\frac{0.5}{319.5}=\frac1{639}&&\text{positive error}\\ r_{\mathrm{upper\ endpoint}}&\ \longrightarrow\ \frac{-0.5}{320.5}=-\frac1{641}&&\text{limiting negative error}\\ \frac1{639}&>\frac1{641}&&\text{compare magnitudes} \end{aligned}

Why this step? The exact denominator changes across the interval. Using 0.5/3200.5/320 would miss the worst case, which occurs at the smallest possible exact value.

Answer. 319.5≤p<320.5319.5\le p<320.5, and the maximum relative-error magnitude is 1/6391/639, approximately 0.0015650.001565.

Check. At p=319.5p=319.5, the approximation is exactly 0.50.5 too large, and the resulting relative error attains the stated maximum.

Change only the representation rule: the same positive approximation 320320 is now obtained by chopping.

Worked example 4: a chopped integer

Question. A positive exact value qq is chopped to give Q=320Q=320. Find its possible interval and a sharp bound for the magnitude of relative error.

Plan. Use the interval from discarding fractional digits, then locate the largest error magnitude as qq approaches its upper endpoint.

320≤q<321chopping intervalr=320−qqsigned relative error−1321<r≤0endpoint bounds∣r∣<1321sharp magnitude bound\begin{aligned} 320&\le q<321&&\text{chopping interval}\\ r&=\frac{320-q}{q}&&\text{signed relative error}\\ -\frac1{321}&<r\le0&&\text{endpoint bounds}\\ |r|&<\frac1{321}&&\text{sharp magnitude bound} \end{aligned}

Why this step? Both the discrepancy and the exact denominator approach their limiting values together. Using 1/3201/320 is not the sharp bound.

Answer. 320≤q<321320\le q<321, with relative-error magnitude less than 1/3211/321, approximately 0.0031150.003115.

Check. Values just below 321321 give errors as close to −1/321-1/321 as desired. The bound is a limiting value; the excluded endpoint means it is not attained.

A question may call such a limiting bound the maximum possible error. Keep the endpoint distinction clear, and use the exact-value interval to justify the number.

Exam tip

In a relative-error bound, determine which exact value gives the extreme ratio. Do not replace that denominator by the rounded or chopped approximation merely because it is the number printed in the question.

Change only the sign of the chopped approximation.

Watch out

Wrong: 1/3201/320 is the sharp relative-error bound after chopping to 320320. The definition divides by the unknown exact value, and the largest discrepancy occurs as that value approaches 321321. Correct: ∣r∣<1/321|r|<1/321, with the limiting bound not attained.

Keep the exact denominator and endpoint inclusion in view for the negative case below.

A worst case and an expectation answer different questions

A maximum-error bound allows all input errors to reinforce one another. An expected error instead uses a probability model for the discarded digits. You cannot infer an average error from a precision statement alone.

Continue this lesson.

You’re previewing NovaLearn without an account. Create a free account to read the rest.

Create free account