2.02h–2.02j - Outliers, missing values and data cleaning

2.02h–2.02j - Outliers, missing values and data cleaning

Identify possible outliers with H240's precise boundary rules, clean missing or erroneous observations without changing their meaning, and choose displays that answer the statistical question. You will use the census large data set to distinguish a large count from a large proportion.

Outlier rules identify values to investigate

Can making a spreadsheet look tidier make its answer less truthful? Suppose a blank mass is replaced by zero, or an unusually large but genuine observation is deleted. Both changes make calculations easier; neither is justified just by appearance. First learn to flag values, then decide what the evidence allows you to do with them.

You already know

The interquartile range is IQR=Q3−Q1\mathrm{IQR}=Q_3-Q_1. Standard deviation measures spread about the mean. Both describe the data; neither decides whether an observation was recorded correctly.

A conspicuous point may be meaningful, mistaken or part of a different group. Investigate its context before choosing an action.

Possible outlier

A possible outlier is an observation unusually far from the rest of the data or inconsistent with a visible relationship. In a scatter diagram, a moderate journey distance with an exceptionally long journey time can be inconsistent with the pattern, while a very long journey following the pattern need not be.

H240's two numerical rules use more than, so equality at a boundary is not flagged by that rule.

H240 outlier boundaries

The quartile rule flags

x<Q1−1.5IQRorx>Q3+1.5IQRx<Q_1-1.5\mathrm{IQR}\quad\text{or}\quad x>Q_3+1.5\mathrm{IQR}

The standard-deviation rule flags

∣x−xˉ∣>2SD|x-\bar x|>2\mathrm{SD}

The quartile rule concerns values beyond the nearer outer quartile, not points between the quartiles.

Not in the formula sheet: learn it.

The rules need not agree: each uses different information. State which rule you applied, and retain enough precision to decide a case near a fence.

Worked example 1: equality is a boundary case

Question. Parcel masses have Q1=12Q_1=12, Q3=20Q_3=20, mean 1616 and SD 55, all in kg. Use each rule separately to classify 2626 kg and 3232 kg.

Plan. Calculate both pairs of boundaries, then use strict inequalities.

IQR=20−12=8middle-half spreadL=12−1.5(8)=0,U=20+1.5(8)=32quartile fencesxˉ−2SD=6,xˉ+2SD=26mean-based boundaries\begin{aligned} \mathrm{IQR}&=20-12=8 && \text{middle-half spread} \\ L&=12-1.5(8)=0,\quad U=20+1.5(8)=32 && \text{quartile fences} \\ \bar x-2\mathrm{SD}&=6,\quad \bar x+2\mathrm{SD}=26 && \text{mean-based boundaries} \end{aligned}

Why distinguish >> from ≥\ge? H240's definitions say more than the stated distance, so the fence itself is excluded.

Answer. The IQR rule flags neither value: 3232 is on its fence. The SD rule flags 3232, but not 2626, which is on its boundary.

Check. ∣26−16∣=10=2(5)|26-16|=10=2(5), whereas ∣32−16∣=16>10|32-16|=16>10.

Change only the second candidate, from 3232 to 32.132.1. A tiny numerical change matters when it crosses a boundary.

Worked example 2: cross a fence

Question. With the same summaries Q1=12Q_1=12, Q3=20Q_3=20, mean 1616, SD 55 kg, classify 2626 kg and 32.132.1 kg using each rule.

Plan. Keep the existing boundaries and compare the changed observation with them.

IQR fences:0,32summaries unchanged32.1>32strictly beyond the upper fenceSD boundaries:6,26same mean and SD32.1>26beyond the upper boundary\begin{aligned} \text{IQR fences}&:0,32 && \text{summaries unchanged} \\ 32.1&>32 && \text{strictly beyond the upper fence} \\ \text{SD boundaries}&:6,26 && \text{same mean and SD} \\ 32.1&>26 && \text{beyond the upper boundary} \end{aligned}

Why retain the supplied summaries? This question compares specified candidates with a stated distribution; it does not ask you to add candidates to the data and recalculate the summaries.

Answer. Both rules flag 32.132.1 kg. Neither flags 2626 kg.

Check. The second candidate moved outwards, so it cannot stop being flagged by the SD rule.

In a box plot, establish what the whiskers represent: they may show extrema or the furthest non-outlying observations. If a question asks for the outliers actually present, compare the theoretical fences with the displayed data range. A region that contains no observed values contributes no actual outliers.

Watch out

Wrong: “Anything more than 1.5IQR1.5\mathrm{IQR} from the median is an outlier.” Right: move 1.5IQR1.5\mathrm{IQR} outwards from Q1Q_1 and Q3Q_3, then use the strict inequalities.

Try a new set of boundaries.

Cleaning preserves the meaning of observations

Cleaning makes data suitable for the intended investigation while preserving a record of what changed and why.

Data cleaning

Data cleaning identifies and deals with missing values, recording errors, inconsistent formats and possible outliers using the variable definitions and source evidence. An unusual genuine observation belonging to the target population is not automatically an error.

Keep the source unchanged and work on a copy. Establish the unit of observation, units, coding and target population. Check suspected duplicates by identifier: equal measurements from different people are not duplicates. Convert documented units, correct confirmed errors from reliable evidence, and mark an uncorrectable erroneous value as missing for the relevant analysis. Log exclusions and the count left.

Worked example 3: a blank is not a measurement

Question. Four mass records are 2.42.4 kg, 25002500 g, a blank, and 2.82.8 kg. The blank means unrecorded. Calculate the observed mean in kg and state a defensible treatment of the blank.

Plan. Convert units, retain the three observed values, and report the count.

2500 g=2.5 kgconvert to a common unitn=3blank is unrecordedxˉ=2.4+2.5+2.83=7.73≈2.57average observed masses\begin{aligned} 2500\,\mathrm g&=2.5\,\mathrm{kg} && \text{convert to a common unit} \\ n&=3 && \text{blank is unrecorded} \\ \bar x&=\frac{2.4+2.5+2.8}{3}=\frac{7.7}{3}\approx2.57 && \text{average observed masses} \end{aligned}

Why not use zero? Zero would invent a fourth mass, changing both the total's meaning and the divisor.

Answer. Observed mean 2.572.57 kg, to 33 significant figures, based on three records. Keep the blank missing and document its omission from this calculation.

Check. The mean lies between 2.42.4 and 2.82.8 kg. Coding the blank as zero would give 7.7/4=1.9257.7/4=1.925 kg, below every observed mass.

Change only the missing record: source evidence now recovers its true value as 2.62.6 kg. This is a correction supported by evidence, rather than a guess.

Worked example 4: recover a confirmed measurement

Question. The same four records are 2.42.4 kg, 25002500 g, a recovered 2.62.6 kg, and 2.82.8 kg. Calculate the mean and describe the cleaning log.

Plan. Include the recovered observation and use the new valid count.

2500 g=2.5 kgsame unit conversionn=4all four masses now knownxˉ=2.4+2.5+2.6+2.84=2.575include the recovered value\begin{aligned} 2500\,\mathrm g&=2.5\,\mathrm{kg} && \text{same unit conversion} \\ n&=4 && \text{all four masses now known} \\ \bar x&=\frac{2.4+2.5+2.6+2.8}{4}=2.575 && \text{include the recovered value} \end{aligned}

Why is this inclusion justified? A source record supplies the measurement. Replacing a blank by a convenient average would manufacture information.

Answer. Mean 2.582.58 kg, to 33 significant figures. Log the unit conversion, the recovered value and its evidence, and that all four observations are now used.

Check. The recovered value 2.62.6 is just above the previous observed mean, so including it raises the mean slightly.

Continue this lesson.

You’re previewing NovaLearn without an account. Create a free account to read the rest.

Create free account