MD9, MD13, MD14 - Outliers, missing values and data cleaning | OCR A-Level Mathematics B (MEI)

Apply MEI's inclusive outlier rules and make cleaning decisions that respect missing values, genuine extremes and paired observations. You will use current LDS9 measurements and choose displays whose scales and time order support a fair conclusion.

MEI outlier boundaries include equality

Should a measurement exactly on an outlier boundary be investigated? For MEI, yes: the definitions say at least the stated distance. The small word “least” changes a boundary decision, even when every calculation is correct.

Consider a mass of $28$ kg in data with quartiles $8$ and $16$ kg. It lies exactly $12$ kg above the upper quartile, which is $1.5$ times the IQR. It meets MEI's rule.

You already know

The IQR is $Q_3-Q_1$ . MEI sample SD $s$ uses squared-deviation divisor $n-1$ . When a question supplies the summaries, use them as given rather than adding a candidate observation and recalculating them.

A numerical threshold is one way to identify inconsistency. A plotted relationship may reveal another.

Outlier

An outlier is an item inconsistent with the rest of the data. In a scatter diagram this can be a pair far from the relationship, even when neither coordinate is extreme on its own. A large coordinate that follows the pattern is not automatically an outlier.

MEI permits the term for values at least two SDs from the mean, or at least $1.5\mathrm{IQR}$ beyond the nearer quartile.

MEI inclusive outlier rules

Using $\mathrm{IQR}=Q_3-Q_1$ , the quartile rule is

x\le Q_1-1.5\mathrm{IQR}\quad\text{or}\quad x\ge Q_3+1.5\mathrm{IQR}

The standard-deviation rule is

|x-\bar x|\ge2s

Equality is included. The quartile rule concerns the tails beyond the quartiles, not the region between them.

Not in the formula sheet: learn it.

Use enough precision to settle a near-boundary value. Rounded summaries may be insufficient to decide whether a point is exactly on the theoretical boundary.

Worked example 1: include the fence

Question. A mass dataset has $Q_1=8$ , $Q_3=16$ , mean $12$ and sample SD $3$ , all in kg. Use each rule separately to classify $18$ kg and $28$ kg.

Plan. Calculate the two sets of boundaries and include equality.

\begin{aligned} \mathrm{IQR}&=16-8=8 && \text{quartile difference} \\ L&=8-1.5(8)=-4,\quad U=16+1.5(8)=28 && \text{outward fences} \\ \bar x-2s&=6,\quad\bar x+2s=18 && \text{mean-based boundaries} \end{aligned}

Why include $28$ and $18$ ? Each is exactly the required distance from its relevant reference point, so each satisfies “at least”.

Answer. The IQR rule flags $28$ kg only. The SD rule flags both $18$ kg and $28$ kg.

Check. $|18-12|=6=2s$ , and $28-16=12=1.5\mathrm{IQR}$ .

Change only the second candidate, from $28$ to $27.9$ . Moving just inside a fence can change one rule's conclusion while leaving the other unchanged.

Worked example 2: move inside one boundary

Question. With the same summaries $Q_1=8$ , $Q_3=16$ , mean $12$ , sample SD $3$ kg, classify $18$ kg and $27.9$ kg using each rule.

Plan. Retain the boundaries and compare each candidate with the relevant outer regions.

\begin{aligned} -4&<18<27.9<28 && \text{both inside the quartile fences} \\ |18-12|&=6\ge6 && \text{on the SD boundary} \\ |27.9-12|&=15.9\ge6 && \text{beyond the SD boundary} \end{aligned}

Why can the rules disagree? Quartiles and mean/SD summarise different features of the same distribution; they are alternative definitions, not guaranteed equivalent tests.

Answer. Neither candidate is flagged by the IQR rule. Both are flagged by the SD rule.

Check. The changed mass remains well above the upper SD boundary $18$ , despite dropping below the IQR fence $28$ .

In a box plot, check whether whiskers show extrema or the furthest non-outliers. When asked where actual outliers lie, intersect the theoretical outlier regions with the values shown: a possible lower-tail region can contain no observations. A genuine unusual member of the population may be worth keeping, so an outlier flag is not a deletion instruction.

Watch out

Wrong: “Use the median plus or minus $1.5\mathrm{IQR}$ .” Right: move outwards from the nearer quartile, $Q_1$ or $Q_3$ , and include the boundary for MEI.

Apply both definitions to another dataset.

Cleaning decisions depend on the analysis

A useful cleaning decision starts with the variables you need. Deleting every row with any missing cell may throw away perfectly good measurements.

Data cleaning

Data cleaning deals with missing values, errors, inconsistent formats and possible outliers using source evidence and the intended analysis. Preserve the source, work on a copy, and record corrections, exclusions and the valid count.

Check the observational unit, units, category codes and identifiers. Repeated values from different people are not duplicates. Correct an error if its true value is known; otherwise mark the affected value missing for relevant calculations. A genuine extreme observation remains part of the data unless a stated, defensible selection rule excludes it.

The current MEI LDS9 contains $200$ people aged $16$ or over from the US NHANES $2003$ – $2004$ survey. Its missing code is “#N/A”, deliberately chosen to prevent software reading a blank as zero. The following are the first five people in the workbook's data sheet. Row labels identify positions, not new survey identifiers.

Data-sheet row	Age, years	Pulse, beats per minute
$2$	$34$	$74$
$3$	$85$	#N/A
$4$	$48$	$92$
$5$	$61$	$70$
$6$	$68$	$68$

Worked example 3: select for a pulse calculation

Question. From the five-row LDS9 extract above, find the observed mean pulse and the number of usable age–pulse pairs. Give the mean to $3$ significant figures.

Plan. Filter on pulse availability, keeping each remaining person's age and pulse together.

\begin{aligned} n_{\text{pulse}}&=4 && \text{row three has no pulse} \\ \overline{\text{pulse}}&=\frac{74+92+70+68}{4}=76 && \text{average four observed values} \\ n_{\text{pairs}}&=4 && \text{those four ages are present} \end{aligned}

Why exclude the whole pair for the scatter diagram? Keeping age $85$ and sliding another person's pulse into that row would create a false pair.

Answer. Observed mean pulse $76.0$ beats per minute; four complete age–pulse pairs, from rows $2,4,5,6$ .

Check. The mean is between the smallest and largest observed pulses, $68$ and $92$ .

Change only the mean being requested: use age rather than pulse. The source rows remain exactly the same.

Worked example 4: keep a valid value in another analysis

Question. From the same five-row extract, find the mean age and the number of usable age–pulse pairs. Give the mean age to $3$ significant figures.

Plan. Use every observed age for the one-variable mean, but still require both fields for a pair.

\begin{aligned} n_{\text{age}}&=5 && \text{all ages are present} \\ \overline{\text{age}}&=\frac{34+85+48+61+68}{5}=59.2 && \text{include row three's valid age} \\ n_{\text{pairs}}&=4 && \text{paired requirements unchanged} \end{aligned}

Why does the valid count depend on the calculation? Missing pulse does not erase a known age. A mean age needs one field; an age–pulse plot needs two in the same row.

Answer. Mean age $59.2$ years for these five people; four complete age–pulse pairs.

Check. Excluding row $3$ unnecessarily would give $211/4=52.75$ years, a different mean for a different subset.

Document unit conversions too. Masses $1.8$ kg, $2100$ g, missing and $2.4$ kg give three observed masses with mean $2.1$ kg and sample SD $0.300$ kg. Entering a zero for missingness invents a fourth measurement. Even careful omission can leave bias if missing observations differ systematically from the observed ones.

Continue this lesson.

You’re previewing NovaLearn without an account. Create a free account to read the rest.

Create free account

Last Accessed

Pure Mathematics

Statistics

Mechanics

MD9, MD13, MD14 - Outliers, missing values and data cleaning

MEI outlier boundaries include equality

Cleaning decisions depend on the analysis

Continue this lesson.

Learn faster with Nova