Statistics 2.4 - Outliers in data sets

Statistics 2.4 - Outliers in data sets

Recognise possible outliers, apply a supplied numerical rule and interpret unusual points on diagrams. Clean missing values, coding issues and genuine errors without automatically deleting valid extremes, and explain how those decisions affect the conclusions.

An outlier is a flag to investigate

Should a very wet day be deleted just because most days were much drier? Pearson's Large Data Set records 51.6 mm51.6\,\mathrm{mm} at Heathrow on 26 August 2015. That is a real entry in the supplied data, rather than a made-up mistake. An unusual rainfall value may describe the very event you need to study.

You already know

The IQR is Q3−Q1Q_3-Q_1. A standard deviation measures spread about a mean. To use a strict inequality such as x>48x>48, the boundary value 4848 itself does not qualify.

A flag tells you which record to investigate, not whether the instrument or typist was wrong. Check the original record, units, weather conditions and intended population before deciding. A rainfall reading of 20.6 mm20.6\,\mathrm{mm} among small values could be a storm or a recording error; its size alone cannot distinguish them.

Possible outlier

A possible outlier is an observation unusually far from the other values or from a relationship's main pattern, or one that meets a stated numerical rule. It can be a genuine extreme observation. A missing entry is not an unusually large or small numerical value, so it is not an outlier merely because it needs attention.

Any numerical identification rule needed in an examination is supplied. Read its multiplier, whether it is one-sided or two-sided, and whether its endpoints are included. Do not silently substitute a favourite rule. The conclusions here are informal interpretations, rather than a new significance test.

Key idea

Apply the supplied rule, then investigate in context. “Possible outlier” and “incorrect observation” are different conclusions.

Decide what the mathematics can establish on its own.

Using IQR fences

An IQR rule measures how far an observation lies beyond the central box, using quartiles rather than a mean that an extreme value can pull.

IQR and a supplied fence rule

IQR=Q3−Q1\mathrm{IQR}=Q_3-Q_1

In the formula booklet.

A question might define outliers as values more than 1.51.5 IQR below Q1Q_1 or above Q3Q_3. That particular choice of multiplier is a supplied rule, not a universal definition.

Fences when the multiplier is 1.5

Under that stated rule,

L=Q1−1.5 IQRU=Q3+1.5 IQRL=Q_1-1.5\,\mathrm{IQR} \qquad U=Q_3+1.5\,\mathrm{IQR}

Possible outliers satisfy x<Lx<L or x>Ux>U. If the wording changes, follow its inequalities and multiplier.

Not in the formula booklet: learn it.

The fences are comparison thresholds. They are not normally the ends of a box plot's whiskers.

[DIAGRAM: asset_name: Quartile fences and raw-data whiskers; asset_slug: p3_2_4_recognise_and_interpret_possible_outliers_in_data_sets__diagram_01; caption: The upper fence is 48, but the upper whisker ends at the largest non-outlier, 35; description: Horizontal box plot has lower whisker 12, Q1 18, median 23, Q3 30, upper whisker 35 and separately plotted outlier 51. Dashed fences at 0 and 48 are labelled above; the IQR bracket spans 18 to 30 above the plot.]
The upper fence is 48, but the upper whisker ends at the largest non-outlier, 35

Worked example 1: testing a high value

Question. Delivery times have Q1=18Q_1=18 and Q3=30Q_3=30 minutes. The rule flags values more than 1.51.5 IQR below Q1Q_1 or above Q3Q_3. Determine whether 5151 minutes is an outlier.

Plan. Calculate the IQR, then the relevant upper fence and compare.

IQR=30−18=12width of the middle halfU=30+1.5(12)=48measure above Q351>48compare with the fence\begin{aligned} \mathrm{IQR}&=30-18=12 && \text{width of the middle half}\\ U&=30+1.5(12)=48 && \text{measure above }Q_3\\ 51&>48 && \text{compare with the fence} \end{aligned}

Why this step? The distance 1.5(12)=181.5(12)=18 starts at Q3Q_3; it is not itself the upper fence.

Answer. 5151 minutes is an outlier under the stated rule. It may still be a genuine delayed delivery.

Check. The lower fence is 18−18=018-18=0. The numerical interval between fences is [0,48][0,48], with 5151 outside.

Change only the tested value, from 5151 to the boundary 4848. The strict wording now matters.

Worked example 2: a value on the fence

Question. With the same quartiles and rule, determine whether 4848 minutes is an outlier.

Plan. Use the same fence and test the strict inequality.

U=4848>48 is falseU=48 \qquad 48>48\text{ is false}

Why this step? “More than” excludes equality; an observation exactly 1.51.5 IQR above Q3Q_3 is not more than that distance away.

Answer. 4848 minutes is not an outlier under this rule.

Check. Its distance above Q3Q_3 is 48−30=18=1.5(12)48-30=18=1.5(12), exactly the threshold.

In the diagram the largest non-outlier observation is 3535, so the upper whisker ends there, even though the fence is 4848. A fence need not be a recorded value. If a plot uses a different stated whisker convention, read that convention explicitly.

Watch out

Wrong: compare 5151 with 1.5 IQR=181.5\,\mathrm{IQR}=18. Right: compare it with Q3+1.5 IQR=48Q_3+1.5\,\mathrm{IQR}=48. The quartile establishes where the extra distance starts.

Use fresh quartiles and keep the strict comparison visible.

Using mean and standard deviation

A second kind of rule measures distance from the mean in standard deviations. Its multiplier is supplied too. A lower-only rule is not the same as a two-sided rule.

A supplied three-standard-deviation rule

If a question defines possible outliers as more than three SDs from mean xˉ\bar x, with SD s≥0s\ge0, use

x<xˉ−3sorx>xˉ+3sx<\bar x-3s \quad\text{or}\quad x>\bar x+3s

For a rule referring only to values below the mean, use only the lower inequality. This is the stated example rule, rather than a compulsory rule for every dataset.

Not in the formula booklet: learn it.

Means and SDs can themselves change when an extreme observation is added. That is another reason to treat a flag as a prompt to investigate rather than proof of an error.

Worked example 3: a two-sided rule

Question. Rod lengths have mean 52.4 cm52.4\,\mathrm{cm} and SD 3.1 cm3.1\,\mathrm{cm}. A possible outlier is more than three SDs from the mean. Test 62.0 cm62.0\,\mathrm{cm}.

Plan. Find both limits and locate the observation relative to them.

L=52.4−3(3.1)=43.1lower limitU=52.4+3(3.1)=61.7upper limit62.0>61.7outside the upper limit\begin{aligned} L&=52.4-3(3.1)=43.1 && \text{lower limit}\\ U&=52.4+3(3.1)=61.7 && \text{upper limit}\\ 62.0&>61.7 && \text{outside the upper limit} \end{aligned}

Why this step? A two-sided rule can flag either unusually short or unusually long rods.

Answer. The 62.0 cm62.0\,\mathrm{cm} rod is a possible outlier.

Check. Its distance from the mean is 9.6 cm9.6\,\mathrm{cm}, greater than 3s=9.3 cm3s=9.3\,\mathrm{cm}.

Change only the rule's direction. The same long rod will not meet a rule concerned only with unusually short rods.

Worked example 4: a one-sided rule

Question. With the same mean and SD, a new rule flags only lengths more than three SDs below the mean. Test 62.0 cm62.0\,\mathrm{cm} again.

Plan. Calculate the lower limit and apply only the lower inequality.

L=52.4−3(3.1)=43.162.0<43.1 is falseL=52.4-3(3.1)=43.1 \qquad 62.0<43.1\text{ is false}

Why this step? The rule's purpose determines which side counts; computing an upper limit cannot override the word “below”.

Answer. 62.0 cm62.0\,\mathrm{cm} is not flagged by this lower-only rule.

Check. The observation is above the mean, so it cannot be more than three SDs below it.

A one-sided rule can be appropriate when investigating unusually low measurements, but the question supplies that choice. Keep the threshold's units when interpreting it.

Watch out

Wrong: “Anything more than three SDs above or below is an outlier” when the question says “below”. Right: translate the wording into its particular inequality before substituting.

Test a new observation against a lower-only rule.

Continue this lesson.

You’re previewing NovaLearn without an account. Create a free account to read the rest.

Create free account