Use of data in statistics - Large Data Set weather data

Why the Large Data Set matters

Was Heathrow warmer in 2015 than in 1987, or does the answer depend on the month you choose? In the official data, the daily mean temperatures on 1 June were $15.4\,^{\circ}\mathrm C$ in 1987 and $12.05\,^{\circ}\mathrm C$ in 2015. That one day favours 1987. Before making a claim about a month or a year, we need a better question and more of the data.

Pearson supplies this weather dataset for the Statistics section of Paper 3. An examination question may give an extract, a diagram or summary statistics and assume that you already understand the weather context. Familiarity lets you spend your time interpreting the evidence rather than decoding a heading.

You do not need a copy of the spreadsheet in the examination and are not required to memorise detailed raw values. During the course, however, explore the actual spreadsheet with a calculator, spreadsheet or statistical software. Select data, clean it, plot it and check the summaries: reading a list of headings is only the start.

For example, a printed rainfall extract still needs a reader who recognises the code tr and knows that the entries are daily totals in millimetres.

Places, periods and the population represented

The data are Met Office weather records for eight stations, covering May to October in 1987 and the same months in 2015.

Here is a useful surprise: the workbook contains $2944$ station-day rows, since $8\times2\times184=2944$. Yet it contains no November observations at all. A large row count cannot repair a gap in the calendar.

Specify the variable, place and time. “Weather at Leeming” leaves too much open. If the target is all May-to-October days at Leeming in 2015, the workbook covers that period. If the target is all days in 2015, November to April are absent. If the target is long-term climate, two selected years are a further limitation.

Using Leeming's May-to-October temperatures to estimate its annual mean is likely to overestimate it because the colder winter months are omitted. Location changes interpretation too: July is a winter month in Perth, in the southern hemisphere, and a summer month at the UK stations. Coastal setting, latitude and urban surroundings may help explain patterns, but they do not replace evidence.

The same observations can be the whole population for one narrow question and a sample from a broader population for another.

Weather variables and units

A daily rainfall total of $2.4\,\mathrm{mm}$ is a depth of precipitation, not the number of hours during which it rained. Read the heading before touching the calculator.

Daily totals and daily means also use particular recording periods. The Information sheet defines daily rainfall over the $24$ hours starting at 0900 GMT on the date of entry, whereas daily mean windspeed uses the $24$ hours starting at 0000 GMT. A date does not guarantee that every variable covers exactly the same interval. Temperature is the mean of hourly readings over 0900–0900 GMT; it is not defined here as the mean of the daily maximum and minimum.

A mean or standard deviation of windspeed is measured in knots, just as the original observations are. A correlation coefficient has no units. Later, a regression gradient for rainfall against sunshine has units $\mathrm{mm}$ per hour of sunshine: dividing one variable by another changes the units. These dataset conventions are not supplied as a list in the formula booklet; learn them through use.

Units can expose a wrong interpretation even when the arithmetic looks tidy.

Cleaning codes and unusual values

An empty measurement and a tiny measurement are different things. Cleaning means recording defensible decisions before calculation, not changing inconvenient observations until a pattern appears.

Follow any cleaning convention given in a question. Otherwise, state your treatment. Omitting a missing value reduces the usable count for calculations involving that variable. For a numerical rainfall calculation, a stated approximation such as $0.025\,\mathrm{mm}$ for each trace is defensible; it gives an estimate, not the exact unrecorded rainfall. Rounding a trace to zero can also be a stated approximation, but calling a trace exactly zero is false.

Change one feature: the unavailable reading is now an observed zero. This changes the denominator even though it does not add to the total.

An extreme but physically possible reading stays unless there is evidence of an error. Check the station, units and neighbouring records; a wet day is not a typing mistake simply because it raises the mean.

Keep enough detail in your cleaning notes for someone else to reproduce the calculation.

Sampling and fair comparison

Comparing May 1987 with September 2015 changes two things at once: the year and the season. A difference cannot then be attributed to the year alone.

First define the target, then choose eligible days. A simple random sample gives every possible sample of the chosen size the same chance of selection. Select without replacement from a complete list of eligible dates. A block of consecutive dates is not a simple random sample: weather systems can persist, so neighbouring observations may also be related. Later sampling lessons develop the selection methods in detail.

Change one feature: both original samples now come from May. Seasonal imbalance between the two samples disappears, but the target still covers six months.