1.3 - Recording data and numerical precision

Designing raw data tables

Raw data are the original recorded observations or measurements before summaries such as averages are calculated. A raw data table should retain individual results so the researcher can check, analyse and trace them without inventing missing information.

Use an informative title, participant or observation codes, condition headings, and units or scoring scales. Decide the layout before collecting data. In a repeated measures word-recall task, each row can retain the two scores belonging to the same person. The table below contains original illustrative data; each score is the number of correctly recalled words from a list of 20.

Words correctly recalled in two sound conditions

This layout preserves pairing. For independent groups, use separate columns or a condition column that makes clear that different people supplied the scores; do not imply that two entries are a pair just because they share a row. A time measurement needs units such as seconds in its heading, while a rating needs its defined scale. An observation record may need session duration and category definitions as well as event counts, so its meaning can be recovered later.

Codes help protect confidentiality; identifying details, if needed and permitted, should be kept separately and securely. A numerical table is not suitable for replacing the full wording of an open interview response. Preserve qualitative raw material in an appropriate transcript or record before any coding.

Recording zero, missing data and precision

Recording rules should distinguish a measured value from no measurement. In the recall table, P03's zero is a valid result: no words were recalled in silence. P04's missing speech score means the measurement was not obtained. Replacing it with zero would falsely state poor recall and distort later calculations. Leave a clearly labelled missing entry with its reason where known; do not guess a score to complete the table.

Record data promptly using the agreed scoring rule and units. Check values against the possible range: 24 correct words is impossible for a 20-word list unless the task or heading was misrecorded. Check that a time in milliseconds has not been entered into a seconds column. If correcting a transcription error, retain a traceable correction rather than silently changing data to fit a hypothesis. Recheck the original record when available; an implausible value is a reason to investigate, not permission to delete it.

Preserve the resolution of the measurement. Counts of correct words are whole numbers; a timing device reading to the nearest hundredth of a second may produce values such as $1.27\,\mathrm{s}$. Writing $1.270000\,\mathrm{s}$ does not make the measurement more accurate. Device resolution is not the only source of uncertainty: a person's delay in pressing a stopwatch can matter even if its display has many digits. Keep unrounded calculated values during working and round the final reported answer as requested. Do not repeatedly round intermediate values unless a supplied procedure explicitly asks for it.

Decimal and standard form

Decimal form writes a number using place value; standard form expresses it compactly using a power of ten. They represent the same value, not different measurements.

For a large number, moving the decimal point left to obtain a coefficient between 1 and 10 produces a positive exponent. Thus $48\,200=4.82\times10^4$: multiplying 4.82 by 10,000 recovers 48,200. For a small positive number, moving the decimal point right produces a negative exponent: $0.0036=3.6\times10^{-3}$, because $10^{-3}=0.001$.

To return to decimal form, a positive exponent multiplies by that power of ten and a negative exponent divides by its positive counterpart. For example, $2.45\times10^{-2}=0.0245$, while $2.45\times10^2=245$. Check the size as well as counting decimal places: a negative exponent should make a positive coefficient smaller. $36\times10^{-4}$ equals 0.0036 but is not standard form because its coefficient is not below 10.

Changing the numerical representation does not change units. $3.6\times10^{-3}\,\mathrm{s}$ is 0.0036 seconds. Converting seconds to milliseconds is a separate unit conversion: multiply the numerical number of seconds by 1,000. That same time is 3.6 milliseconds.

Significant figures and rounding

Significant figures count meaningful digits from the first non-zero digit, whereas decimal places count positions after the decimal point. These are different instructions.

Leading zeros locate the decimal point and do not count as significant figures. Zeros between significant digits do count, and trailing zeros after a decimal point can indicate reported precision. For example, 0.0048 has two significant figures; 10.2 has three; 0.0100 has three because the two final zeros follow the first non-zero digit. A whole number such as 500 can leave precision ambiguous; $5.00\times10^2$ makes three significant figures explicit.

To round, retain the requested number of significant digits, then inspect the next digit. If it is 5 or more, increase the last retained digit; otherwise leave it. Remove later digits while preserving place value. For example, 23.746 to three significant figures is 23.7, because the next digit is 4. To two decimal places it is 23.75, because the next digit after the hundredths digit is 6. Similarly, 0.004876 to two significant figures is 0.0049; rounding it to two decimal places would give 0.00 and obscure its size.

Use the precision specified by the question or procedure, and avoid reporting more measurement precision than the record supports. Exact counts do not become uncertain just because they contain few written digits: 12 people is an exact count in that sample. Calculations can still require a stated number of significant figures. Keep enough digits in intermediate working to avoid introducing rounding error, then make the final precision visible. For example, writing 2.0 rather than 2 can communicate two significant figures when requested.

Estimation and order of magnitude

An estimate deliberately simplifies values to judge likely size; it is not a replacement for the original data. It can reveal a misplaced decimal point or implausible calculator result.

Suppose 198 participants together obtain 4,012 points on a test scored out of 40. To estimate the mean score, round the total to 4,000 and the number of people to 200, then divide:

The exact division gives about 20.2626, or 20.3 to three significant figures. A claimed mean of 202.6 is about ten times too large and exceeds the maximum possible individual score. The estimate detects the scale error without pretending that the exact answer is 20. Estimating the spread of scores can work similarly: if the smallest time is 9.8 seconds and the largest is 30.4 seconds, their spread is roughly $30-10=20$ seconds; exact subtraction gives 20.6 seconds.

Order of magnitude concerns the power-of-ten scale of a quantity. Write rounded values in standard form to compare sizes. A total near $4\times10^3$ divided by a sample near $2\times10^2$ gives a typical score near $2\times10^1$, on the scale of tens. A duration of 0.02 seconds is $2\times10^{-2}$ seconds; 20 seconds is $2\times10^1$ seconds and is 1,000 times larger, a difference of three powers of ten. Compare quantities in the same units before drawing such a conclusion.

Choose rounding that keeps an estimate useful. Approximating 19.9 by 20 is sensible for a size check, but rounding every small measured value to zero is not. State that the answer is approximate and retain the actual values for accurate analysis. An estimate can check a result's plausibility; it cannot establish a psychological effect or statistical significance.