1.3 - Reliability and consistent measurement

Consistency within and between occasions

Reliability concerns whether measurement produces consistent results under comparable conditions. A changing score is not automatically evidence of an unreliable test: the person's behaviour or psychological state may really have changed.

Internal reliability concerns consistency within a measure. If several questionnaire items are intended to measure the same characteristic, people's responses should fit together. Items about enjoyment of teamwork and preference for working together may belong together; an item about eyesight does not become a good teamwork item merely because it is easy to score.

External reliability concerns consistency when a measure or procedure is repeated on another occasion. A clearly described procedure makes repetition possible, but standardisation alone does not demonstrate that repeat results agree. We must collect and compare evidence. Repeating a study with a new sample can also show whether its pattern of findings is replicable; individual test-retest reliability instead follows the same people.

Reliability and validity ask different questions. A timer that always adds two seconds is consistent but gives inaccurate times. A questionnaire may consistently measure reading difficulty when it is meant to measure anxiety. Improving consistency is useful, but does not by itself show that the intended construct was measured.

Checking internal reliability with split-half

The split-half technique compares two equivalent sets of items from one administration of a test. For an eight-item questionnaire about confidence in public speaking, the researcher might compare each person's total for odd-numbered items with their total for even-numbered items. Both halves must contain comparable content and be scored in the same direction; negatively worded items need appropriate reverse scoring before totals are calculated.

Across the participants, similar rankings on the halves support split-half reliability. Researchers often calculate a correlation between the paired half scores. A strong positive relationship supports internal consistency; a weak relationship suggests the halves may not measure the same thing consistently. A correlation is more informative than comparing only the two group means: equal means can hide very different rankings of individuals.

The division matters. Putting all easy questions in one half and all difficult questions in the other creates a poor comparison. Nor is a high correlation proof that the questionnaire measures public-speaking confidence accurately: both halves might consistently reflect general optimism. There is no universal coefficient in this lesson that guarantees an acceptable test for every purpose.

Checking stability with test-retest

Test-retest reliability checks whether the same people obtain similar results when the same measure is administered again. Give a spatial-reasoning test to the same participants twice, keep instructions and testing conditions comparable, then compare their paired scores. A strong positive relationship supports stable relative performance over time.

Choosing the interval requires judgement. An immediate repeat may be inflated by remembered answers or practice. A very long interval allows development, training or other real changes. If the intended construct is a temporary mood, a difference after a major event may reflect valid sensitivity to change rather than poor reliability. Record the interval and relevant changes, rather than treating every discrepancy as error.

To improve consistency, pilot unclear questions, standardise timing and instructions, and use suitable equivalent conditions. Those controls reduce variation caused by the procedure. They should not prevent the test detecting real changes in the characteristic it was designed to measure.

Agreement between raters

Inter-rater reliability concerns agreement when different people independently record or score the same behaviour. Two observers might code the same video of a discussion, using an agreed definition of an interruption. They must code independently before comparing; agreeing on scores together cannot establish independent agreement.

Clear operational definitions help. “Dominant behaviour” is vague; “speaking before another speaker finishes and continuing for at least two seconds” is more reproducible. Train observers using practice examples, discuss ambiguous cases, revise the coding rules, and then check agreement on new material. Do not remove genuine disagreement simply to make the reported agreement look better.

For simple categorical coding, a transparent descriptive check is the proportion of decisions on which observers agree. If they agree on 18 of 24 independently coded intervals, agreement is $\frac{18}{24}\times100=75\%$. This does not adjust for agreement expected by chance; it is a limited descriptive index, not a universal reliability threshold. For scored ratings, correlation can show similar rankings even when one observer consistently awards higher scores. Check the actual scores as well as their relationship.

Choosing a reliability check

The appropriate check depends on the possible source of inconsistency.

A workplace may use a questionnaire to allocate training. Inconsistent items could classify the same worker differently depending on which questions are included; unstable scores could produce different decisions on different days. Establishing the relevant reliability helps prevent arbitrary decisions. Fair use also requires valid measurement and evidence that the tool suits the population and purpose.