4.2.3.1 - Reliability

What reliability means across methods

Reliability is about consistency. In research methods, the question is not "Did the study find something interesting?" but "Would this method produce the same or very similar results again if the thing being measured had not really changed?"

AQA places reliability "across all methods of investigation". That means the idea applies to experiments, observations, questionnaires, interviews, correlations, content analysis and case studies. The exact reliability problem changes with the method:

• In an experiment, an unreliable procedure might vary from participant to participant because instructions, timing or apparatus are inconsistent.
• In an observation, unreliable behavioural categories might mean two observers record different things from the same behaviour.
• In a questionnaire, unreliable items might produce different answers from the same person on two occasions even though their attitude has not changed.
• In an interview, unreliable questioning might mean one participant is prompted much more than another.
• In content analysis, unreliable coding might mean two coders classify the same magazine article differently.
• In a correlational study, each co-variable needs consistent measurement; a sleep questionnaire and an attention task could each be checked for stability where genuine change is not expected.
• A case study may concern a unique person or event that cannot be repeated exactly. Nevertheless, standardised interview schedules, consistent records and independent coding of the same evidence can still be checked for reliability.

Reliability is not the same as validity. A measure can be reliable but invalid. For example, a questionnaire might consistently measure social desirability rather than real anxiety. Reliability asks whether the measurement is steady; validity asks whether it is measuring what it claims to measure.

Why consistency matters

Psychological research often deals with hidden constructs such as stress, aggression, memory or depression. Researchers cannot usually measure these directly, so they use operationalised measures: scores, ratings, categories, reaction times or physiological readings. If those measures are inconsistent, random error increases and real patterns are harder to detect.

A reliable method supports replication because later researchers can repeat the procedure and compare findings. It also makes the results more trustworthy: if two observers, two time points or two similar administrations give very different results, the researcher should question whether the method is clear enough.

Spotting the reliability issue

Imagine a researcher observes "aggression" in a playground but only tells observers to tally "aggressive behaviour". One observer records shouting, pushing and angry facial expressions. Another records only physical contact. Their totals will probably differ because the category is vague. This is a reliability issue, especially inter-observer reliability.

Now imagine a questionnaire on exam stress. The same students complete it on Monday and again on Wednesday. If the scores are very different, the questionnaire may lack test-retest reliability, but unrecorded changes in stress must also be considered.

Consistency and accuracy

High reliability is necessary for strong research, but it is not enough on its own. A method can be consistently wrong. The most exam-useful judgement is: reliability is a foundation for valid measurement, but researchers still need to check that the operationalised measure captures the intended construct.

Measuring reliability with test-retest

Test-retest is a way of measuring whether a method is consistent over time. It is most useful when the construct should be fairly stable over the chosen time period, such as a personality trait, an attitude that is not expected to change quickly, or a questionnaire score for a relatively stable characteristic.

The basic procedure is simple:

1. Give the same participants the same test, questionnaire or measure.
2. Wait for an appropriate time interval.
3. Give the same participants the same measure again.
4. Compare the two sets of scores, often by calculating a correlation.
5. A strong positive correlation suggests good test-retest reliability. A coefficient around +0.80 or above is often used as an A-level rule of thumb, not a universal pass threshold; interpret it for the measure and purpose.

The time gap matters. If the second test is too soon, participants may remember their previous answers, which artificially increases reliability. If the second test is too late, the construct may genuinely change, which makes the measure look unreliable even if the measure itself is adequate.

AO2: applying test-retest

Leila creates a questionnaire measuring attitudes towards online learning. She gives it to 40 students in Week 1 and again in Week 3. There has been no new online-learning policy during that period. If students who scored high in Week 1 also tend to score high in Week 3, and students who scored low also tend to score low, the questionnaire has good test-retest reliability.

A correlation of +0.88 would indicate strong positive consistency in relative scores; +0.20 would indicate weak positive consistency. A negative correlation, where initially high scorers become low scorers, would not support test-retest reliability when both administrations use the same scoring direction. Neither result tells Leila whether the questionnaire measures attitudes rather than a different construct.

If the scores show no relationship, Leila should not simply blame the students. She should look at the questionnaire. Some items may be unclear, double-barrelled or too dependent on the participant's mood on that day.

AO3: strengths and limits

Test-retest is useful because it gives direct evidence that a measure is stable over time. It is especially suitable for questionnaires, interviews with standardised rating scales, psychometric tests and some physiological measures.

However, it is not suitable for every psychological measure. Mood, pain, stress and memory performance can genuinely change over short periods, so a low test-retest correlation may reflect real change rather than poor reliability. Repeating a task can also create practice effects, boredom or demand characteristics. Participants may work out the aim of the study and answer in a more deliberate way the second time.

A strong exam answer therefore says what is being repeated, who repeats it, when it happens, how the two sets of results are compared and why the time interval is appropriate.

Measuring reliability with inter-observer agreement

Inter-observer reliability is about consistency between people who record or judge the same material. It is most obvious in observations, but the same principle applies whenever researchers code behaviour, interview responses, written answers, media content or case notes.

To measure inter-observer reliability, researchers usually:

1. Define clear behavioural categories or coding rules.
2. Train two or more observers or coders using the same instructions.
3. Ask them to observe or code the same event, recording independently.
4. Compare their records using agreement, correlation or a reliability statistic.
5. Treat close agreement as evidence that the categories and instructions are reliable.

The observers must work independently. If one observer can see another observer's tally sheet, their agreement might come from conformity rather than genuine reliability.

AO2: applying inter-observer reliability

Nadia is studying helping behaviour in a classroom. She defines "helping" as "giving another student a requested object, explaining a task step, or asking if another student needs assistance". Two observers watch the same 20-minute lesson from different positions and independently tally each category. Similar tallies support consistency, but comparing each same event or interval is stronger evidence: equal totals could hide disagreement about which behaviours occurred.

If the tallies are very different, Nadia should check the behavioural categories. "Explaining a task step" might be clear; "being supportive" would be too vague unless it is broken down into observable behaviours.

Beyond observations

Inter-observer reliability also matters in content analysis. Suppose two coders classify posts as "pro-social", "anti-social" or "neutral". If they often disagree, the coding frame is unreliable. It matters in interviews too: if two researchers rate the same answer for "severity of anxiety" and give very different ratings, the rating scale needs clearer anchors.