9.1.12, 2.2.9, 4.2.4, 6.2.3 - Reliability, validity and credibility

Reliability and replicability

Reliability concerns consistency of measurement, whereas replicability concerns whether a study can be repeated to examine its findings.

Test–retest reliability compares the same measure taken by the same people at different times. If an attention score varies greatly while the underlying ability and testing conditions should be stable, it may be an unreliable measure. However, low agreement can also reflect real change. A measure of today's mood should not necessarily remain identical weeks later, so interpret consistency against the intended construct and time period.

Inter-rater reliability concerns agreement between observers or coders. “Aggressive behaviour” is too vague if one observer includes every raised voice and another includes only threats. Clear operational definitions, a coding manual, training and a pilot with independently scored examples improve consistency. Discussion after scoring is useful for refining the rules but is not evidence that the initial independent ratings agreed.

Internal consistency asks whether items intended to measure the same construct work coherently. A split-half check compares two comparable halves of a questionnaire. Consistency is not a reason to remove every different-looking item: a complex construct may have several relevant dimensions. Nor does repeatedly asking the same biased question establish accuracy.

Replicability requires enough method detail for other researchers to repeat a study: materials, recruitment, operational definitions and procedures should be explicit. Repeating a study with a new sample helps test how robust a finding is. An exact repeat tests a narrower claim than a different method aimed at the same idea. A non-replication may reflect chance, context or a procedural difference; it should prompt investigation rather than an automatic accusation of fraud. A standardised procedure can be consistently repeated yet consistently measure the wrong thing.

Internal validity and causal claims

Internal validity asks whether the evidence supports the claimed relationship within the study rather than an alternative explanation.

In an experiment, this often means whether the manipulated independent variable, rather than a confounding variable, accounts for differences in the dependent variable. Suppose one memory group learns words in silence in the morning and another learns with music late at night. Music condition and time of day vary together. If recall differs, tiredness or time may explain it, so attributing the difference to music has weak internal validity.

Testing both conditions at comparable times removes that particular confound. Random allocation reduces the likelihood that pre-existing participant differences systematically favour a condition. A repeated-measures design controls stable individual differences but needs counterbalancing or equivalent materials to address order and practice effects. The best improvement depends on the identified threat; “use a larger sample” does not separate music from time of day.

Demand characteristics occur when participants infer the study's purpose and adjust their behaviour. Researcher expectancy can influence instructions, interaction or scoring. Standardised instructions, neutral information and blinded scoring can reduce those influences. Blinding is not always possible, and any concealment must be ethically justified. A realistic setting can still contain confounds; a laboratory can improve control without making every inference valid.

In a correlation, good measurement and consistent procedures strengthen evidence that an association is present, but no manipulation isolates a cause. Using the phrase “internally valid” does not transform the association into an experiment. Link each claim to the actual design and consider reverse direction and third variables.

What a measure means and predicts

A measure must capture the intended construct before researchers can interpret its scores confidently.

If a supposed “memory test” mainly requires advanced vocabulary, language knowledge may affect the result. Reliable vocabulary differences can produce stable scores without those scores accurately reflecting the memory ability of interest. Construct validity concerns whether the measurement meaningfully represents the intended psychological concept. Clarifying the construct, using appropriate tasks and comparing relevant evidence helps establish it; one convenient score is not a complete validation.