1.3 - Validity and what findings mean

Measurement and causal claims

Validity concerns whether evidence supports the interpretation or conclusion being made. It is useful to ask separately whether a measure captures the intended characteristic, whether an experimental effect can be attributed to the independent variable, and whether findings apply beyond the study.

Internal validity concerns the credibility of the conclusion within the investigation. In an experiment claiming a causal effect, alternative explanations for the result must be ruled out. Suppose one group reads in silence in the morning while another reads with music after a tiring school day. Music and time of day vary together. A difference in recall cannot confidently be attributed to music because fatigue is a confounding variable.

Use equivalent materials, comparable instructions and appropriate controls; allocate participants appropriately and address order effects where relevant. A control should address the actual alternative explanation. Random allocation helps reduce participant differences between conditions, but does not correct a confound deliberately built into the timetable.

A measured outcome must also represent the intended variable. Counting recalled words gives evidence about recall of that list; it does not directly measure every aspect of intelligence. A causal claim can therefore be weakened by both confounds and an inappropriate operational measure.

Face and construct validity

Face validity and construct validity concern a measure's relationship to its intended psychological characteristic, but they require different evidence.

Face validity asks whether a measure appears, on inspection, to measure what it claims. An item such as “I feel nervous before speaking to a group” appears relevant to public-speaking anxiety. Asking participants or knowledgeable judges to inspect the items can identify obviously irrelevant wording. Appearance is a preliminary judgement; it is not proof that scores measure anxiety accurately.

Construct validity concerns how well a measure captures the theoretical psychological construct. Anxiety is not directly visible in the way height is. A researcher defines its relevant features, designs suitable items and looks for evidence that scores behave as the theory predicts. A public-speaking anxiety score should relate to other relevant anxiety indicators, while being distinguishable from reading ability or dislike of questionnaires.

A questionnaire containing only “Do you avoid public speaking?” may have face validity but confuse anxiety with lack of opportunity or simple preference. Adding items about anxious thoughts and physical feelings broadens the relevant evidence; checking their relation to independent indicators and rival constructs strengthens the case. No single appealing item or correlation establishes construct validity conclusively.

Concurrent and criterion evidence

Criterion validity checks a measure against a relevant independent standard or outcome. The criterion must itself be appropriate; agreement with a biased or irrelevant standard does not establish useful accuracy.

Concurrent validity is evidence from agreement with another relevant measure or criterion obtained at approximately the same time. The same participants could complete a new anxiety questionnaire and an established validated anxiety scale during the same period. A strong positive relationship supports the new measure's concurrent validity. This compares different measures; administering the same test twice checks test-retest reliability instead.

Criterion validity is the broader relationship between test scores and a suitable external criterion. For example, a selection test intended to predict performance might be compared with independently assessed later performance. That later comparison supplies predictive criterion evidence. Contemporary usage can place concurrent evidence within criterion validity; the labels therefore overlap rather than forming eight mutually exclusive boxes. When asked for the specific same-time comparison, use concurrent validity. APA measurement terminology.

The criterion must match the intended use. A driving-attention test compared only with school examination grades has a questionable criterion. Agreement with an established attention task, or relevant independently recorded driving performance under appropriate safeguards, would be more informative. A shared self-report response bias can also inflate agreement between questionnaires, so seek converging evidence using different methods.

Population and ecological validity

External validity concerns the extent to which findings generalise beyond the particular study conditions. It requires a specified target: “generalisable” is incomplete unless we say to whom, to which setting or to which task.

Population validity concerns generalisation from the sample to the target population. A study of a revision technique using only volunteers from one high-achieving class may not apply equally to every student in a college. Motivation, age and prior attainment could matter. Sampling a wider range of relevant students can improve the evidence, but a large sample from that same class does not remove its restricted composition.

Ecological validity concerns whether findings apply to behaviour in relevant everyday settings and tasks. Memorising unrelated words alone for two minutes may differ from learning a meaningful chapter with interruptions. A realistic task may improve transfer, but a field setting is not automatically ecologically valid: its task can still be artificial. Nor does a laboratory automatically lack ecological validity; a controlled simulation can reproduce important features of the target activity.

Control and generalisation can involve a trade-off. Keeping distractions identical helps isolate an effect, while everyday distractions vary. Researchers can first control a mechanism, then check it in realistic conditions and varied samples. The relevant conclusion is bounded by the evidence actually obtained.

Matching the check to the claim

Choose validity evidence by examining the claim rather than memorising an isolated label.

A council choosing a workplace-wellbeing questionnaire needs evidence about both the measure and its intended use. Strong agreement with a validated measure does not establish that an intervention caused improvement, nor that findings from one office generalise to every workplace. Reliable repeated scores are helpful but may consistently reflect an irrelevant construct. Validity is supported by an accumulating body of relevant evidence, not declared by one label.