02 Section A, Key question 1 - Evaluating research quality (part 2)

Validity and what a measure means

Validity concerns how accurately research captures the social phenomenon it claims to investigate.

A concept such as financial insecurity is an idea, rather than something a researcher can simply count. Operationalisation means turning that idea into something that can be observed or recorded: perhaps income, uncertainty about next month's work, or an account of worrying about bills. Choosing the indicator shapes what the research will find. Before praising a measure, ask whether it fits the particular question.

Consider an original teaching example. A researcher wants to understand financial insecurity and asks only, “What was your household's income last month?” Two households give the same income. One has regular earnings and savings; the other has unpredictable hours, debt and no money available for an emergency. The income answer may accurately record last month's money, yet it does not capture the whole experience of insecurity. The problem is the inference from income to insecurity, rather than proof that income is always an invalid measure.

This distinction appears in real statistical practice. In its November 2019 well-being publication, the Office for National Statistics treated disposable income and wealth alongside people's perceptions of their financial situation. These are related aspects of economic well-being, rather than interchangeable measures. ONS publication.

Validity can also be weakened during data collection. A person may conceal debt because they feel embarrassed, or interpret “income” as their wages while another includes benefits. A researcher who assumes that both answers have the same meaning could draw an inaccurate comparison. Clear definitions and confidential conditions can help, but their effectiveness must be considered in the actual setting.

An interview can reveal meanings through follow-up questions, yet an interview is not automatically valid. The interviewer might steer the conversation, the participant might present an acceptable image, or the researcher might interpret a remark without its context. Checking an interpretation against the fuller account or other relevant evidence can strengthen it. Evaluation therefore explains the link between the design and the particular claim, rather than awarding validity to every qualitative method.

Reliability under comparable conditions

Reliability concerns the consistency of the research procedure and its measurements under comparable conditions.

In an original teaching example, two interviewers ask about monthly income. One asks for wages before tax; the other asks for all household income after tax. Their answers cannot be compared consistently because the procedure has changed. A common question, the same reference period, agreed definitions and interviewer training would reduce this source of inconsistency. Standardisation means applying such common procedures.

Consistency also matters when researchers analyse material. Suppose two researchers read the same interview accounts and classify references to insecure work. Agreed coding rules can help them distinguish uncertainty about hours from dissatisfaction with pay. Coding means assigning categories to material for analysis. Comparing their decisions may reveal ambiguous rules and improve the consistency of the analysis. This does not require treating every account as identical or removing its context.

Repeated findings need careful interpretation. If the same household's income falls after someone loses a job, a lower answer next year is a real change, not necessarily unreliable measurement. Reliability asks whether the same procedure measures comparable circumstances consistently; it does not require a changing society to produce unchanged results. Differences can also arise by chance when two different samples are drawn.

Reliability and validity answer different questions. In another original example, every respondent is consistently asked about wages before tax, but the researcher labels the result “money available to spend after tax”. The question may be reliable because it is applied consistently. Its interpretation is invalid because the recorded measure does not match the claimed one. Consistently repeating an error does not make it accurate.

A highly structured questionnaire can make comparisons easier, but fixed categories may impose meanings that participants do not share. A flexible interview can clarify an unexpected meaning, but different follow-up questions make exact replication harder. Neither method wins automatically. A researcher must justify how much consistency the question needs and how the procedure preserves the relevant meaning.

Some qualitative researchers emphasise dependability, judging the trustworthiness of the process through checks such as comparing evidence from different sources and checking interpretations with participants. These support scrutiny without requiring identical conversations. Open University methodological guidance.

Representativeness and who is included

Representativeness concerns how well the people or cases studied reflect the relevant characteristics of the population the researcher wants to describe.

The target population is the group to which the research question refers, such as all employees in a district. The sample is the smaller group selected for study. A sample's size matters, but representativeness depends on its composition and how it was obtained. A large group of volunteers is not necessarily a good reflection of the population.

This original illustrative dataset shows one relevant difference:

Temporary workers are under-represented in the replies. If insecurity is more common among temporary workers, the responses could understate insecurity in the workforce. That conclusion is conditional: the table shows an imbalance in contract status, not the actual level of insecurity. Even matching these proportions would not prove that the respondents match the population in every relevant respect.

Two different problems can create such imbalances. Coverage error arises when the way people are reached leaves some of the target population out. For example, a questionnaire sent only to employees listed on large employers' websites misses workers at employers without such lists. Non-response bias arises when people selected do not reply and differ from those who do in ways relevant to the question. For example, workers worried that an employer might see their answers may be less willing to report insecure conditions. A low response rate raises a concern, but bias depends on who is missing and how their absence affects the finding.

Do not confuse non-response bias with response bias: distortion in the answers people actually give, such as concealing debt. The former can change the sample's composition; the latter can weaken the validity of the recorded answers. A study can suffer from both.

Real researchers investigate these problems. Lynn, Cabrera-Álvarez and Clarke (2023) analysed the sample composition of Understanding Society, the UK Household Longitudinal Study. They compared initial respondent profiles with Census-based estimates and examined changes as people stopped participating. The profiles were broadly similar initially, but some groups were under-represented. Adjusted weights reduced many later imbalances. Their comparisons also had limits because population coverage, question wording and the timing of data collection differed. Original research.