02 Section A, Key question 2 - Interpreting research and informing policy (part 5)

Checking interpretations with respondents

Respondent validation asks people who took part in research to comment on the researcher's account of their experiences.

Checking an interview transcript can reveal that a word was recorded incorrectly. Checking a summary of the findings can reveal a deeper problem: the words were recorded accurately, but the researcher interpreted them differently from the speaker. This matters for validity, whether the research captures what it claims to capture. An accurate transcript alone does not guarantee an accurate interpretation.

Consider this original teaching scenario. A parent says, “I don't go to the school meetings.” A researcher interprets this as a lack of interest in the child's education. When shown the interpretation, the parent explains that the meetings take place during their paid shift and they cannot afford to lose the income. The researcher has evidence of an access barrier, rather than sufficient evidence of indifference. Revisiting the account can reduce researcher imposition: forcing the researcher's assumptions onto participants' meanings.

In their 2016 methodological paper, Birt and colleagues described returning interview material and developing findings to people diagnosed with melanoma, several months after semi-structured interviews. Participants could add to the developing interpretation, rather than merely approve a transcript. This is an illustration of a checking procedure, not proof that all member checking produces valid results. Birt et al., institutional author abstract.

There are practical and interpretive limits. Participants may not reply, may struggle with academic language, or may hesitate to challenge a researcher. Some may change their understanding after the original interview. A participant also knows their own experience, but may not know the wider pattern across the sample. Disagreement therefore needs investigation: it might reveal a mistake, a changed viewpoint or a defensible difference in interpretation. Agreement can improve confidence in an account; it does not establish objective truth, representativeness or a causal relationship.

Researchers should explain how they used the feedback and preserve a reasoned record of changes. They should protect confidentiality when sharing material, avoid identifying other participants, and make further participation voluntary. Respondent validation is not a scientific rule giving participants a veto over every conclusion; separate consent and withdrawal commitments still have to be honoured.

Following change over time

Following the same people over time can reveal changes that are hidden by a single survey or by comparing different samples.

A longitudinal study follows people or other study units over time, collecting evidence on more than one occasion. A panel study repeatedly collects data from the same individuals. A cohort is a group sharing a starting experience, such as birth in a particular period; researchers can follow members of a birth cohort through their lives. By contrast, a repeated cross-sectional study takes a new sample from a population on each occasion. It can show a population trend, but cannot by itself identify which individuals changed.

Here is an original illustrative panel of the same 100 people, interviewed one year apart, with no missing responses:

The employment proportion is 60% on both occasions. That does not mean nobody changed: ten people left employment and ten entered it. A pair of surveys of different people could show the same unchanged overall proportion, but would not establish those individual movements. Longitudinal evidence is useful for distinguishing persistent disadvantage from temporary spells and for studying the sequence of events.

A documented example is the 1970 British Cohort Study (BCS70), managed by UCL's Centre for Longitudinal Studies. It follows a birth cohort from a single week in 1970 in England, Scotland and Wales, gathering evidence on education, health, social development and economic circumstances. Its sweeps include birth in 1970, age 16 in 1986 and age 42 in 2012. Earlier circumstances can therefore be linked to later outcomes for cohort members. The design does not mean every original member responds at every sweep. UCL study description and sweep record.

Sullivan and Brown's 2015 analysis used 3,583 cohort members with vocabulary and mathematics test results at age 16, linking reading behaviour and earlier test scores. More frequent childhood reading was associated with greater progress between ages 10 and 16, especially in vocabulary, after accounting for measured background and prior attainment. Repeated measures let researchers examine progress, rather than simply compare high and low scores at one age. This was observational research, with missing-data adjustments: it strengthens the argument but cannot exclude every unmeasured influence or guarantee the adjustments' assumptions. Authors' paper, methods and findings.

Three limits matter. Attrition is the loss of participants over time, for example through refusal or loss of contact. If people with insecure housing are harder to trace, the remaining sample may understate their experiences. Researchers can examine missingness and make statistical adjustments, but cannot simply assume the missing people resemble those who remain. Sustained funding, tracing, trained staff and continuing consent also make long studies demanding.

Finally, a cohort effect arises when a generation's shared historical experiences help shape its outcomes. Members born in 1970 entered adulthood under different conditions from members born decades later. Changes within a cohort can also reflect ageing or events affecting society at the time. Following people establishes a sequence, but does not automatically separate these explanations or make one generation representative of every generation.

Reading numerical evidence carefully

Interpreting numerical data means identifying the measured pattern and judging how far the evidence supports an explanation.

Start with who was studied, when, how the concept was measured and the denominator behind a percentage. These original illustrative data come from a one-off survey of 200 students at one fictional college in one term. Students reported whether they had a quiet place to study and whether they had missed at least one school day that term:

More students with a quiet place missed a day in absolute numbers: 30 compared with 20. However, that group is three times larger. The proportion is 20 out of 50, or 40%, without a quiet place, compared with 30 out of 150, or 20%, with a quiet place. The first proportion is twice the second and the gap is 20 percentage points. A percentage-point gap subtracts percentages; it is different from a percentage increase.

The table supports description: 40% of one group reported an absence. It also supports an association: reported lack of quiet space occurs alongside a higher absence proportion in this sample. An explanation proposes a mechanism, such as housing insecurity disrupting school attendance. Causation is the stronger claim that changing one factor produces a change in the other. The proposed mechanism is an interpretation to investigate; the table has not measured housing insecurity or tested that mechanism.

This one-off survey cannot establish that providing a quiet place would reduce absence. Family income or health could influence both reported study space and attendance: such an alternative influence is a confounding factor. The survey also gives limited evidence about which event came first. Both measures are self-reported, and one college does not automatically represent all students. A conclusion should name the sample and the measure rather than turn “missed at least one day” into “frequent truant”.