1.3 - Bias, representativeness and generalisability

Sample and target population

A sample is representative when it reflects relevant characteristics of the target population. Relevant characteristics depend on the research question: age may matter for a memory study, while access to public transport may matter for a study of commuting stress.

Suppose a council wants to understand commuting stress among all local workers. An online survey advertised only in a cyclists' group may overrepresent motivated cyclists and exclude workers without internet access. Its sample could describe those respondents accurately but poorly represent all local workers. A high response count does not correct that selection problem.

Use a suitable sampling frame and recruitment that reaches relevant groups. A random sample gives listed members a chance of selection, but an incomplete frame and nonresponse can still introduce bias. A stratified approach can represent chosen proportions, but its value depends on choosing relevant strata and recruiting appropriately within them. Neither method guarantees that every unmeasured characteristic is representative.

Generalisation also depends on the task and setting. A more representative sample does not automatically make an artificial stress task applicable to real commuting. State a bounded conclusion, then test transfer rather than claiming that “people” universally behave like the sample. Policymakers who overlook recruitment bias might fund services that meet only one group's needs.

Demand characteristics and social desirability

Participants may change their behaviour because of what they infer about the research or how they wish to be seen. These are different possible sources of response bias.

Demand characteristics are cues that help participants infer the research aim or expected response. A leaflet praising a concentration app before its test may imply that better performance is expected. Participants might try harder to help, deliberately resist the expected result, or change their answers. The direction is not necessarily towards the hypothesis. Performance then partly reflects reaction to the study rather than the effect of the app itself.

Social desirability is responding in a way that presents oneself favourably according to perceived social expectations. A worker interviewed by a manager may underreport taking long breaks to appear conscientious. This can occur without guessing a hypothesis. A person could also honestly misunderstand a question; every inaccurate self-report is not deliberate impression management.

Neutral wording and standardised instructions can reduce cues. Privacy, reassuring confidentiality and separating researchers from authority figures can reduce pressure to present a favourable image. Behavioural indicators or carefully phrased indirect questions may provide complementary evidence, but can bring their own validity and ethical problems. Concealing a precise hypothesis never removes the need for ethical consent and review.

Researcher bias and researcher effects

Researcher or observer bias occurs when expectations influence recording, interpretation or reporting. A researcher expecting a training programme to increase cooperation might classify an ambiguous gesture as helpful for trained participants but ignore the same gesture for controls. The observed data become selectively coded even if participants' behaviour has not changed.

Researcher or observer effects occur when the researcher's presence or behaviour influences participants' actual behaviour. An observer standing visibly beside a recycling bin might make people recycle more. A researcher giving warmer encouragement to one condition might increase effort. The mechanism differs from selective recording, although both can occur together.

Single-blind or double-blind procedures can limit knowledge of condition or expectations for relevant people; state exactly who is unaware and of what. Blind independent coders reduce expectancy in scoring. Independent raters and agreement checks help reveal inconsistency, but two raters sharing the same expectations can agree on biased scores. Reliability therefore does not guarantee absence of bias.

Reasoned improvements and limits

A good evaluation identifies a particular source of distortion and explains how a design choice addresses it. “Use a bigger sample” does not repair leading questions; “make it double-blind” does not ensure a representative sample.

Consider a transport survey with broad recruitment across neighbourhoods, but interviews conducted by uniformed staff from the transport provider. Broad recruitment is a strength for representing different local users. The interviewer may nevertheless make users reluctant to criticise the service, underestimating dissatisfaction. An independent confidential survey may reduce this response pressure, but poorer access or response rates could create a different sample bias. Weigh the actual consequences rather than assigning automatic labels of good or bad.

Report the target population, recruitment, response rate and safeguards transparently. Interpret findings within those boundaries and compare them with evidence collected using other methods. Consistent replication across different samples and procedures can make a wider claim more credible; repeating the same bias simply reproduces a biased pattern.