1.9 - Evaluating health-risk studies

Choosing a study design

A health study must compare an exposure, such as smoking or physical inactivity, with a clearly defined outcome, such as a newly diagnosed case of cardiovascular disease (CVD). The design determines which comparisons are possible and which alternative explanations remain.

An observational study records people's existing exposures; researchers do not decide who smokes or who exercises. An intervention study changes something, such as offering an activity programme, and measures its effects. The question and ethical constraints help determine the design.

The designs are compared by what they measure and the weaknesses their methods can introduce.

For example, a cross-sectional survey could find that people with CVD walk less. CVD itself may make walking difficult, so this design cannot establish the direction of the relationship. A prospective cohort that records activity before CVD develops deals better with this timing problem, although it still needs to address other differences between the groups. Case–control and cohort designs are explained in the CDC's study-design guidance.

Researchers must not allocate people to smoke to test whether smoking causes harm. An ethically reviewed trial could instead compare offers of a safe, suitable activity programme with usual support, with informed consent and protection of health information. Its conclusion would concern the tested programme and participants. Randomisation strengthens evidence; it does not make every trial result automatically valid. The principle of random allocation is also explained by the National Cancer Institute.

Selecting a representative sample

Before choosing participants, define the target population: the people to whom the conclusion is intended to apply. A sample suitable for adults aged 40–69 in one town does not automatically represent children or adults in every country.

The sampling frame is the list or source from which participants are selected. Randomly choosing names from a complete frame gives each person a known chance of selection and reduces the researcher's selection preferences. It cannot include people missing from the frame. For example, random selection from a gym membership list still excludes non-members and may overrepresent active people.

Selection bias is a systematic distortion caused by who enters or remains in the study. Volunteers answering a health advert may differ in health or behaviour from people who do not respond. Even a well-selected invitation sample can become unrepresentative through non-response. Record how many were invited, who responded and whether response differs between important groups; offer accessible ways to participate rather than silently replacing non-responders with convenient volunteers.

In stratified random sampling, divide the population into relevant groups, then select randomly within each group. Proportional selection can preserve known population composition. It improves representation of those chosen characteristics; it does not guarantee that every unmeasured characteristic is represented. Sampling frames, stratification and non-response are part of the Office for National Statistics' sample-design guidance.

Worked sampling allocation

An original practice population contains 10,000 adults aged 20–79. Researchers want a proportional sample of 400.

For ages 60–79, the allocation is $2{,}500/10{,}000\times400=100$ people. The check is $160+140+100=400$. Researchers would select the allocated number randomly within each age group.

Suppose 400 volunteers instead include only 20 people aged 60–79: $20/400\times100=5\%$, compared with 25% of the population. The oldest group is underrepresented by 20 percentage points. Recruiting 4,000 volunteers with the same age percentages leaves this mismatch unchanged. A bigger sample can still be systematically biased.

Sample size and random variation

Different samples from the same population can give different results by chance, even when the selection method is sound. This sampling variation is especially influential when few people or few disease cases are observed.

A larger number of independent participants usually makes a population estimate less sensitive to random variation. In an original practice cohort, 3 new CVD cases among 20 participants gives $3/20\times100=15\%$. One additional case changes it to 20%, a change of 5 percentage points. In a cohort of 2,000, one additional case changes the percentage by only $1/2{,}000\times100=0.05$ percentage points. This illustrates the influence of an individual case; it is not a statistical significance test or a complete measure of uncertainty. The distinction between sampling and other errors is discussed in the ONS guide to survey uncertainty.

Judge the numbers in each comparison group and the number of outcome events, rather than only the total recruited. A study of 5,000 participants with just 15 in the exposure group has a small exposed group. A rare outcome may require a larger sample or longer follow-up to yield enough cases for a useful comparison. Small effects and highly variable measurements also require more information. There is no universal sample size that makes every health study adequate; size should be planned for the question, expected outcome frequency and useful difference to detect.

Repeatedly measuring the same 20 people can improve estimates of each person's usual measurement. It does not turn them into 200 independent people or broaden the population represented. Repeating the study in independently recruited groups gives a stronger check on whether the finding is consistent. Increasing numbers cannot repair a systematically biased sampling frame, a faulty instrument or an uncontrolled confounder.

Confounding and fair comparisons

Groups that differ in one suspected risk factor often differ in other ways too. A comparison needs to separate the factor being investigated from alternative explanations.

For example, a study of coffee consumption and CVD may find more smoking among frequent coffee drinkers. Smoking is related to coffee consumption in this sample and increases CVD risk, so some of the apparent coffee association could be due to smoking. Merely naming smoking is incomplete evaluation: explain both links and why they make attribution difficult.

Record likely confounders such as age, smoking and relevant prior health. Compare exposure groups within similar age or smoking categories, or use an appropriate analysis that adjusts for measured differences. Matching recruitment on age can make a suitable comparison easier, but matched data still need an appropriate comparison and analysis. Do not match on the very exposure whose relationship is being investigated. Restricting a study to one age range reduces age variation but also narrows who the findings represent. Measured adjustments cannot remove an important confounder that was not recorded or was measured badly. These issues are addressed in the CDC's guidance on interpreting epidemiological data.

Random sampling selects people from a population; it helps population representation. Random allocation puts recruited people into intervention groups; it reduces systematic differences between those groups, including differences not chosen by the researcher. They solve different problems. Allocation tends to balance characteristics, especially in larger groups, but does not guarantee exact balance. Observational groups are not randomised simply because participants were randomly sampled.

A well-designed observational comparison can strengthen evidence for a health-risk relationship. It still needs a cautious conclusion about association and consideration of remaining explanations. A causal conclusion is strengthened by other evidence, including timing, biologically plausible mechanisms and suitably controlled interventions where these are ethical.

Valid measurements and follow-up

A representative sample cannot rescue a measurement that fails to record the exposure or outcome that the question requires.

Internal validity concerns whether the comparison within the study is sound, including measurement and control of alternative explanations. External validity concerns whether the findings can be generalised to the intended population. Both matter, but a study may be strong in one respect and limited in the other.

Define exposures precisely. “Do you exercise?” may be interpreted differently by different people; frequency, duration and type of activity over a stated period are clearer. Use a suitable, checked method consistently in both groups. Self-reports can contain recall errors or socially desirable answers; a suitable activity monitor may reduce some reporting errors, although it also needs a defined recording period and may miss some activities. Repeated exposure measurements can detect changes during a long study.

Define the outcome using the same clinical criteria and comparable records for both groups. Measuring blood pressure is not the same as counting heart attacks. If the question concerns CVD cases, blood pressure alone is an incomplete outcome measure. Where feasible, assessors unaware of exposure group reduce the risk that expectations influence disease classification. Pilot questions, train collectors and check instruments against appropriate reference measurements. These data-collection safeguards are described in the CDC's collection guidance.

Repeatability means getting similar measurements using the same method under the same conditions. Accuracy means closeness to the true or accepted reference value. A blood-pressure instrument that always reads 6 mmHg too high could be repeatable but inaccurate. Repeats and a mean reduce random measurement variation; calibration addresses a systematic instrument error. Neither improvement substitutes for controlling confounding.

Follow-up must be long enough for the chosen outcome and comparable between groups. If people leave, record when and why. Loss to follow-up can reduce the effective sample and introduce selection bias if those lost differ in exposure or disease risk from those retained. Unequal losses are a reason to investigate bias, not proof of its direction. Results from the remaining people alone may give a misleading comparison.

Evaluating a complete health study

Evaluating a study means linking features of its design to the confidence and scope of its conclusion. A feature can be a strength for one purpose while leaving another problem unresolved.

Original practice study: walking and new CVD cases

Researchers invite 5,000 adults aged 40–69 from a town's general-practice registers. Of these, 1,200 respond and consent. All responders are free of diagnosed CVD at the start. A single questionnaire classifies 800 as frequent walkers and 400 as rare walkers. The researchers do not record age within the 40–69 range or smoking. For five years, clinical records are checked using the same CVD diagnosis criteria by assessors unaware of walking group. During follow-up, 40 frequent walkers and 120 rare walkers are lost. Only those retained are analysed. The report claims, “This proves that walking prevents CVD in people of all ages.”

This is an original NovaLearn teaching example, not a report of an actual study or Pearson's annual article.

The prospective design is a strength: recorded walking precedes new diagnoses. Excluding people with diagnosed CVD at baseline reduces the possibility that an existing diagnosis caused the recorded inactivity. Common clinical criteria and assessors unaware of group improve the fairness of outcome recording. These strengths support a useful comparison; they do not eliminate undiagnosed disease or other explanations.

Only $1{,}200/5{,}000\times100=24\%$ of invited adults enter the study. The responders may differ from non-responders. The study also includes only ages 40–69 in one town, so its sample does not justify a claim about all ages everywhere. Within its participants, unrecorded age and smoking could confound the comparison, while one questionnaire may misclassify walking or fail to capture later changes.

Check the participants actually retained

The rare-walking group loses a much larger proportion, even though the study began with 1,200 people. If participants leave for reasons related to health, the retained groups may no longer provide a fair comparison. The information given does not reveal whether those lost had higher or lower disease risk, so it cannot establish the direction of the bias. Disease counts and an appropriate analysis would be needed to judge the observed association itself.

A better design would seek and report wider participation, measure likely confounders and analyse comparable categories, define and reassess walking, and improve comparable follow-up while recording reasons for leaving. Each change addresses a stated problem. Increasing recruitment alone would not correct the missed confounders or reporting errors. A suitable ethically reviewed randomised activity-programme trial could add evidence about an intervention, but it would investigate the programme actually assigned.