03 · What You Need to Know
Different Threats Distort Research in Different Ways
A useful starting point is to distinguish systematic threats from ordinary statistical uncertainty.
Sampling variability means that estimates from a sample will generally differ somewhat from the population quantity even when the study is otherwise well designed. Increasing sample size can often improve precision.
Bias is different. Bias refers to systematic deviation that can move an estimate away from the quantity the researcher intends to estimate. A larger sample does not necessarily remove it. Indeed, a very large biased study may provide a highly precise estimate of the wrong quantity.
Risk-of-bias frameworks therefore focus on features of study design and conduct that could systematically distort results. Cochrane's tools, for example, separately examine domains involving randomization or confounding, participant selection, deviations from intended interventions, missing outcome data, outcome measurement, and selection of reported results.
Bias Is Not One Single Problem
The word “bias” is sometimes used loosely for anything that could go wrong in research. Methodologically, it is more useful to identify the mechanism producing the distortion.
Selection bias concerns systematic problems arising from how participants enter, remain in, or are included in an analysis. Information bias concerns systematic error in information about participants, exposures, outcomes, or other variables. Measurement bias can arise when the way a variable is assessed systematically favors or disadvantages particular observations or groups.
These problems can occur differently across study designs. Cochrane's guidance for non-randomized intervention studies, for example, distinguishes confounding, selection bias, information bias, and reporting bias rather than treating them as one generic methodological weakness.
The distinctions among selection, information, and measurement bias are important because preventing one does not automatically prevent another.
Selection Can Distort the Comparison Before Analysis Begins
Who enters the study matters when selection is related to the exposure, intervention, outcome, prognosis, or other factors relevant to the intended inference.
Imagine comparing academic performance between students who voluntarily adopt an optional AI tutoring system and students who do not. The users may be more motivated, more technologically confident, or more academically engaged before using the system.
The observed performance difference could therefore reflect more than the tutoring system itself.
Selection problems can also emerge after recruitment. Differential loss to follow-up, exclusions made after observing participant characteristics, or analysis restricted to participants with complete data can change the comparability of groups.
The important issue is not simply whether some participants were excluded or lost. It is whether the selection process creates systematic differences relevant to the outcome or comparison being estimated.
Information and Measurement Problems Can Distort What Was Observed
Even if the right participants enter a study, the information collected about them may be systematically wrong.
Participants may remember previous exposures inaccurately. Interviewers may probe one group more intensively than another. Outcome assessors who know which intervention a participant received may interpret ambiguous results differently. A diagnostic method may have different sensitivity across groups.
Cochrane's risk-of-bias guidance explicitly considers whether outcome measurement is inappropriate, differs between intervention groups, or could be influenced by knowledge of intervention status.
Measurement quality therefore involves more than choosing an instrument described as valid. Researchers also need to consider how the measure is administered, by whom, under what conditions, and whether measurement procedures differ systematically across comparison groups.
Confounding Is Not Just Another Name for Bias
Confounding occurs when the relationship of interest becomes mixed with the influence of another factor associated with both the exposure or intervention and the outcome in a way relevant to the causal question.
Suppose students who use an optional study application obtain higher grades. Prior academic motivation could influence both the likelihood of using the application and subsequent grades. If so, some of the observed association between application use and grades may reflect motivation rather than the effect of the application itself.
Cochrane describes confounding in non-randomized intervention research in terms of common causes of intervention choice and outcome. Randomization is valuable for causal inference because successful random assignment prevents prognostic factors from systematically determining intervention assignment.
The detailed logic of how confounding distorts a relationship deserves separate treatment because identifying a confounder requires more than finding a variable correlated with the outcome.
Randomization Addresses Confounding but Does Not Eliminate Every Bias
Randomized studies are powerful because successful random assignment can protect treatment allocation from systematic influence by known and unknown prognostic factors.
That does not make randomized trials immune to bias.
Participants may deviate from assigned interventions. Outcome data may be missing. Outcome assessors may know treatment assignments and make judgments influenced by that knowledge. Researchers may choose among several analyses or outcome measurements after seeing which produces the most favorable result.
Cochrane's RoB 2 framework therefore evaluates several bias domains even after randomization has occurred, including deviations from intended interventions, missing outcome data, outcome measurement, and selection of the reported result.
Watch Out
“Randomized” should not be used as shorthand for “free from bias.” Randomization addresses a crucial source of confounding when implemented successfully, but the conduct, measurement, analysis, and reporting of a randomized study can still introduce important threats.
Missing Data Are Not Just a Sample-Size Problem
Missing data reduce the amount of information available, but their more serious consequence can be systematic distortion.
Suppose participants experiencing adverse effects are more likely to leave an intervention study. An analysis based only on those who remain may make the intervention appear more favorable than it really is.
The percentage of missing observations alone does not determine the degree of bias. Researchers need to consider why information is missing, whether missingness differs between groups, whether it relates to the true unobserved outcome, and whether the analytic approach appropriately addresses the missing-data process.
Risk-of-bias guidance accordingly evaluates not simply the amount of missing data but whether the result is likely to be biased because of why those data are missing.
Statistical Adjustment Is Not a Universal Repair Tool
Regression models, propensity scores, matching, weighting, stratification, and related techniques can be valuable for addressing measured confounding under appropriate assumptions.
They cannot automatically fix every threat to validity.
An unmeasured confounder cannot simply be adjusted away with variables that were never collected. A poorly measured confounder may leave residual confounding. Selection mechanisms created by conditioning on certain variables can themselves introduce bias. Measurement error in exposure or outcome variables may persist after statistical adjustment.
This is why statistical adjustment does not automatically eliminate confounding. Good causal analysis depends on substantive knowledge, study design, measurement quality, and defensible assumptions as well as statistical technique.
Selective Reporting Can Distort the Published Result
A study can be conducted competently and still become misleading at the reporting stage.
Suppose researchers measure an outcome at four time points, fit several plausible models, examine multiple subgroups, and then report only the analysis producing the strongest statistically significant result.
The reported estimate is no longer simply a neutral summary of the original analysis plan. Its availability has become dependent on what the data happened to show.
Cochrane distinguishes bias in selection of the reported result from broader non-reporting bias. The former occurs when a reported result is selected from multiple eligible measurements, analyses, or subgroups based on the findings.
Prespecification, transparent reporting of deviations, protocol registration where appropriate, and reporting results consistently with the research question can help readers judge this risk.
Overinterpretation Can Create an Invalid Claim From a Valid Estimate
Not every threat occurs before the statistical result is produced. Researchers can also make claims that exceed what a reasonably unbiased estimate supports.
An association may be interpreted causally despite a design that cannot adequately address alternative explanations. A statistically significant result may be presented as practically important without examining magnitude and uncertainty. Findings from one restricted population may be generalized far beyond the evidence.
The distinction between internal and external validity is relevant here. A study can support a credible inference under its own conditions without automatically supporting every extension of that inference to other populations and settings.
Threats Should Be Anticipated Before Data Collection
Many validity problems are easier to prevent than repair.
Researchers can improve participant-selection procedures, define eligibility criteria prospectively, measure important confounders, standardize outcome assessment, blind assessors when feasible, develop data-quality procedures, plan analyses before inspecting results, and collect information needed to understand missingness.
Once data collection is complete, some opportunities have disappeared. No statistical model can reconstruct an important confounder that was never measured accurately or determine an outcome that was assessed systematically differently between groups without additional information.
This is one reason a valid and defensible research design begins by anticipating threats to the intended inference rather than adding a limitations paragraph after the analysis is finished.
