03 · What You Need to Know
Adjustment Is a Method for Addressing Confounding, Not Proof That It Is Gone
Confounding occurs when the relationship of interest becomes mixed with the influence of other factors in a way that distorts the effect or association the researcher wants to estimate. In non-randomized research, exposure or intervention groups can differ systematically because assignment was influenced by participant characteristics, preferences, institutional practices, clinical decisions, or other factors related to the outcome.
Statistical adjustment attempts to make relevant comparisons more appropriate by accounting for measured differences. Common methods include stratification, multivariable regression, standardization, matching, propensity-score approaches, and weighting.
These techniques can be powerful. None creates randomization retrospectively.
What Does an Adjusted Estimate Actually Mean?
Suppose an unadjusted analysis shows that students using an AI tutoring platform score six points higher than non-users. After adjusting for prior achievement, age, program, and several other variables, the estimated difference is three points.
The adjusted estimate represents a comparison produced by a particular statistical model after accounting for the specified variables in the specified way. Whether that three-point difference can be interpreted causally depends on more than the fact that adjustment occurred.
The relevant confounding structure must have been identified adequately. Important variables must have been measured sufficiently well. The model or weighting strategy must represent the necessary relationships appropriately. Other biases must also be considered.
This is why understanding what makes a variable a confounder should come before deciding what to include in a regression model.
You Cannot Directly Adjust for a Confounder You Never Measured
The most obvious limitation of conventional covariate adjustment is also one of the most consequential: the analysis generally cannot directly control for information that is absent from the data.
Suppose academic motivation influences both voluntary use of an AI tutoring platform and later examination performance. If motivation was never measured, adding age, sex, academic program, prior grades, and device ownership to a regression model does not guarantee that motivation has somehow disappeared from the comparison.
Those variables might capture some information correlated with motivation, but that is an empirical and causal question. The mere presence of many covariates does not establish adequate control of the missing confounding domain.
Cochrane's guidance for non-randomized intervention studies explicitly recognizes residual confounding when an important confounding domain is not measured. It therefore treats confounding as a substantive risk-of-bias problem rather than something automatically resolved by multivariable analysis.
Measuring a Confounder Does Not Mean You Measured It Well Enough
A variable can appear in the dataset and still leave important confounding behind.
Imagine that socioeconomic circumstances influence both access to an educational technology and academic outcomes. The researcher attempts to control for socioeconomic circumstances using a single yes-or-no question about employment status.
Employment may capture some relevant information, but socioeconomic circumstances could also involve household resources, parental education, financial insecurity, housing conditions, access to technology, or other factors relevant to the causal question.
If the measured variable represents the confounding domain poorly, adjustment may be incomplete.
Cochrane identifies measurement error in a confounding domain as one source of residual confounding. The lesson is straightforward: putting a variable name into a statistical model does not guarantee that the underlying confounding process has been adequately represented.
Coarse Categories Can Leave Residual Confounding
Even a relevant variable can lose important information when categorized too crudely.
Suppose age is an important confounder, but researchers classify everyone simply as younger than 30 or 30 and older. Two participants aged 31 and 70 are then treated identically for adjustment purposes even if the outcome changes substantially across that age range.
The same problem can arise when continuous measures such as baseline achievement, income, disease severity, or prior exposure are divided into a few arbitrary categories.
Categorization may sometimes be justified for substantive or analytic reasons, but it should not be assumed to provide complete control merely because the confounder appears somewhere in the model.
The Statistical Model Can Represent the Confounder Incorrectly
Regression adjustment depends not only on which variables are included but also on how their relationships are specified.
Suppose baseline achievement has a nonlinear relationship with the outcome, yet the model assumes a simple linear relationship. Or suppose the effect of one confounder depends strongly on another variable, but the model does not represent that relationship.
The model may then leave residual differences inadequately controlled.
Cochrane specifically notes that residual confounding can remain when relationships involving confounding domains are imperfectly modeled. A regression model is an approximation to a data-generating process, not a cleansing procedure that automatically purifies an estimate.
More Covariates Do Not Necessarily Mean Less Bias
Researchers sometimes respond to confounding concerns by including every available variable in the model.
This can be problematic because not every predictor of the outcome is a confounder.
A variable may be a mediator lying on the causal pathway between exposure and outcome. Another variable may be a consequence of both exposure and outcome-related causes, creating a collider structure. Some variables may be measured after the intervention and affected by it.
Cochrane cautions that adjustment for factors that are not confounders, particularly variables affected by an intervention, can introduce bias. The correct adjustment set therefore depends on causal structure rather than the number of columns available in the dataset.
Watch Out
“We adjusted for all available covariates” is not automatically a methodological strength. If the adjustment set includes inappropriate variables or omits important common causes, adding more covariates can leave the estimate biased or introduce new bias.
Adjusting for a Mediator Can Change the Question
Suppose an instructional intervention increases students' study time, which subsequently improves examination performance.
If the research question concerns the intervention's total effect on examination performance, adjusting for post-intervention study time can remove part of the pathway through which the intervention works.
The resulting estimate may answer a different question, perhaps something closer to an effect not operating through study time under additional causal assumptions.
This does not mean mediators should never be included in models. Mediation analysis may specifically require them. The problem occurs when researchers call every covariate a confounder and interpret adjustment as though it always moves an estimate closer to the same target effect.
Adjusting for a Collider Can Introduce Bias
Some variables should not be conditioned on because doing so can create a noncausal association between other variables.
Consider a simplified situation in which both academic ability and severe financial difficulty influence whether a student receives a competitive support scholarship. Among scholarship recipients, knowing that a student has relatively lower academic ability may imply that financial difficulty was particularly severe, because either factor could contribute to selection.
If both academic ability and financial difficulty also relate to variables in the research question, restricting or adjusting on scholarship receipt can create associations that did not exist in the underlying population.
This phenomenon is commonly called collider bias. It illustrates why covariate selection should be based on causal reasoning rather than a rule that all measured variables should be controlled.
Statistical Significance Does Not Tell You Whether Confounding Is Gone
Researchers sometimes compare a crude and adjusted model and conclude that confounding has been controlled because the focal coefficient remains statistically significant.
That reasoning confuses hypothesis testing with bias assessment.
A biased estimate can be highly statistically significant, especially in a large sample. Conversely, a reasonably unbiased estimate may be imprecise and fail to cross a conventional significance threshold.
The relevant questions concern the magnitude and uncertainty of the estimate, the plausibility of remaining confounding, and the assumptions required for interpretation. A P value cannot certify that the adjustment set was causally appropriate.
A Small Change After Adjustment Does Not Prove There Was No Confounding
Another common approach is to compare crude and adjusted estimates. If they are similar, researchers conclude that confounding was negligible.
That comparison can be informative, but it is not definitive.
If important confounders were unmeasured, both estimates can remain similarly biased. Confounding from several variables can also operate in different directions. Model misspecification or poor measurement can leave little apparent change despite residual confounding.
Likewise, a large change after adjustment does not automatically prove that the adjusted estimate is correct. The adjustment itself may have introduced bias or changed the estimand.
Propensity Scores Do Not Solve the Unmeasured-Confounding Problem
Propensity-score matching, weighting, stratification, and related approaches can improve balance in measured baseline characteristics when used appropriately.
They do not generally balance variables that were never measured simply because a propensity score was calculated.
A matched sample may therefore look impressively balanced across the variables displayed in a table while remaining substantially different in an unobserved characteristic that influences both exposure and outcome.
The same conceptual limitation applies to many adjustment methods: sophisticated treatment of measured confounding does not establish absence of unmeasured confounding.
Adjustment Does Not Fix Other Forms of Bias
Even perfect control of confounding would not eliminate every threat to a study.
The exposure could be misclassified. Outcomes could be measured differently between groups. Participants with particular outcomes could be more likely to leave the study. The analyzed sample could have been selected through a process that creates bias. Researchers could selectively report favorable analyses.
These mechanisms belong to the broader set of threats to valid research. Confounding adjustment should therefore not be presented as a general-purpose correction for all observational-study limitations.
Randomization and Statistical Adjustment Solve Different Problems
Successful random assignment prevents baseline characteristics from systematically determining intervention allocation, providing strong protection against baseline confounding. Statistical adjustment in observational research instead attempts to reconstruct an appropriate comparison from observed information.
That distinction matters because observational adjustment depends critically on whether researchers measured and modeled the relevant confounding structure.
Randomized studies may still use adjusted analyses for precision, chance imbalance, prespecified estimands, or other reasons. They can also experience post-randomization complications and other biases. The point is not that randomization makes statistics unnecessary. It is that regression adjustment does not recreate the causal protection of successful randomization merely by including many covariates.
Sensitivity Analysis Can Make Remaining Uncertainty More Visible
When unmeasured or residual confounding is plausible, researchers need not choose between pretending it does not exist and abandoning the study.
Depending on the research problem, sensitivity analyses can examine how strong an unmeasured confounder would need to be to materially change the conclusion, how alternative model specifications affect estimates, or how different assumptions alter results.
These analyses do not prove that confounding is absent. Their value lies in making the robustness of the inference to specified assumptions more explicit.
That is a more defensible objective than declaring a model “fully adjusted,” a phrase that often promises considerably more than the analysis can establish.
