What If an Important Assumption Behind Your Research Design Turns Out to Be Wrong?

What Happens When Your Study Depends on Something That Turns Out Not to Be True?

Every research design rests on assumptions. Some are explicit. Others become visible only when the study starts behaving differently from what you expected.

You may have assumed that enough eligible participants would be available, that two study groups could remain meaningfully distinct, that an instrument would measure the intended construct in your population, that participants would complete follow-up, or that the variation needed for an analysis would actually occur.

Then the study begins, and one of those assumptions turns out to be wrong.

The important question is not simply whether an assumption failed. It is how much of the study's inferential logic depends on that assumption and whether a credible path still connects the data you can collect to the research question you intended to answer.

A Wrong Assumption Matters According to What Depends on It

Find the Assumption That Connects the Design to the Conclusion

Research Designs Depend on More Than Statistical Assumptions

Researchers often hear the word “assumption” and think immediately of normality, independence, homoscedasticity, proportional hazards, or another assumption associated with a statistical procedure. Those matter, but they are only one category.

A research design also makes substantive and operational assumptions about how the study will generate evidence. These may concern who can be recruited, what a comparison group represents, whether participants will adhere to a condition, whether an exposure varies sufficiently, whether a measure captures the intended construct, whether observations occur at the right time, or whether the study setting permits the intended contrast.

These assumptions interact. A recruitment assumption that fails may eventually become a population problem. A comparison assumption that fails can alter the effect being estimated. A measurement assumption can expose a deeper mismatch between the method and the research question.

State the Failed Assumption Explicitly

“The design isn't working” is not sufficiently diagnostic. Try completing a more precise sentence:

We designed the study assuming that ________, but we now have evidence that ________.

For example: “We assumed participants in different classrooms would have little opportunity to exchange the intervention materials, but substantial cross-class sharing is occurring.” Or: “We assumed the questionnaire would distinguish participants across the relevant range, but most scores cluster at the upper end.”

Once the assumption is stated clearly, ask which part of the study depends on it. That dependency is what determines the methodological consequence.

Distinguish a Failed Planning Assumption From a Failed Identifying Assumption

Not all assumptions carry the same inferential weight.

A recruitment rate that is lower than expected may sometimes be addressed by extending recruitment. An assumption required to identify the quantity of scientific interest can be much harder to repair. The study may continue producing data while losing the inferential property that originally made those data useful.

Ask Whether the Assumption Was Testable in the Study

Some assumptions can be examined directly or indirectly using observed information. Others cannot be established from the study data alone.

You might observe that recruitment is much slower than anticipated, that comparison-group participants are receiving intervention materials, or that a measurement device is malfunctioning. Other assumptions, particularly some assumptions underlying causal inference or missing-data methods, may not be fully testable from the observed data.

This distinction matters because “we did not find evidence that the assumption was violated” is not equivalent to “we proved the assumption true.” Where assumptions cannot be verified directly, researchers may need substantive justification, sensitivity analysis, alternative analyses, or appropriately qualified conclusions.

Do Not Automatically Change the Analysis Until the Assumption Appears to Pass

When an analytical assumption appears problematic, researchers sometimes cycle through transformations, exclusion rules, models, covariates, or statistical tests until one produces acceptable diagnostics or a preferred result.

A defensible response begins with the scientific problem rather than the desired output. Determine why the assumption matters, whether the diagnostic actually indicates a consequential violation, and which alternative method is justified by the data-generating process.

If the alternative analysis was not prespecified, record why it was chosen and distinguish it appropriately from the original plan. OSF guidance on preregistration encourages researchers to specify contingencies in advance where possible, such as what alternative procedure will be used if a statistical assumption is violated. It also emphasizes documenting significant deviations when unexpected changes occur.

Sometimes the Assumption Is Wrong Because the World Changed

Not every failed assumption reflects poor planning. External circumstances can change while research is underway.

A new policy may alter usual practice. A technology may become widely available during an intervention study. An institution may change procedures. A major external event may affect participant behavior or access. The assumption may have been reasonable when the study was designed and become false later.

The methodological consequence still needs assessment. Record when the external change occurred, which observations were potentially affected, and whether participants studied before and after the event remain meaningfully comparable.

Sometimes the Assumption Reveals a Problem in Another Part of the Study

Suppose you assumed the control condition would remain distinct from the intervention, but substantial contamination occurs. The immediate issue is a control or comparison group that is not working as planned.

If you assumed an instrument would work in the target population but participants systematically misunderstand it, the problem may instead concern a measure or instrument that is not functioning as expected.

The value of identifying the underlying assumption is that it helps locate the actual methodological problem rather than treating every unexpected event as a reason to redesign the entire study.

Consider Whether a Sensitivity Analysis Can Address Uncertainty

When a conclusion depends on an assumption that cannot be fully verified, sensitivity analysis may sometimes show how the results change under alternative plausible assumptions.

This can be particularly useful for assumptions involving missing data, model specification, measurement, unmeasured factors, or other analytical choices. A sensitivity analysis does not make an assumption true. Instead, it asks whether the substantive conclusion remains similar when that assumption is changed within a defensible range.

If conclusions change dramatically under reasonable alternatives, the study may be more assumption-dependent than a single primary analysis suggests. That dependence should inform interpretation.

A Failed Assumption Can Change the Research Question You Are Actually Answering

The link between a study objective and its design should be explicit. In randomized trials, for example, the estimand framework described in CONSORT 2025 connects the clinical question to the population, treatment conditions, endpoint, summary measure, and handling of post-randomization events. This illustrates a broader methodological principle: when circumstances alter one of the elements defining the intended comparison or quantity of interest, researchers should reconsider what question the resulting evidence actually answers.

If the failed assumption means the current design can no longer support a central part of the original inference, consider whether the planned method still fits the research question.

The Appropriate Response May Range From Accommodation to Redesign

There is no universal rule that every violated assumption requires redesign. The response should be proportional to the role the assumption plays.

A minor operational assumption may require only a procedural adjustment. A statistical assumption may have a well-established alternative analysis. A population assumption may require narrower generalization. A central design assumption may be so fundamental that the intended inference no longer survives.

If adaptation becomes necessary after data collection has started, preserve the original design and document what prompted the change. The study should not acquire a cleaner history during manuscript preparation than it had during conduct.

An Assumption About Separation Between Study Groups Fails

The wrong response would be to conclude automatically that contamination explains any weak or non-significant effect. The failed assumption changes how the comparison should be interpreted, but it does not reveal what the result would have been in a perfectly separated study.

Common Mistakes When an Assumption Fails

Trace the Consequences Before Changing the Design

When an assumption fails, resist jumping directly from discovery to redesign. Trace the dependency first: what procedure, comparison, estimate, or conclusion relied on that assumption?

When a modification is needed, ask whether adapting the design is methodologically defensible rather than assuming that any solution preserving the project is acceptable.

If the failed assumption removes the central logic of the study and no realistic correction restores it, continuing unchanged may consume additional participants, time, and resources without answering the intended question. At that point, consider whether the study should be stopped or redesigned because the research design no longer works.

When a Research Design Assumption Fails, Check These Points

Questions About Failed Research Design Assumptions

What Matters Is What the Assumption Was Holding Up

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.