When Is Existing Uncertainty Large Enough to Justify Another Study?

How uncertain must the evidence be before another study is justified?

Researchers often encounter evidence that is incomplete, imprecise, inconsistent, or sensitive to assumptions. At some point, the question arises: is the remaining uncertainty large enough to justify collecting more data?

It is tempting to search for a statistical threshold. Perhaps a confidence interval is too wide, the probability of one conclusion is too low, or the number of existing studies is simply too small.

But no single numerical threshold can answer the question across research problems. The significance of uncertainty depends on what is uncertain, what decisions depend on it, what happens if those decisions are wrong, and how much a feasible new study could improve the evidence.

Uncertainty is large enough when its consequences make reducing it worthwhile

The size of uncertainty cannot be judged independently of its consequences

Statistical uncertainty and consequential uncertainty are different

Researchers commonly describe uncertainty through standard errors, confidence intervals, posterior distributions, prediction intervals, sensitivity analyses, or variation across studies. These tools characterize uncertainty in different ways, depending on the analytical framework and research problem.

They do not, by themselves, determine whether another study is worth conducting.

A wide interval may span values that all support the same action. A narrower interval may straddle a critical decision threshold. In the first situation, statistical uncertainty may be substantial while decision uncertainty is limited. In the second, relatively modest statistical uncertainty may be highly consequential.

There is no universal threshold for “enough” uncertainty

A statement such as “the confidence interval is wide, therefore another study is needed” skips an important part of the reasoning. Wide relative to what?

The relevant reference point is often the decision. Suppose an intervention becomes worthwhile only if its effect exceeds a particular level. Uncertainty concentrated near that level may be important because small changes in what is believed could reverse the preferred action.

If the entire plausible range lies comfortably on the same side of the decision threshold, additional precision may be less consequential. The evidence can remain imperfect while the decision is relatively robust.

This is one reason uncertainty can become too small to warrant further research even though it has not disappeared.

Ask how often current uncertainty could lead to the wrong choice

Decision analysis reframes uncertainty in terms of actions and consequences. Given current evidence, one option has the highest expected value. Yet uncertainty means that another option could turn out to have been preferable.

The probability of making the wrong decision can help characterize this problem, but probability alone is insufficient. A 5% chance of choosing incorrectly could be extremely important if the consequences are catastrophic or affect millions of people. A 30% chance might matter much less if the alternatives have nearly identical outcomes and the decision is easily reversible.

Thus, both the probability of error and the magnitude of its consequences need consideration.

The expected opportunity loss captures both probability and consequence

One useful decision-theoretic concept is opportunity loss: the value forgone because the chosen option is not the best option under the true state of the world.

Before uncertainty is resolved, researchers do not know which state is true. The expected opportunity loss therefore averages the potential losses across the uncertainty represented in the decision model.

The expected value of perfect information (EVPI) can be interpreted as the expected opportunity loss caused by making the decision with current information. Equivalently, it is the expected improvement that would be possible if all relevant uncertainty could be eliminated before the decision.

If EVPI is effectively negligible for the relevant population and decision horizon, then even perfect information offers little expected improvement. That places a strong upper bound on the value of conducting additional research solely to inform that decision.

A small individual value can become large across a population

The consequences of uncertainty may accumulate when a decision affects many people or is repeated over time. An expected loss that appears trivial for one individual can become substantial when multiplied across a large population.

Formal population-level value-of-information analysis therefore considers the number of people expected to face the decision and the period over which the information would remain useful. The appropriate population and time horizon must be justified for the particular decision.

This helps explain why seemingly modest uncertainty can justify substantial research in high-volume decisions, while greater uncertainty may not justify research for a rare or one-off decision.

Identify which uncertainty is driving the decision

A research problem may contain many uncertain quantities, but they need not contribute equally to decision uncertainty.

Perhaps an intervention's acquisition cost is known precisely, its immediate effect is reasonably established, but its long-term persistence is highly uncertain. If the adoption decision is sensitive mainly to the long-term effect, collecting yet more short-term outcome data may add relatively little value.

The expected value of partial perfect information (EVPPI) extends value-of-information analysis by estimating the value of completely resolving uncertainty in a particular parameter or group of parameters. This can help identify where additional evidence would potentially matter most.

The broader principle applies even without formal modelling: identify the uncertainty that actually drives the possibility of a wrong decision before designing another study.

High-value uncertainty still does not automatically justify a study

Suppose eliminating uncertainty would be extremely valuable. It still does not follow that a particular proposed study should be conducted.

Real research provides sample information, not perfect information. A small or poorly targeted study may barely reduce the uncertainty that matters. A well-designed study may reduce it substantially but cost more than the information is expected to be worth.

The expected value of sample information (EVSI) addresses the expected value of the uncertainty reduction produced by a particular study design and sample size. Comparing that value with research costs provides a more realistic basis for deciding whether the proposed study itself is worthwhile.

This is the distinction between asking whether uncertainty is worth reducing and asking whether a particular research project is an efficient way to reduce it.

The study must address the uncertainty that matters

Another study can add data without resolving the relevant uncertainty. Replicating the same methodological limitations, measuring outcomes that do not determine the decision, studying a poorly matched population, or using a sample too small to provide useful information may leave the central problem largely intact.

The justification for further research should therefore specify not only that uncertainty exists, but what evidence is needed to reduce it.

This shifts the reasoning from “we need another study” to “we need this kind of evidence because this particular uncertainty currently prevents a sufficiently robust decision.”

Research costs and delay determine whether reducing uncertainty is worthwhile

Research uses resources that could be allocated elsewhere. It may also delay decisions while evidence is collected. Those costs need to be weighed against the expected benefit of reducing uncertainty.

Consequently, the point at which uncertainty is “large enough” cannot be defined by uncertainty alone. It depends partly on the cost of research compared with the cost of remaining uncertain.

This framework is strongest when research informs a decision

Value-of-information reasoning is particularly useful for applied research in which evidence is expected to inform identifiable choices. It should not be treated as the sole criterion for all scientific inquiry.

Exploratory and basic research may investigate uncertainties precisely because their eventual implications are not yet known. The scientific value of such work may involve theory development, discovery, methodological progress, or opening new lines of inquiry rather than reducing uncertainty around an existing decision.

The appropriate conclusion is therefore conditional: when another study is justified on the grounds that decision-makers need better evidence, the size of the relevant uncertainty should be evaluated through its potential consequences for that decision.

A narrower uncertainty can matter more than a wider one

The lesson is not that learning effects always matter more than usage. In another decision model, usage might strongly determine cost, implementation capacity, or effectiveness. The point is that the importance of uncertainty depends on how it propagates through the particular decision.

Common mistakes when deciding whether uncertainty warrants another study

Judge uncertainty by what it could cause, not merely by how large it looks

When deciding whether another study is justified, begin by identifying the action that current evidence supports. Then ask what plausible states of the uncertain evidence would make another action preferable.

If those states are sufficiently plausible and the consequences of choosing incorrectly are meaningful, the uncertainty deserves closer attention. The next question is whether research can reduce it efficiently.

This approach also prevents a common research-prioritization error. You should not automatically study the question with the greatest uncertainty rather than the greatest consequences. The two may point to entirely different research priorities.

More broadly, deciding whether uncertainty warrants another study is one part of determining whether more research is actually needed before a decision can be made.

Before deciding that uncertainty justifies another study

Questions about how much uncertainty warrants more research

Uncertainty becomes research-worthy when its consequences matter

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.