Should You Study the Question With the Greatest Uncertainty or the Greatest Consequences?

Should researchers investigate what they know least about or what matters most?

When several unanswered questions compete for limited research resources, one seems like an obvious priority: investigate the area with the greatest uncertainty.

That approach can be misleading. Researchers may know very little about one parameter without that uncertainty having much influence on any important decision. Meanwhile, another parameter may already be estimated reasonably well, yet the remaining uncertainty could determine whether a costly intervention is adopted, a policy is implemented, or a large population is exposed to an inferior option.

The most uncertain question and the most consequential uncertainty are therefore not necessarily the same research priority.

Prioritize uncertainty according to its consequences, not its size alone

Research priority emerges from the interaction between uncertainty and consequences

The largest knowledge gap is not automatically the most important one

Researchers often identify priorities by looking for areas where evidence is sparse, inconsistent, imprecise, or absent. These are legitimate ways to identify uncertainty, but they do not establish how valuable reducing that uncertainty would be.

Suppose researchers are highly uncertain about Parameter A but nearly all plausible values lead to the same decision. Parameter B is estimated more precisely, but its remaining uncertainty straddles a threshold at which the preferred decision changes. Further information about Parameter B may be more valuable even though researchers already know more about it.

This distinction is why the size of uncertainty cannot by itself establish whether another study is justified.

Consequences determine what is at stake when uncertainty leads us astray

Uncertainty matters because decisions made with incomplete information can turn out to be inferior to decisions that would have been made with better information. The importance of this possibility depends partly on what is lost when the wrong action is chosen.

A relatively small uncertainty may deserve substantial attention when an incorrect decision could cause serious harm, consume large resources, affect many people, or be difficult to reverse. By contrast, substantial uncertainty can be less urgent when the competing alternatives produce nearly equivalent outcomes.

This does not mean that researchers should simply rank questions by the severity of their possible consequences. A catastrophic outcome may be associated with a decision that additional evidence cannot realistically improve. Consequence is necessary context, not a standalone prioritization rule.

Probability of error and magnitude of loss must be considered together

Imagine Decision A has a 30% probability that the currently preferred option is not actually best, but choosing incorrectly would have only minor consequences. Decision B has only a 5% probability of error, yet a wrong choice would cause a much larger loss.

The 30% figure does not automatically make Decision A the higher research priority. Nor does the severe potential loss automatically make Decision B the priority. What matters is how the probability and consequences combine into expected losses and how much research could reduce them.

This is one reason decision analysis often evaluates expected outcomes rather than uncertainty measures in isolation.

Expected opportunity loss connects uncertainty to consequences

Suppose current evidence supports Action A. Because some uncertainty remains, there are possible states of the world in which Action B would actually produce a better outcome.

The difference between the outcome obtained from the selected action and the outcome that could have been obtained from the best action is an opportunity loss. Before the uncertainty is resolved, the expected opportunity loss incorporates both how likely different states are and how consequential choosing incorrectly would be.

The expected value of perfect information (EVPI) can be interpreted as the expected gain from eliminating all uncertainty relevant to the decision. Equivalently, it reflects the expected opportunity loss associated with making the decision using current information rather than perfect information.

EVPPI can help identify which uncertainty matters most

When a decision model contains several uncertain parameters, the expected value of partial perfect information (EVPPI) can estimate the value of eliminating uncertainty in a particular parameter or group of parameters while uncertainty in the others remains.

This is particularly useful for research prioritization. The parameter with the widest distribution is not necessarily the parameter with the largest EVPPI. What matters is how uncertainty in that parameter affects which decision is optimal and the consequences of making the wrong choice.

A parameter may therefore be highly uncertain yet have low EVPPI because changes in its value barely affect the decision. Another may have modest uncertainty but high EVPPI because it is a major driver of decision uncertainty.

Formal value-of-information methods are especially established in health economic decision modelling, but the underlying reasoning is more general: identify the uncertainty that is actually worth reducing rather than assuming the largest gap deserves priority.

The number of people affected can transform research priority

An uncertainty may have modest consequences for each individual decision but substantial consequences when the decision is repeated across a large population.

For example, information that produces a very small expected improvement per person can become highly valuable when the same evidence informs decisions for hundreds of thousands of people. Conversely, an uncertainty with large individual consequences may have a smaller population-level research value if very few people will ever face the decision.

Formal value-of-information analyses can therefore consider the eligible population and the period during which the evidence is expected to remain useful.

Consequential uncertainty still needs to be researchable

Identifying a high-value uncertainty does not mean that any study addressing it should receive priority. The proposed research must actually be capable of reducing that uncertainty.

A parameter may have high potential value if known perfectly but be extremely difficult to estimate. A feasible study may reduce only a small fraction of its uncertainty. Another parameter with lower theoretical importance might be much easier to investigate and yield greater expected improvement from a realistic study.

This is where the expected value of sample information (EVSI) becomes important. EVSI concerns the expected value of information from a particular proposed study rather than from hypothetical perfect knowledge.

The best research priority depends on value after research costs

Research resources are finite. Funding one study means those resources cannot be used for another purpose. Consequently, research prioritization should consider not only the information a study may produce but also what must be spent to obtain it.

The expected net benefit of sampling (ENBS) compares the expected value of sample information with the expected cost of the study. A study aimed at a highly consequential uncertainty can still be a poor investment if it is extremely costly and expected to reduce little uncertainty.

Conversely, a relatively inexpensive study addressing a moderately consequential uncertainty may have high net value.

This makes the relationship between research costs and the costs of remaining uncertain central to choosing among competing research opportunities.

Not all research priorities can be ranked through immediate decision consequences

Decision-oriented reasoning is most directly applicable when research is intended to inform identifiable choices. Some research questions have value for theory development, discovery, measurement, methodological innovation, or foundational knowledge even when their downstream consequences cannot yet be estimated credibly.

In those settings, uncertainty and consequences may still inform priority setting, but formal value-of-information analysis may not capture all relevant forms of scientific value.

The least-known parameter may not be the most valuable one to study

This conclusion would change if usage strongly determined costs or learning outcomes. Research priority comes from the role an uncertainty plays in the particular decision, not from the name of the parameter or the apparent width of its uncertainty alone.

Common mistakes when choosing which uncertainty to study

Prioritize the uncertainty with the greatest reducible consequences

When several research questions compete for attention, begin by asking what decision each uncertainty affects. Then consider what could be lost if current evidence leads to the wrong choice.

Next, determine whether a realistic study can meaningfully reduce that uncertainty. This prevents research priority from being determined either by uncertainty alone or by dramatic consequences that research cannot actually address.

The practical goal is therefore not to find the biggest blank space in the literature. It is to determine which question would be most valuable to answer next.

Before choosing which uncertainty to research

Questions about prioritizing uncertainty and consequences

Study the uncertainty whose reduction is expected to matter most

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.