03 · What You Need to Know
Start with the decision or purpose, then work backward to the uncertainty
Do not begin by ranking knowledge gaps by size
A conventional research-gap exercise often identifies where evidence is absent, sparse, inconsistent, or methodologically weak. These are useful observations, but they do not establish research priority.
An area can be almost completely unknown and still have little immediate consequence. Another issue may already have a substantial evidence base, yet a small unresolved uncertainty determines whether one action is preferable to another.
This is why the question with the greatest uncertainty is not automatically the question with the greatest research value.
First define what the evidence is supposed to inform
For applied research, begin with the decision rather than the parameter. What choice must be made? What are the realistic alternatives? Which outcomes matter? Who experiences those outcomes?
Only after the decision has been specified should you ask which uncertainties prevent confidence that the currently preferred action is actually the best one.
This reversal is important. Starting with whatever happens to be easiest to measure can produce increasingly precise answers to questions that have little influence on what anyone should do.
Map the uncertainties that could affect the decision
Once the decision is defined, identify the uncertain quantities or assumptions that could influence it. Depending on the problem, these might concern treatment or intervention effects, harms, costs, implementation, long-term outcomes, uptake, generalizability, baseline risk, measurement, or model structure.
Different types of uncertainty should not automatically be treated as interchangeable. NICE, for example, recognizes that important uncertainties may arise because evidence is absent, insufficient, methodologically limited, inconsistent, not applicable to the relevant population, focused on a different question, or outdated.
The purpose of mapping uncertainty is not to produce the longest possible list. It is to expose which unknowns could plausibly alter the conclusion.
Ask whether plausible values of the uncertainty change the preferred action
An uncertainty becomes particularly relevant when plausible values lead to different decisions.
Suppose the long-term effect of an intervention is uncertain. If the intervention remains preferable whether that effect is at the lower or upper end of the plausible range, reducing the uncertainty may add relatively little decision value.
If the lower end favors rejection while the upper end favors adoption, the uncertainty is directly connected to decision risk.
Large uncertainty
A quantity, effect, assumption, or outcome is poorly known or estimated imprecisely.
Decision-relevant uncertainty
Plausible values of that uncertainty can materially affect which action is preferred or the consequences expected from it.
Then ask what happens if the uncertainty leads to the wrong choice
Decision sensitivity is only part of the problem. The consequences of being wrong also matter.
An uncertainty might frequently reverse which of two nearly equivalent options is technically preferable, yet choosing incorrectly has almost no practical consequence. Another uncertainty may rarely reverse the decision, but when it does, the resulting loss is severe.
The uncertainty worth reducing therefore depends on both the probability of making an inferior decision and the magnitude of the resulting loss.
Expected value of perfect information provides the overall ceiling
The expected value of perfect information (EVPI) estimates the expected improvement available if all uncertainty represented in a decision problem could be eliminated before the decision is made.
EVPI is useful as an initial ceiling. If eliminating all decision uncertainty would provide little expected benefit, there is correspondingly little decision value available from further research within that model.
But EVPI does not identify which individual uncertainty should be studied. For that, a more targeted analysis is needed.
EVPPI can identify which parameters drive valuable uncertainty
The expected value of partial perfect information (EVPPI) estimates the expected value of completely resolving uncertainty in a particular parameter or group of parameters while other uncertainty remains.
This is especially useful when several uncertain quantities compete for research attention. The parameter with the largest variance does not necessarily have the largest EVPPI. A less uncertain parameter can have greater EVPPI if its plausible values frequently alter the preferred decision or carry larger consequences.
ISPOR guidance identifies EVPPI as a method for assessing which parameters or groups of parameters contribute to decision uncertainty and therefore where additional evidence may potentially be valuable.
High EVPPI identifies a target, not automatically a study
Perfectly resolving an uncertainty is hypothetical. Real research may only reduce part of it.
An uncertainty can have high EVPPI yet be extremely difficult or expensive to investigate. A feasible study may provide little information about it. Another uncertainty with somewhat lower EVPPI may be much easier to reduce and therefore offer greater research value in practice.
The expected value of sample information (EVSI) addresses this difference by estimating the expected value of the information obtainable from a particular proposed study design.
This distinction prevents a common mistake: identifying an important uncertainty and assuming that any study nominally related to it must therefore be worthwhile.
Ask whether the uncertainty is reducible
Some uncertainties are much more amenable to research than others. A larger sample may reduce sampling uncertainty. Longer follow-up may clarify persistence. A different study population may address generalizability. Better measurement may reduce measurement uncertainty.
Other uncertainties may be difficult to resolve because relevant outcomes occur decades later, the necessary experiment is unethical, the population is extremely rare, or important structural assumptions cannot be tested directly.
Research prioritization should therefore distinguish between uncertainty that matters and uncertainty that can realistically be reduced.
Match the study design to the source of uncertainty
Once the high-value uncertainty is identified, ask what evidence would actually reduce it.
If uncertainty concerns long-term persistence, another short-term study may add little. If uncertainty concerns causal effects, a larger descriptive dataset may not solve the problem. If uncertainty concerns generalizability, another study in the same narrow population may provide limited information.
The research design should follow the uncertainty rather than the other way around.
Population and time horizon can change which uncertainty matters most
An uncertainty with a small expected consequence for each individual can still have substantial research value when the decision affects a large population or will be repeated for many years.
Conversely, an uncertainty with a large per-person consequence may have lower total research value when very few people will ever face the decision.
Formal value-of-information analysis therefore commonly scales information value to the population expected to benefit over a relevant time horizon. This can materially change research priorities.
Research costs determine whether valuable uncertainty is worth pursuing
After identifying a consequential and reducible uncertainty, the remaining question is whether reducing it is worth the resources required.
A study with high EVSI can still be unattractive if its cost is even higher. The expected net benefit of sampling (ENBS) compares the expected value of information from a proposed study with its expected research cost.
This is why identifying the uncertainty worth reducing eventually requires comparing the cost of research with the cost of remaining uncertain.
Do not overlook structural and contextual uncertainty
Not all important uncertainty appears as a wide interval around a parameter. A decision model may omit relevant mechanisms, assume relationships that are poorly supported, or apply evidence from a population that differs substantially from the target setting.
ISPOR-SMDM guidance distinguishes parameter uncertainty from other forms such as structural uncertainty and heterogeneity. This matters because simply collecting a larger sample may reduce parameter uncertainty while leaving a more consequential model or applicability problem untouched.
The most valuable next study may therefore require a different design rather than simply more observations.
Watch Out
Do not identify research priorities solely by inspecting which estimate has the widest confidence interval or which topic has the fewest published studies. Those indicators can reveal uncertainty, but they do not show whether reducing that uncertainty would improve an important decision or whether a feasible study can resolve it.
For non-decision-oriented research, define value differently but explicitly
Not every worthwhile research question is linked to an immediate choice among interventions or policies. Basic, exploratory, theoretical, and methodological research may create value by explaining phenomena, developing theory, improving measurement, creating methods, or enabling future inquiry.
In those settings, the same discipline is still useful: ask what becomes possible if the uncertainty is reduced. The answer may be scientific rather than an immediate change in action.
What should be avoided is treating “we do not know this yet” as a complete research justification. Uncertainty identifies an opportunity to learn. It does not by itself establish the value of learning it.
