03 · What You Need to Know
When a Missing Comparison Creates a Genuine Research Gap
A Research Question Includes the Comparator
In comparative research, what an intervention, exposure, policy, strategy, or other option is compared against is part of the research question itself.
Evidence-synthesis frameworks commonly make this explicit. AHRQ has classified content gaps according to the PICOS elements: population, intervention, comparison, outcome, and setting. A missing or inadequate comparison can therefore represent a specific limitation in the evidence rather than merely an incidental feature of previous studies.
The same principle appears in Cochrane guidance, where review questions involving interventions specify both the intervention and comparator. Evidence about an option cannot always answer a question about its performance relative to the alternative that actually matters.
Evidence That A Works and B Works Does Not Necessarily Tell You Whether A Is Better Than B
Suppose randomized studies show that Intervention A performs better than a control condition. A separate group of studies shows that Intervention B also performs better than a control condition.
It can be tempting to compare the results informally and conclude that whichever intervention produced the larger effect in its own studies must be superior.
That conclusion may be unjustified. The A studies and B studies may involve different populations, baseline risks, control conditions, outcome definitions, follow-up periods, settings, or other factors affecting their estimates.
Separate effectiveness questions
Does A perform better than its comparator? Does B perform better than its comparator?
Comparative question
How does A perform relative to B under conditions relevant to the decision?
The third question does not necessarily follow directly from the first two.
The Missing Comparison Must Be Relevant to a Real Question
Almost any literature can be made to contain a missing comparison if you generate enough possible combinations.
If five interventions exist, there are multiple pairwise comparisons that could theoretically be studied. Add different doses, delivery formats, populations, time points, and outcomes, and the number of possible comparisons grows rapidly.
Not every absent pairwise study is therefore a meaningful research gap. The missing comparison matters when researchers or decision-makers genuinely need to choose between the alternatives, when theory requires distinguishing them, or when the comparison could otherwise change an important conclusion.
Watch Out
Do not manufacture a comparison gap simply because two options have never appeared together in one study. Explain who needs the comparison, what decision or scientific question depends on it, and why the existing evidence cannot answer that question adequately.
The Best Comparator Is Usually the One Relevant to the Decision
A comparison can technically exist while still being poorly aligned with the real question.
Suppose a new program is repeatedly compared with doing nothing, even though organizations deciding whether to adopt it would actually replace an established program. Evidence against no program may show that the new approach has some benefit, but it may not establish whether changing from the established option is worthwhile.
Cochrane's guidance on indirectness explicitly recognizes comparator mismatch as one reason evidence may be less direct for the question of interest. If studies compare interventions against options that differ from the relevant comparator, the evidence may not directly answer the decision at hand.
A Missing Head-to-Head Study Is Not Always the Same as Missing Comparative Evidence
This distinction is particularly important.
Two interventions may never have been compared directly within the same trial, yet comparative evidence may still be available indirectly. Network meta-analysis is one method that can combine direct and indirect evidence across a connected network of interventions.
Cochrane describes an indirect comparison using a simple example: if trials compare A with B and other trials compare A with C, information from those studies may permit an estimate of B versus C even though B and C have not been compared directly.
Therefore, “no head-to-head trial exists” does not automatically mean “nothing is known about the comparison.”
Indirect Comparisons Depend on Important Assumptions
Indirect evidence is not a free substitute for direct comparison.
Cochrane identifies transitivity as a core assumption underlying valid indirect comparisons. In simple terms, the sets of studies contributing to the indirect comparison need to be sufficiently comparable with respect to characteristics that could modify the relative effects.
Suppose A versus B studies involve participants with substantially different characteristics from A versus C studies, and those characteristics affect treatment effects. Combining those studies to infer B versus C may be inappropriate.
The credibility of an indirect comparison therefore depends on the evidence network and the plausibility of its assumptions, not simply on whether the necessary mathematical connection exists.
Direct Evidence Can Be Valuable When Indirect Evidence Is Uncertain
If the comparison matters but available indirect evidence is uncertain or relies on questionable assumptions, a direct head-to-head study may provide valuable information.
Cochrane's GRADE guidance treats indirect comparison as one form of indirectness. Where A and B have not been directly compared but both have been compared with another option, an indirect estimate may be possible, but its certainty must account for the indirect nature of the evidence and other relevant limitations.
This does not mean direct evidence is automatically flawless. A poorly designed head-to-head study can be less informative than strong indirect evidence. The appropriate question is what evidence would most effectively reduce the comparative uncertainty.
A Missing Comparison Can Be an Evidence Gap Rather Than a Knowledge Void
The distinction between knowledge, evidence, methodological, and contextual gaps is useful here.
Researchers may already know a great deal about both options individually. The problem is that the evidence needed to compare them for a particular question is absent, indirect, weak, or otherwise inadequate.
That makes a missing comparison especially easy to overlook. A literature can appear mature because both options have been extensively studied while still failing to provide the comparative evidence that a real decision requires.
The Outcome of the Comparison Matters Too
A direct comparison is not automatically sufficient merely because two options appear in the same study.
The studies must also measure outcomes relevant to the question. If A and B have been compared only for a short-term surrogate outcome but the decision depends on long-term benefits, harms, costs, implementation, or another outcome, the comparative evidence may remain inadequate.
The gap is therefore best described at the level of the complete question: which options, compared for which outcomes, in which population and setting, over what relevant timeframe?
Different Comparisons Can Produce Different Conclusions About the Same Option
An intervention can look effective against one comparator and unimpressive against another.
For example, a new approach may clearly outperform no intervention while offering little advantage over an established alternative. Those findings are not contradictory. They answer different questions.
This is why phrases such as “the intervention is effective” can be incomplete without specifying the comparator. Comparative conclusions are relational: better, worse, equivalent, safer, cheaper, or more feasible than what?
A Missing Comparison Can Be Theoretically Important
Not all comparison gaps concern choosing between interventions.
Researchers may need to compare competing explanations, models, measurement approaches, populations, conditions, or theoretical predictions. If two explanations both fit existing observations but have not been tested under conditions that distinguish between them, the missing comparison can leave a theoretical question unresolved.
The same logic applies: the comparison matters because it discriminates among plausible conclusions, not merely because nobody has performed that exact study.
A Missing Comparison Can Also Be Methodological
Sometimes the gap concerns methods themselves.
Suppose several analytical approaches are commonly used for the same task, but little reliable evidence compares their performance under realistic conditions. Researchers may not know which approach produces more accurate estimates, handles particular data problems better, or is more robust under specified assumptions.
A comparative methodological study can address that uncertainty if the comparison is relevant to how researchers actually choose among the methods.
Do Not Confuse “Different” With “Comparative”
A study can include two groups without answering a meaningful comparative question.
If groups differ simultaneously in many important ways, attributing observed differences to the factor of interest may be difficult. Likewise, comparing results from two unrelated studies informally is not equivalent to designing research around a valid comparison.
Good comparative research requires a design and analysis capable of supporting the intended comparative inference.
Sometimes Existing Evidence Can Answer the Comparison Without New Primary Research
Before proposing a new head-to-head study, examine the entire evidence base.
If multiple relevant interventions form a sufficiently connected and comparable evidence network, methods such as network meta-analysis may provide comparative estimates using both direct and indirect evidence. Cochrane notes that such methods can be particularly useful when decision-makers face several competing interventions.
In other cases, existing datasets may permit a valid reanalysis, or a systematic review may reveal direct comparative studies that were missed in the initial search.
The goal is to answer the comparative question reliably, not necessarily to collect new data.
| Evidence situation |
Is there necessarily a comparison gap? |
What to check |
| A and B have each been compared with the same relevant control |
Possibly |
Whether a valid indirect comparison is possible and sufficiently certain |
| A and B have never been studied together |
Not automatically |
Whether other direct or indirect evidence answers the comparative question |
| A has only been compared with placebo, but decisions concern A versus current practice |
Potentially |
Whether the existing comparator is sufficiently relevant to the real decision |
| A and B have been compared directly, but only for an irrelevant outcome |
Potentially |
Whether the outcome needed for the decision remains unstudied |
| A and B are both well studied, but in very different populations |
Potentially |
Whether cross-study differences make indirect comparison unreliable |
| An arbitrary pair of options has never been compared |
Not by itself |
Whether anyone needs the comparison and what knowledge it would add |
The Missing Comparison Must Still Be Worth Studying
Even after establishing that comparative evidence is genuinely inadequate, another question remains: does resolving the comparison matter enough to justify research?
AHRQ's work on research prioritization emphasizes considerations such as potential research impact, whether research is needed to answer the question, foundational value, and timeliness. A comparison affecting major decisions may therefore deserve much higher priority than an academically possible but inconsequential comparison.
This is the broader distinction between finding a gap and determining whether that research gap is actually worth filling.
The Proposed Study Must Address the Actual Comparison Gap
If the evidence gap is A versus B, another A-versus-no-intervention study may do little to resolve it.
The new research should generate information relevant to the missing comparison. That could mean a direct head-to-head study, a stronger evidence network, an appropriate comparative observational design, a methodological comparison, or another design suited to the question.
Do not identify one gap and then conduct a study that answers a different question simply because it is easier.
