01 · The Question
What If There Are Already Too Many Systematic Reviews?
Systematic reviews were developed partly to make large bodies of primary research more manageable. In some fields, however, researchers and decision-makers now face a second problem: there are so many systematic reviews that understanding the reviews themselves has become difficult.
That is where umbrella reviews enter the picture.
An umbrella review moves the unit of evidence up one level. Rather than primarily searching for individual trials, surveys, cohort studies, or other primary studies, it systematically identifies and synthesizes existing systematic reviews or research syntheses addressing a broader question.
This can provide a high-level view of a mature evidence base. It can also introduce methodological complications that do not arise in quite the same way when working directly with primary studies, particularly when several reviews contain the same studies.
03 · What You Need to Know
What Changes When the Review Becomes the Evidence?
A Systematic Review Usually Starts With Primary Studies
A conventional systematic review defines a research question and uses explicit, reproducible methods to identify eligible primary studies. The researchers then appraise and synthesize those studies using methods appropriate to the question and evidence.
For example, a systematic review examining whether an educational intervention improves student achievement might identify individual randomized trials and quasi-experimental studies. Those primary studies become the evidence base from which the review draws its conclusions.
A systematic review may or may not include a meta-analysis of compatible quantitative findings. Statistical pooling is a possible synthesis technique rather than the defining feature of the review.
An Umbrella Review Starts With Reviews
An umbrella review operates at a different level. JBI describes umbrella reviews as syntheses of existing research syntheses, while Cochrane uses the term overview of reviews for a closely related approach.
In a Cochrane Overview, the unit of searching, inclusion, and data analysis is the systematic review rather than the primary study. The overview systematically identifies multiple reviews addressing related questions and extracts and analyzes their findings across important outcomes.
You may also encounter terms such as review of reviews, overview of systematic reviews, meta-review, and systematic review of reviews. Terminology varies, so the methodology and eligibility criteria should make clear what kind of synthesis is actually being conducted.
Systematic review
Usually searches for, selects, appraises, and synthesizes eligible primary studies relevant to a defined review question.
Umbrella review
Searches for, selects, appraises, and synthesizes eligible systematic reviews or other specified research syntheses relevant to a broader review question.
| Feature |
Systematic Review |
Umbrella Review |
| Typical unit of inclusion |
Primary studies |
Systematic reviews or other eligible research syntheses |
| Typical scope |
Defined question, often relatively focused |
Often broader question spanning multiple related review questions, interventions, outcomes, or comparisons |
| Search targets |
Primary-study records |
Systematic reviews or research syntheses |
| Critical appraisal |
Appraises eligible primary studies using tools appropriate to their designs |
Appraises included reviews or syntheses using tools appropriate to review-level evidence |
| Major overlap problem |
Multiple reports of the same primary study |
The same primary studies appearing in multiple included reviews |
| Useful when |
The question requires direct synthesis of primary evidence |
Multiple suitable systematic reviews already address related parts of a broader question |
| Limited when |
Relevant primary evidence is absent or inadequate |
Existing reviews are absent, outdated, poor quality, or mismatched to the question |
Umbrella Reviews Are Especially Useful for Broad Decision Questions
Suppose a decision-maker wants to compare several interventions for the same broad problem. Separate systematic reviews already examine intervention A, intervention B, intervention C, and intervention D. Asking the decision-maker to locate and reconcile all of those reviews may defeat part of the purpose of evidence synthesis.
An umbrella review can bring those review-level findings together. Cochrane notes that overviews can describe the current body of systematic review evidence or address a new review question that was not itself the focus of the included systematic reviews.
This higher-level perspective can be particularly useful when the decision involves choosing among several alternatives rather than determining whether one narrowly defined intervention works.
Reviewing Reviews Does Not Automatically Mean Less Work
The phrase "review of reviews" can sound like an efficient shortcut. Sometimes it is more efficient than starting again from thousands of primary records, but it introduces another layer of complexity.
Researchers need to determine what qualifies as an eligible systematic review, assess the methodological quality or risk of bias of those reviews, extract findings consistently, reconcile different definitions and outcomes, evaluate how current each review is, and decide how to handle conflicting conclusions.
They must also understand enough about the underlying primary evidence to avoid treating each review as an entirely independent piece of information.
Overlap Is One of the Central Problems
Imagine that an umbrella review includes five systematic reviews. Review A contains 20 primary studies, Review B contains 18, and Review C contains 25. Those numbers do not necessarily represent 63 different studies. The same influential trials may appear in all three reviews.
If overlapping reviews are treated as independent bodies of evidence, the same primary study can effectively influence the umbrella review several times. This can create a misleading impression of how much independent evidence exists.
Cochrane therefore identifies management of overlapping systematic reviews as a specific methodological issue in overviews. Authors may need to examine the primary-study composition of included reviews and use an explicit approach to dealing with overlap.
Watch Out
Five systematic reviews reaching similar conclusions do not necessarily represent five independent bodies of evidence. They may substantially overlap in the primary studies they include. Always investigate review overlap before interpreting the number of reviews as evidence of replication.
The Umbrella Review Inherits Limitations From the Reviews Below It
An umbrella review cannot automatically repair methodological weaknesses in its included systematic reviews. If the underlying reviews searched poorly, used inappropriate eligibility criteria, failed to assess risk of bias adequately, or are several years out of date, those limitations constrain what the umbrella review can conclude.
This is why critical appraisal occurs at the review level. JBI provides a critical appraisal checklist for systematic reviews and research syntheses as part of its umbrella-review methodology.
The umbrella review should also consider the quality or risk-of-bias information relating to the primary studies where relevant and available. In practice, this can be difficult because different reviews may use different appraisal tools, thresholds, and reporting conventions.
Currency Matters More Than the Publication Date Alone
A systematic review published recently may have conducted its final search considerably earlier. Conversely, an older review may still cover a field in which little new evidence has appeared.
When deciding whether review-level evidence is sufficient, examine the search dates and scope of the existing reviews, not merely the year printed on the article. If the reviews do not include important recent primary evidence, an umbrella review based solely on them may provide an outdated picture.
This issue is closely related to deciding whether an existing review is recent enough to affect the justification for another review.
An Umbrella Review Is Not a Meta-Analysis of Meta-Analyses
An umbrella review can report or analyze quantitative results from included reviews, but its defining feature is the systematic synthesis of review-level evidence, not a requirement to combine pooled estimates into another pooled estimate.
Statistically combining meta-analytic results can be methodologically difficult because included reviews may overlap in their primary studies, use different eligibility criteria, analyze different effect measures, or employ different statistical models. Treating their pooled estimates as independent can be inappropriate.
As with primary-study systematic reviews, the synthesis method should follow the question and structure of the evidence rather than the desire to produce one impressive-looking number.
An Umbrella Review Needs Its Own Protocol and Methods
Using published reviews as evidence does not remove the need for prospective methodological planning. JBI umbrella-review guidance covers development of an umbrella-review protocol, inclusion criteria, search strategy, critical appraisal, data extraction, synthesis, and reporting.
For overviews of reviews of healthcare interventions, the PRIOR statement provides dedicated reporting guidance. PRIOR includes 27 main items with additional sub-items intended to improve complete and transparent reporting of overviews.
Reporting guidance should be matched to the type and field of the overview. PRIOR was specifically developed for overviews of reviews of healthcare interventions, so it should not automatically be treated as a universal reporting standard for every form of umbrella review.
