01 · The Question
If You Conduct a Systematic Review, Are You Supposed to Produce a Meta-Analysis?
Systematic reviews and meta-analyses appear together so often that researchers sometimes treat them as two names for the same thing. A journal article may even describe itself as a "systematic review and meta-analysis," which can reinforce the impression that the meta-analysis is the final step every systematic review is expected to reach.
It is not.
A systematic review is the broader process of systematically identifying, selecting, appraising, and synthesizing evidence relevant to a defined question. Meta-analysis is one statistical method that can be used within that process to combine quantitative results from multiple studies.
Sometimes meta-analysis provides a useful summary of the evidence. Sometimes the available studies cannot be combined appropriately. Knowing when not to pool results is therefore part of competent evidence synthesis.
03 · What You Need to Know
What Meta-Analysis Actually Does
Meta-Analysis Combines Quantitative Evidence Statistically
Suppose several studies have estimated the effect of the same type of intervention on a sufficiently comparable outcome. Each study provides an estimate of that effect, but the estimates differ because of sampling variation and potentially because the studies themselves differ.
Meta-analysis uses statistical methods to combine compatible study estimates. In a conventional meta-analysis of effects, studies contribute to an overall estimate according to a weighting scheme, commonly giving more precise estimates greater influence than less precise ones.
The result is generally presented with a measure of uncertainty such as a confidence interval. The analysis may also examine variation among study results and investigate possible reasons for that variation.
A Systematic Review and a Meta-Analysis Answer Different Methodological Questions
The distinction becomes clearer if you separate two questions.
First: Which studies constitute the evidence relevant to this review question? Systematic review methodology addresses this through the review question, eligibility criteria, searching, study selection, data collection, risk-of-bias assessment, and synthesis planning.
Second: Can the numerical results from some or all of those studies be meaningfully combined? Meta-analysis addresses this statistical question.
Systematic review
A structured evidence-synthesis process for answering a defined question through explicit methods for identifying, selecting, evaluating, and synthesizing relevant evidence.
Meta-analysis
A statistical method for combining quantitative results from multiple studies when such combination is appropriate.
This is why the two terms should not be used interchangeably. A systematic review may contain one meta-analysis, several meta-analyses, or no meta-analysis at all.
What Does a Meta-Analysis Calculate?
The exact quantity depends on the outcome and study design. Meta-analysis may combine effect measures such as risk ratios, odds ratios, hazard ratios, mean differences, standardized mean differences, correlations, prevalence estimates, or other compatible statistics.
The individual study estimates first need to be represented on an appropriate common scale for the planned synthesis. Each study then contributes according to the selected meta-analytic model and weighting procedure.
A Forest Plot Is Not the Meta-Analysis Itself
A forest plot is a graphical display commonly used to show individual study estimates, their uncertainty, and, when calculated, a pooled estimate. The familiar diamond often used at the bottom of a forest plot represents a summary estimate and its confidence interval.
But producing a forest plot is not proof that an appropriate meta-analysis has been conducted. Cochrane explicitly cautions against jumping prematurely into statistical analysis before the review question, eligibility criteria, study identification, risk of bias, comparisons, and meaningfulness of the data have been considered.
Why Might Studies Be Too Different to Pool?
Studies rarely need to be identical, but their differences need to be considered before pooling.
Researchers may use the term heterogeneity to describe variation among studies. Clinical or substantive heterogeneity can arise from differences in participants, interventions, exposures, comparators, settings, or outcomes. Methodological heterogeneity can arise from differences in study design, measurement, follow-up, or risk of bias. Statistical heterogeneity concerns variation in observed effects beyond what would be expected from sampling variation alone.
The existence of heterogeneity does not automatically prohibit meta-analysis. Cochrane notes that meta-analysis may still provide important insights in the presence of clinical, methodological, or statistical diversity when appropriate methods and interpretation are used. The relevant question is whether the studies address a sufficiently coherent question for the proposed combined estimate to have a useful meaning.
Sometimes the Data Simply Cannot Support the Planned Meta-Analysis
Even conceptually compatible studies may report results in ways that prevent straightforward pooling. Effect estimates may be incompletely reported, outcomes may be measured on incompatible scales without an appropriate transformation, necessary measures of uncertainty may be unavailable, or only one study may provide data for a prespecified comparison.
Cochrane identifies limited evidence, incomplete reporting, and incompatible effect information among legitimate reasons why meta-analysis of effect estimates may not be possible.
The solution is not to force every study into one calculation. Reviewers should determine whether missing statistics can be derived appropriately, whether alternative compatible measures can be used, whether studies should be synthesized in separate groups, or whether another synthesis method is necessary.
A Random-Effects Model Does Not Make Incompatible Studies Compatible
A common misconception is that substantial differences among studies can be solved simply by choosing a random-effects model. Random-effects meta-analysis allows for variation in underlying study effects, but it does not answer the conceptual question of whether the studies belong in the same synthesis.
If studies address fundamentally different interventions, populations, outcomes, or questions, calculating an average may produce a statistically valid number with little substantive meaning. Statistical sophistication cannot rescue an incoherent comparison.
Not Performing Meta-Analysis Does Not Mean You Stop at a Descriptive Summary
When meta-analysis is inappropriate or impossible, the alternative should still be a structured synthesis. Cochrane provides guidance on methods for synthesizing and presenting findings when meta-analysis of effect estimates cannot be undertaken.
For systematic reviews of intervention effects, the SWiM reporting guideline provides nine reporting items for synthesis without meta-analysis. It addresses issues such as grouping studies, describing the synthesis method, presenting data, and explaining limitations.
This is more demanding than writing several paragraphs describing whichever studies appear most interesting. A synthesis without meta-analysis still needs a transparent connection between the included data, the method of synthesis, and the conclusions.
Watch Out
Avoid "vote counting" studies according to whether their individual results are statistically significant. Cochrane identifies vote counting based on statistical significance as an unacceptable synthesis method because significance depends partly on sample size and does not reliably represent the direction or magnitude of the underlying effects.
Meta-Analysis Can Increase Precision, but It Does Not Repair Poor Evidence
Combining studies can produce a more precise estimate than individual studies provide. That precision can be valuable when the studies address a coherent question and the synthesis is methodologically sound.
However, a narrow confidence interval around a pooled estimate does not guarantee that the underlying evidence is trustworthy. Bias in primary studies, selective reporting, publication bias, inappropriate eligibility decisions, or flawed analysis can still undermine the conclusion.
Meta-analysis aggregates evidence. It does not automatically improve the quality of the evidence being aggregated.
| Situation |
Meta-Analysis? |
Why? |
| Several sufficiently comparable studies report compatible effect estimates |
Potentially appropriate |
A pooled estimate may provide a meaningful quantitative synthesis |
| Studies use different but appropriately transformable measures of the same construct |
Potentially appropriate |
A suitable common effect measure may permit synthesis |
| Only one study contributes to a prespecified comparison |
Not possible for that comparison |
There are no multiple study estimates to combine |
| Necessary effect information is incompletely reported and cannot be derived |
May not be possible |
The required quantitative inputs are unavailable |
| Studies address fundamentally different questions |
Usually inappropriate to combine them in one pooled estimate |
The resulting average may have little meaningful interpretation |
| Quantitative synthesis would obscure important differences among studies |
May be inappropriate |
Another synthesis method may better represent the evidence |
