01 · The Question
Can You Skip the Systematic Review and Go Straight to the Statistics?
You have found several studies reporting comparable quantitative results. Perhaps they all estimate the effect of the same intervention, report correlations between the same variables, or examine a similar outcome. The effect sizes can be extracted, statistical software can combine them, and a forest plot can be produced.
Technically, you can perform that calculation. But that is not the same as establishing that the calculation represents the relevant body of evidence.
Meta-analysis answers a statistical question: how should results from multiple studies be combined? A systematic review addresses an earlier and equally consequential question: which studies should be in that analysis in the first place?
If the studies were selected unsystematically, even impeccable statistics may produce a precise summary of a biased or incomplete evidence base.
03 · What You Need to Know
Why Meta-Analysis Usually Sits Inside a Systematic Review
Meta-Analysis Begins After an Evidence Base Has Been Defined
A meta-analysis statistically combines quantitative results from multiple studies. Depending on the question, those results might be risk ratios, odds ratios, mean differences, standardized mean differences, correlations, prevalence estimates, or another compatible measure.
Before any of those numbers can be combined, however, researchers must decide which studies are eligible. That decision depends on the review question, population, interventions or exposures, comparators, outcomes, study designs, and other eligibility criteria.
A systematic review provides a structured process for making those decisions, searching for eligible evidence, selecting studies, evaluating them, and synthesizing the resulting evidence. Meta-analysis can then be used when quantitative pooling is appropriate.
This is why meta-analysis and systematic review should not be treated as synonyms. One concerns statistical synthesis; the other establishes and evaluates the evidence base within which that synthesis usually occurs.
The Main Problem Is Selection, Not the Arithmetic
Suppose 20 eligible studies exist. You happen to know about eight of them and successfully extract their effect estimates. Statistical software can calculate a pooled effect from those eight studies without knowing that another 12 exist.
The mathematics may be executed correctly. The scientific inference may still be misleading.
If the studies you happened to locate differ systematically from the studies you missed, the pooled estimate may not represent the evidence relevant to the question. Easily discovered studies may differ from difficult-to-find studies. Published evidence may differ from unpublished evidence. Studies with striking findings may be more visible and memorable than studies reporting small or null effects.
No meta-analytic model can reconstruct relevant studies that were never considered for inclusion.
Statistical validity of the calculation
Were the selected effect estimates combined using an appropriate statistical model and methods?
Validity of the evidence base
Were the studies entering that calculation identified and selected through a defensible process capable of representing the evidence relevant to the question?
A credible meta-analysis needs both considerations. Sophisticated statistics address only the first.
A Convenience Sample of Studies Is Particularly Problematic
Imagine selecting studies because they appear on the first pages of a database search, were cited in a familiar paper, are available through your institution, or are already stored in your reference manager. You may still be able to calculate effect sizes from them.
What you cannot confidently claim is that those studies constitute the relevant evidence base.
The problem becomes more consequential when the pooled estimate is presented as answering a substantive question such as whether an intervention works, how strongly two variables are associated, or how common a condition is. Those claims concern a body of evidence, not merely the subset of papers that happened to be convenient.
Watch Out
A forest plot does not demonstrate that the studies entering it were identified systematically. Always ask where the studies came from, what made them eligible, what potentially eligible evidence was excluded, and whether the selection process could have influenced the pooled result.
Using an Existing Systematic Review Can Be Different From Skipping Systematic Methods Entirely
There are circumstances in which researchers may conduct a new meta-analysis without repeating an entire systematic review from the beginning.
For example, a high-quality systematic review may already have established an eligible evidence base. Researchers might conduct a secondary methodological analysis using those studies, recalculate effect estimates using a different justified statistical approach, or extend an analysis to address a methodological question.
In that situation, the new project may not need to reproduce every search and screening procedure itself. But the evidence base did not appear unsystematically. It came from a prior systematic process whose methods, eligibility criteria, search dates, and limitations should be acknowledged.
Researchers also need to determine whether the evidence base remains current. If important eligible studies have appeared since the earlier review's final search, relying on the old set without updating it may no longer answer the intended question adequately.
Prospectively Defined Collaborative Meta-Analyses Are Another Important Case
Not every meta-analysis begins with published studies discovered through a conventional literature search. Some collaborative meta-analyses are organized prospectively, with eligible studies or research groups identified according to a predefined protocol before results are known or before all participating studies have been completed.
Individual participant data meta-analyses can also involve obtaining and reanalyzing participant-level data from multiple studies rather than relying only on aggregate statistics reported in publications.
These designs do not make systematic thinking unnecessary. Eligibility criteria, identification of eligible studies, data availability, exclusions, analysis decisions, and potential sources of selection bias still require transparent treatment. The route by which studies enter the synthesis may differ, but the need for a defensible evidence base remains.
Meta-Analysis of a Non-Systematic Literature Sample Answers a Narrower Question
Sometimes researchers deliberately want to analyze a defined collection of studies rather than make claims about all evidence relevant to a substantive research question. A methodological researcher, for example, might select a particular corpus to investigate statistical properties or compare analytic methods.
A meta-analysis of that corpus can be legitimate if the research question is explicitly about that corpus or if the sampling logic otherwise supports the intended inference.
The problem arises when conclusions silently expand beyond the sampling frame. "Among these selected studies, the pooled estimate was X" is a different claim from "the evidence shows that the true effect is X."
Systematic Searching Does Not Guarantee an Unbiased Meta-Analysis
The relationship also works in the other direction. Conducting a systematic search does not guarantee that the eventual meta-analysis is trustworthy.
Reviewers can make inappropriate eligibility decisions, extract data incorrectly, select unsuitable effect measures, combine incompatible studies, ignore dependencies among effect estimates, use inappropriate models, conduct data-driven subgroup analyses, or interpret statistical significance as substantive importance.
A systematic review establishes a defensible route to the evidence. Meta-analysis then introduces its own methodological decisions. Both stages require scrutiny.
Publication Bias Remains a Problem Even With Systematic Methods
A systematic review can search comprehensively for accessible evidence, but it cannot guarantee that every study ever conducted is discoverable. Studies may remain unpublished, outcomes may be selectively reported, and publication processes may favor particular kinds of findings.
Methods for examining possible reporting biases can be useful in appropriate circumstances, but no statistical diagnostic magically recovers all missing evidence. A systematic approach reduces avoidable selection problems created by the reviewers themselves; it does not eliminate every bias in the research ecosystem.
Do Not Work Backward From a Desired Meta-Analysis
A particularly risky workflow begins with the statement, "I want to do a meta-analysis," followed by searching for a topic that supplies enough numerical studies.
The research question should come first. The evidence-synthesis design follows from that question. Only after relevant studies have been identified should researchers determine whether statistical pooling is appropriate.
Sometimes the correct outcome of a systematic review is that no meta-analysis should be performed. Studies may be too different, data may be unavailable, or only one study may contribute to a particular comparison. That is a methodological finding, not a failed review.
| Situation |
Can Meta-Analysis Be Defensible? |
Key Consideration |
| Studies identified through a new systematic review |
Yes, when statistical pooling is appropriate |
The evidence base and synthesis are developed together |
| Studies taken from an existing high-quality systematic review |
Potentially |
Verify eligibility, search dates, completeness, and whether updating is necessary |
| Prospectively defined collaborative set of eligible studies |
Potentially |
The inclusion process and analysis protocol must be explicit and defensible |
| Defined corpus used for a methodological research question |
Potentially |
Conclusions must remain within the inference supported by that corpus |
| Studies chosen because they are familiar or easy to obtain |
Poor basis for evidence synthesis |
Convenience selection may substantially distort the pooled result |
| Studies selected after seeing which results support a preferred conclusion |
No defensible evidentiary basis |
Outcome-driven selection fundamentally compromises the synthesis |
