Why Can Ignoring Grey Literature Bias What You Think the Evidence Says?

Can leaving out grey literature actually change your conclusion?

Imagine searching several major databases and finding a remarkably consistent pattern: most published studies suggest that an intervention works. It is tempting to conclude that the evidence itself is remarkably consistent.

But there is another possibility. Studies with less favorable, inconclusive, or statistically non-significant findings may be less likely to reach conventional publication, may take longer to appear, or may remain available only through reports, dissertations, conference materials, registries, or other sources outside mainstream journals. If your search captures mostly the studies that successfully entered the published literature, the apparent pattern may partly reflect which results became visible.

This is why ignoring grey literature can matter. The central problem is not that every review is missing a few obscure documents. It is that the evidence you find may differ systematically from the evidence you miss.

Missing evidence can create a systematically distorted picture

The real problem is selective visibility of evidence

Missing studies do not automatically create bias

A literature review is almost never a perfect census of everything ever investigated. Some relevant documents will be difficult to locate, inaccessible, poorly indexed, or unknown to the reviewer. Missing evidence becomes especially concerning when what is missing differs systematically from what is available.

Suppose ten studies investigate the same intervention. If two studies are missing for essentially random reasons, the review is incomplete, but the missingness does not necessarily push the conclusion in a particular direction. If the two missing studies are absent because they found little or no evidence of benefit, however, the eight visible studies provide a systematically unrepresentative picture.

This distinction is fundamental. A review can contain hundreds of studies and still be biased if the process determining which studies became visible was selective.

Why might published studies differ from less visible research?

Research findings do not all have the same probability of becoming full journal articles. Decisions about submission, acceptance, reporting, timing, and dissemination can be influenced by the direction, magnitude, or statistical significance of results.

The Cochrane Handbook describes this broader problem as non-reporting bias: bias that can occur when decisions about how, when, or where results are reported are influenced by their findings. Statistically significant results suggesting that an intervention works, for example, may be more readily available, appear sooner, reach higher-profile journals, and subsequently become easier for reviewers to identify.

Researchers may also decide that inconclusive findings are not worth submitting. Sponsors may not pursue publication. Editors and reviewers may find novel or statistically significant results more attractive. A conference presentation may never become a full paper. A completed thesis may remain in an institutional repository. None of these mechanisms applies equally in every discipline or to every study, but collectively they create a plausible route through which the conventional published literature can become selective.

How can excluding grey literature distort an effect estimate?

Methodological research has repeatedly examined whether published and grey-literature studies produce different estimates.

A Cochrane methodological review identified five studies comparing meta-analyses with and without grey literature from randomized healthcare trials. Across all five, published trials showed greater treatment effects than grey trials. When data from three studies were combined, published trials showed an average treatment effect approximately 9% greater than grey trials.

Another methodological study examining 41 meta-analyses found that published studies produced intervention-effect estimates approximately 15% larger, on average, than grey literature. Earlier work reviewing the issue similarly concluded that statistically significant findings were more common among published research than among grey literature.

These findings illustrate the potential direction of bias, but they should not be converted into a universal correction factor. You cannot assume that excluding grey literature inflates every effect by 9%, 15%, or any other fixed amount. The size and even the practical importance of the difference depend on the field, evidence base, study designs, dissemination practices, and particular review question.

The effect of adding grey literature is not always dramatic

The relationship is more complicated than "published studies exaggerate effects and grey literature fixes them."

A later systematic review of methodological research examined seven projects covering 187 meta-analyses. Some found larger pooled effects among published evidence, but in several projects adding unpublished or grey-literature data did not significantly change pooled estimates or overall interpretations. Including additional data could nevertheless increase precision.

This matters because grey-literature searching often requires substantial time and effort. Its value is therefore context-dependent. The possibility of bias provides a reason to consider grey literature seriously, particularly in comprehensive evidence syntheses, but it does not establish that every additional grey source will change the answer.

Grey literature and publication bias are related but not identical

It is easy to collapse several ideas into one. Grey literature describes material outside conventional academic or commercial publishing channels. Publication bias concerns selective dissemination associated with study findings.

Some grey literature may contain research that did not become conventionally published, making it useful when investigating possible publication bias. But not all grey literature represents suppressed, unfavorable, or statistically non-significant findings. Government reports, theses, working papers, and conference materials may exist outside journals for many reasons unrelated to their results.

Searching grey literature is therefore one strategy for broadening the evidence base. It is not a direct test for publication bias and should not be presented as one.

The distortion can involve whole studies or individual results

Sometimes an entire study disappears from the accessible evidence base. A completed trial may never become a journal article, for example. In other cases, the study itself is published but particular outcomes or analyses are not reported.

Cochrane distinguishes these forms of missing evidence because both can bias a synthesis. If a study measured several outcomes but selectively reports only favorable ones, finding the journal article does not solve the problem. The article itself may provide an incomplete representation of what was measured.

Protocols, registrations, conference abstracts, regulatory records, theses, reports, and other materials can sometimes reveal that additional studies, outcomes, analyses, or earlier versions existed. That is one reason searching beyond journal articles can be valuable even when a reviewer has already located many published studies.

Selective publication can change more than the pooled effect

The concern is not limited to whether a meta-analysis produces a slightly larger numerical estimate. Selective availability can affect several features of the apparent evidence base.

• The intervention or association may appear more consistently favorable than it really is.
• Harms or null findings may be underrepresented.
• Uncertainty around an effect may be misunderstood when relevant studies are absent.
• The apparent balance between supportive and contradictory findings may change.
• Conclusions about the certainty of evidence may need reconsideration when missing evidence is suspected.

For evidence synthesis, this is consequential because a review does more than count publications. It tries to infer something about the underlying body of research. If the observable literature is a selective sample of that body, the inference can be wrong even when every included article has been extracted perfectly.

A database search cannot retrieve research that was never indexed there

Searching more journal databases can improve coverage of published literature, but it does not necessarily solve selective dissemination. A dissertation sitting in a university repository, an unpublished trial recorded in a registry, or an institutional evaluation available only from an organization will not suddenly appear merely because another conventional bibliographic database is added.

Depending on the question, reviewers may therefore need to search specifically for theses, dissertations, reports, and working papers, examine study registries, search conference materials, inspect reference lists, or contact investigators.

The search strategy should follow plausible dissemination routes in the field rather than an assumption that one source captures everything.

Conference abstracts can reveal research that never reached full publication

Conference records are particularly interesting because they can expose an earlier stage of the research-publication pathway. A study may be presented as an abstract and subsequently become a full journal article, but this progression is not guaranteed.

For a review, an abstract may therefore signal that a study exists even when sufficient information for inclusion is not yet available. The decision about whether to include conference abstracts as evidence is separate from their value for study identification.

If an apparently eligible abstract never progresses to a full report, that absence may warrant further investigation rather than simply treating the study as though it never existed.

Searching grey literature does not eliminate publication bias

This qualification is essential. Grey literature is itself incomplete and selectively available. Some unpublished studies may be impossible to discover. Researchers who respond to requests for unpublished data may not represent those who do not. Institutional repositories vary in coverage. Websites disappear. Conference records can be fragmentary. Search engines rank rather than comprehensively enumerate web content.

There is also no guarantee that grey literature has findings distributed exactly like the total body of missing research. A search can therefore retrieve a selective subset of the already selective unpublished evidence.

Cochrane accordingly treats comprehensive searching and attempts to obtain unpublished results as ways to minimize risk from missing evidence, not as guarantees that the problem has been removed. Review authors may still need to assess explicitly the risk of bias due to missing evidence.

Finding more evidence and evaluating it are different tasks

A desire to reduce publication bias does not justify accepting every grey-literature source uncritically.

Once potentially eligible evidence is found, researchers still need to determine whether it meets the review criteria and whether its methods support the claims being made. A non-peer-reviewed report can contain rigorous research, but it can also omit essential methodological information. The same is true, in different ways, of conference abstracts, dissertations, preprints, and working papers.

Where a report is eligible, researchers should evaluate non-peer-reviewed evidence directly rather than assuming either that it is unreliable because it is grey or trustworthy because including grey literature sounds methodologically comprehensive.

How a published-only search can tell a different story

Common mistakes when thinking about grey literature and bias

Ask whether missing evidence could change the inference

The practical decision is not whether grey literature is universally mandatory. It is whether restricting your evidence base to conventional publications creates a meaningful risk for the question you are answering.

For quantitative syntheses, do not stop at search design. Consider whether the included evidence shows signs that results may be missing and whether the certainty of the evidence should reflect that concern. Statistical methods can contribute to this assessment, but they should not be treated as mechanical corrections for an unknown body of missing studies.

Above all, avoid framing the issue as a competition between "good published research" and "bad grey literature." The methodological question is whether your accessible evidence represents the body of research you are trying to understand.

Check whether your review is vulnerable to missing-evidence bias

Frequently asked questions about grey literature and bias

A published literature can be extensive and still be unrepresentative

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.