03 · What You Need to Know
Comprehensiveness has value, but it also has a cost
Why search for unpublished research in the first place?
The strongest argument is not simply that more studies are better. It is that the published literature may not be a neutral sample of all the research conducted.
If the probability of publication depends on the direction, magnitude, statistical significance, or perceived importance of results, a review restricted to conventional publications can become systematically distorted. Publication bias can make interventions appear more effective, findings more consistent, or uncertainty smaller than the complete evidence base would support.
Searching for unpublished research is one way of reducing dependence on that publication process.
Cochrane therefore recommends comprehensive approaches to study identification for intervention reviews and regards contact with relevant individuals and organizations about unpublished or ongoing studies as highly desirable. Trial registers, results repositories, regulatory sources, clinical study reports, conference sources, and other supplementary routes can reveal studies absent from ordinary bibliographic databases.
But there is no single reliable way to find all unpublished studies
This limitation changes the decision considerably.
Cochrane notes that there is no easy and reliable single method for obtaining information about every completed or terminated study that was never published. Registration systems have improved discoverability in some fields, particularly clinical research, but coverage remains uneven across disciplines, time periods, countries, and study designs.
An unpublished study might leave a trace in:
- a trial or study registry;
- a conference abstract;
- a dissertation or thesis;
- a research report;
- a preprint or working paper;
- a regulatory record;
- a funding database;
- an institutional repository;
- a researcher's curriculum vitae or webpage;
- a reference list;
- correspondence with investigators.
Or it may leave no discoverable public trace at all.
Searching more extensively therefore reduces uncertainty rather than guaranteeing completeness.
The effort is not merely the time required to type more searches
Unpublished-research searching can be resource intensive because discovery is only the first step.
You may need to:
- adapt searches to several poorly standardized interfaces;
- screen large numbers of weakly relevant records;
- match registry records to later publications;
- distinguish several reports of the same study;
- locate archived conference material;
- identify investigators whose contact information has changed;
- send and track correspondence;
- wait for responses;
- translate or interpret incomplete records;
- request data or reports;
- decide whether incomplete information establishes eligibility;
- critically appraise material with limited methodological reporting.
The relevant resource is therefore researcher time, information-specialist time, review-management complexity, and sometimes direct financial cost.
A search can be methodologically admirable and still be inefficient if it consumes a disproportionate share of the review's resources for negligible likely gain.
Evidence that unpublished searching can find additional material is substantial
An audit of 203 systematic reviews published in major medical journals and the Cochrane Database found that 130 reported searching for unpublished or grey-literature data. Among those reviews, 116, or 89%, found such data.
That finding demonstrates that supplementary searching is not merely theoretical. Unpublished or grey information can often be found when reviewers look for it.
But discovery did not automatically translate into inclusion. Only 46 of those 116 reviews that found unpublished material included it in their analyses. This distinction matters because the value of a search depends not simply on whether something is found but on whether it supplies eligible, usable evidence capable of affecting the review.
Finding unpublished material is not the same as improving the review
Suppose you spend several days searching and identify ten additional records. That sounds productive. But imagine that six are duplicate reports of already known studies, two are ineligible, one provides too little information to establish eligibility, and one is an ongoing study with no results.
The search was successful in discovery terms but added no new result to the synthesis.
Conversely, one unpublished study can be extremely valuable if it is large, methodologically relevant, and materially changes an otherwise sparse evidence base.
Search yield
How many additional records, studies, reports, or data sources the supplementary search identifies.
Decision value
Whether the additional evidence meaningfully changes the estimate, interpretation, certainty, comprehensiveness, or understanding of the evidence base.
A high-yield search is not necessarily high value, and a low-yield search is not necessarily low value.
The smaller the evidence base, the more one missing study may matter
Imagine a meta-analysis containing three small published studies. A fourth reasonably large unpublished study could materially alter the pooled estimate, confidence interval, heterogeneity, or overall interpretation.
Now imagine a mature evidence base containing 150 large studies with broadly consistent findings. One additional small unpublished study may have much less influence, although this cannot be assumed in advance.
Cochrane's discussion of research on unpublished studies notes that their inclusion may be particularly consequential when few relevant studies exist or when vested interests raise concerns about the representativeness of the published literature.
This suggests a useful principle: the potential marginal value of finding one additional study generally rises when the existing evidence base is sparse, uncertain, or vulnerable to selective dissemination.
Search more aggressively when publication bias is especially plausible
Some evidence bases deserve greater suspicion of selective availability than others.
More extensive searching may be justified when:
- the topic involves commercially sponsored interventions;
- the published literature consists disproportionately of favorable findings;
- registries indicate completed studies without publications;
- conference abstracts identify studies that never progressed to full reports;
- published papers refer to unpublished studies or data;
- previous reviews report difficulty obtaining complete evidence;
- the field has known concerns about selective reporting;
- the conclusions depend heavily on a small number of studies.
None of these proves that missing evidence exists or would change the result. They increase the plausibility that additional searching could be consequential.
Commercial or other vested interests can increase the value of supplementary searching
Funding source should not be treated as a shortcut for declaring research biased. Nevertheless, when organizations have substantial financial, regulatory, institutional, or political interests in study outcomes, the completeness of public reporting becomes especially relevant.
Cochrane notes that searching for unpublished evidence may have greater impact where there are questionable vested interests in the published literature. Regulatory sources and clinical study reports can be particularly valuable for some pharmaceutical and medical-device questions because they may contain evidence not fully represented in journal publications.
The same logic can extend beyond industry. Governments, institutions, advocacy organizations, and other actors can also have interests in how evaluations are disseminated.
The response should be greater attention to transparency and completeness, not automatic distrust based solely on who funded the research.
Search intensity should increase with the stakes of the decision
A literature review written to provide general background for an undergraduate essay does not necessarily require the same search effort as a systematic review informing clinical guidelines, public policy, regulatory decisions, or major institutional investments.
When an evidence synthesis will influence consequential decisions, the cost of reaching the wrong conclusion because relevant evidence was missed becomes larger.
This changes the cost-benefit calculation.
Low-stakes descriptive use
The review primarily provides orientation, conceptual background, or a bounded account of a literature.
High-stakes decision use
The synthesis may directly influence clinical, policy, regulatory, funding, or other consequential decisions where missing evidence could cause substantial harm or misallocation.
The second situation generally warrants stronger efforts to understand whether the visible literature is complete.
The type of review matters
Different review designs make different claims about comprehensiveness.
A systematic review typically attempts to identify all studies meeting predefined eligibility criteria. Searching for relevant unpublished or ongoing evidence may therefore be integral to minimizing bias.
A scoping review may also search extensively when its purpose is to map the complete range of available evidence, including non-journal sources.
A narrative review may legitimately adopt a more selective approach, depending on its question and stated methodology. A review explicitly examining peer-reviewed journal discourse might exclude unpublished material by design.
The more comprehensive the claim your review makes, the stronger the obligation to investigate evidence that conventional publication may have missed.
Not every grey-literature search is a search for unpublished research
This distinction helps keep the search proportional.
Grey literature includes many documents that are publicly available but sit outside conventional academic or commercial publishing, such as government reports and dissertations. Unpublished research is a narrower category involving studies or results that have not been formally disseminated as conventional research outputs.
If your question requires government reports because government agencies are the primary producers of the evidence, searching those reports is not merely an attempt to combat publication bias. It is necessary to find the relevant literature at all.
By contrast, contacting investigators about completed but unreported studies is a more targeted attempt to recover missing research.
The effort required and expected value differ substantially.
Start with high-yield, structured sources before labor-intensive detective work
Searching for unpublished research does not need to begin with emailing hundreds of authors.
A sensible sequence is often to start with structured sources that can reveal multiple studies efficiently, then pursue individual studies when they appear important.
Structured sources Search relevant trial or study registries, results repositories, dissertation databases, conference collections, preprint servers, regulatory sources, and other topic-appropriate resources.
Study tracing Use registration numbers, study acronyms, investigators, sponsors, institutions, and distinctive study characteristics to connect records with publications and other reports.
Targeted searching Search organizational websites, repositories, funding records, and the open web for unresolved studies or likely evidence producers.
Direct contact Contact investigators, sponsors, organizations, or experts when a potentially important study remains unresolved and additional information could affect the review.
This layered approach concentrates the most labor-intensive effort where it has the greatest potential value.
Author contact can be useful, but the return is uncertain
Cochrane describes correspondence as a potentially important source of information about unpublished studies, while also noting that attempts to obtain such information have not always been fruitful.
Researchers may not respond. Contact details may be outdated. Data may no longer be accessible. Investigators may confirm that a study existed but be unable to provide usable results.
Author contact is therefore most defensible when the target is specific and consequential.
For example, contacting an investigator may be worthwhile when a registry identifies a large completed study that appears eligible but has no results. Sending broad requests to every author in a large mature literature may offer a much lower expected return.
Use clues from the published literature to target the search
The published evidence itself can indicate where additional searching is likely to be productive.
Look for:
- trial registration numbers;
- references to unpublished observations;
- conference presentations;
- study protocols without corresponding results papers;
- funded projects with no obvious outputs;
- multiple publications from research programs that imply additional studies;
- statements such as "data not shown";
- citations to dissertations, reports, or working papers.
Each is a lead rather than proof of missing evidence.
Once a specific study is identified, the methods for tracing unpublished and ongoing research become much more efficient than continuing to search the entire topic indiscriminately.
Ask whether the missing evidence is likely to be usable
Some fields leave rich traces of unpublished research. Others do not.
A registered clinical trial may provide detailed eligibility criteria, outcomes, recruitment status, sponsor information, and sometimes structured results. A decades-old unregistered laboratory experiment mentioned only in a narrative review may be almost impossible to reconstruct.
Before committing substantial resources, consider:
- whether the field uses study registration;
- whether repositories or regulatory sources exist;
- whether investigators can still be identified;
- whether study-level data are likely to survive;
- whether enough methodological information would exist to establish eligibility and appraise the study.
A theoretically important study that cannot be identified, retrieved, or evaluated may have low practical recoverability even if you strongly suspect that it exists.
Finding an unpublished study does not guarantee that it belongs in the analysis
The same eligibility criteria should govern published and unpublished evidence.
Do not include a poorly documented study merely because considerable effort went into finding it. That would allow sunk cost to become an eligibility criterion, which is rarely listed in the protocol.
Once identified, ask:
- Does the study meet the predefined inclusion criteria?
- Can eligibility be established confidently?
- Are usable results available?
- Can the methods be appraised?
- Is this an independent study or another report of an already included study?
An unpublished study may still matter even when its results cannot be synthesized because its existence can indicate that the available evidence is incomplete.
Sometimes updating the review may be more valuable than searching indefinitely
There is genuine methodological debate about how much effort should be invested in locating unpublished evidence.
A study examining unpublished trials in Cochrane reviews argued that many identified "unpublished" trials were eventually published and that truly unpublished studies often had poor or unclear methodological quality. The authors suggested that regular review updating might sometimes be a better investment than extensive searching for unpublished data.
This conclusion should not be generalized into a rule against unpublished searching. It represents evidence from a particular context and has to be weighed against other research showing that supplementary searches frequently identify unpublished material and that missing evidence can bias effect estimates.
The useful lesson is that search effort has an opportunity cost. Time spent pursuing a marginally informative study is time unavailable for updating searches, improving screening, resolving duplicate reports, checking extraction, appraising bias, or strengthening the synthesis.
Set stopping rules before the search becomes endless
Unlike a bibliographic database search, open-web and investigator searches do not have an obvious endpoint.
You could always:
- search another website;
- try another query;
- contact another expert;
- wait another week for a response;
- look through another conference archive;
- follow another citation trail.
For a structured review, define reasonable boundaries in advance.
These might include:
- which registries will be searched;
- which conference years will be covered;
- which organizations or regulatory sources are relevant;
- which unresolved studies warrant author contact;
- how many contact attempts will be made;
- how long you will wait for responses;
- how web-search results will be screened;
- when unresolved studies will be rechecked.
A stopping rule does not mean the search is perfectly complete. It means the effort is transparent and reproducible enough to be defended.
Consider the expected impact on the synthesis
One way to think about proportionality is to ask what would happen if an unpublished study were found.
Would it:
- add the first evidence for an important subgroup?
- substantially increase the sample size?
- provide contradictory findings?
- alter an effect estimate?
- change the direction of a conclusion?
- affect certainty or risk-of-bias judgments?
- show that a supposedly empty research gap is already being studied?
If the answer could plausibly be yes, further searching has greater expected value.
If even a reasonably large additional study would be unlikely to change any substantive conclusion, the marginal value may be lower, although documenting the completeness of the evidence can still matter.
Do not decide whether to search based on whether you like the published answer
This sounds obvious, but it is an important protection against researcher discretion.
You should not search aggressively for unpublished evidence only because the published studies produce an inconvenient result. Nor should you stop searching because the existing evidence already supports the conclusion you expected.
For systematic reviews, supplementary search plans should be specified prospectively whenever possible. The decision should be driven by the review question, likely dissemination patterns, risk of missing evidence, and methodological standards rather than the emerging direction of the results.
Watch Out
Do not use unpublished-research searching as a post hoc hunt for studies that will move the synthesis in a preferred direction. The purpose is to reduce selective evidence availability, not to introduce a new form of selective evidence retrieval.
Searching can be worthwhile even when it finds nothing
A well-designed supplementary search that identifies no additional eligible studies is not necessarily wasted effort.
It can strengthen the methodological basis for saying that the review made reasonable efforts to identify evidence beyond conventional publication. It may also reveal ongoing studies, confirm that suspected unpublished records correspond to already included publications, or show that particular sources are unlikely to contain additional eligible research.
But "we found nothing" should not become "nothing exists." No unpublished search can establish that with certainty.
Document the search well enough for readers to judge it
Report:
- which sources were searched;
- why those sources were selected;
- search dates;
- relevant search terms or approaches;
- which organizations or individuals were contacted;
- how unresolved studies were handled;
- important limitations in coverage.
If resources forced you to limit the search, say so. Transparency about a bounded search is more useful than describing it vaguely as "comprehensive."
The goal is sufficient comprehensiveness, not methodological infinity
There is an understandable temptation in systematic reviewing to equate rigor with doing everything imaginable. That is not quite the same thing.
A rigorous search is aligned with the review's purpose, sensitive to plausible sources of bias, appropriately resourced, transparent about its boundaries, and proportionate to the consequences of missing evidence.
For some reviews, this means extensive registry, regulatory, grey-literature, and investigator searching. For others, a more targeted supplementary strategy may be defensible.
The relevant question is not "Have I searched everywhere?" No researcher can answer yes. It is "Have I searched far enough that the remaining risk of missing evidence is acceptable for the claims this review intends to make?"
