03 · What You Need to Know
Peer Review, Publication Status, and Research Quality Are Different Things
A study is not the same thing as the article reporting it
This distinction is fundamental to systematic searching.
A study is the underlying research investigation. A report is a document or record that provides information about that study.
One study may generate several reports:
- a conference abstract presenting preliminary findings;
- a trial registration record;
- a dissertation or thesis;
- a preprint;
- a peer-reviewed journal article;
- a later follow-up publication;
- a report submitted to a funder or regulatory body.
PRISMA 2020 explicitly defines a report broadly enough to include journal articles, preprints, conference abstracts, study register entries, clinical study reports, dissertations, unpublished manuscripts, government reports, and other documents containing information about a study.
Cochrane similarly emphasizes that systematic reviews are interested in studies rather than merely journal articles. Multiple reports may need to be linked so that information from the same study is not mistaken for evidence from several independent studies.
Study
The underlying investigation that generated the evidence.
Report
One document or record containing information about that study, which may or may not be a peer-reviewed journal article.
If you search only peer-reviewed journal articles, you are making a decision about report type before you necessarily know which studies those reports represent.
Peer review is a publication process, not a guarantee of correctness
Peer review is an important part of scholarly communication, but its role should not be overstated.
A peer-reviewed article has passed through the editorial and review procedures of a journal. That does not mean the study is automatically methodologically sound, unbiased, correctly analyzed, or appropriate for your research question.
Conversely, a study reported outside a peer-reviewed journal is not automatically poor research.
A dissertation may contain extensive methodological detail. A trial registry can reveal outcomes that were pre-specified before results were known. A clinical study report may contain information omitted from a journal article. A government technical report may be the primary source for a policy evaluation.
Publication venue and methodological quality therefore need separate evaluation.
Peer-review status
Describes whether a report underwent a particular editorial review process before publication.
Methodological quality or risk of bias
Concerns how the study was designed, conducted, analyzed, and reported in relation to the question being evaluated.
Using “peer reviewed” as a proxy for “high quality” collapses these two judgments into one.
What counts as grey literature?
The term grey literature is commonly used for material produced outside traditional commercial or academic journal publishing channels. The boundaries of the category vary somewhat across disciplines and organizations.
Cochrane specifically points review authors toward sources such as reports, dissertations, theses, and conference abstracts when discussing grey literature searching. JBI likewise recommends considering both published and unpublished literature and identifies sources including theses, conference proceedings, government reports, organizational websites, and other forms of grey literature.
| Source type |
What it may contribute |
Potential limitation |
| Dissertations and theses |
Detailed methods, analyses, null findings, and studies that never became journal articles |
May be difficult to locate or may later overlap with journal publications |
| Conference abstracts or proceedings |
Early evidence and studies that may never receive full publication |
Often provide limited methodological and outcome detail |
| Trial or study registers |
Planned methods, outcomes, recruitment status, and sometimes results |
Entries may be incomplete or not kept current |
| Government and organizational reports |
Evaluations, surveillance, policy research, and evidence not intended for journal publication |
Reporting standards and methodological detail vary |
| Preprints |
Recent research available before formal journal publication |
May change substantially after review or may never be formally published |
| Unpublished manuscripts or data |
Evidence unavailable through conventional publication channels |
Can be difficult to identify, obtain, verify, and appraise |
The fact that these sources require careful appraisal is not an argument for pretending they do not exist.
Restricting by publication status can contribute to publication bias
One of the strongest reasons to search beyond peer-reviewed journal literature is the possibility of publication bias.
Studies do not all have the same probability of being published, and publication decisions may be associated with the nature or direction of the findings. If studies with statistically significant, favorable, or otherwise noteworthy results are more likely to appear in journals, then searching only published journal literature can produce an evidence base that differs systematically from all the research that was actually conducted.
Cochrane therefore recommends that searches for systematic reviews be as extensive as possible to reduce the risk of publication bias and identifies searching relevant grey literature sources as highly desirable. Its eligibility guidance states that studies should be included irrespective of publication status unless exclusion is explicitly justified.
Watch Out
Restricting a search to peer-reviewed journal articles does more than remove inconvenient document types. It can remove studies according to whether, where, and how their findings entered the formal publication system, which may be related to the results themselves.
Published and unpublished are not a simple binary
The terminology can become confusing because “published” may mean different things.
A dissertation is publicly available but not necessarily published in a journal. A conference abstract may appear in a proceedings supplement. A preprint is publicly disseminated but has not completed journal peer review. A clinical trial result may be publicly posted in a registry without appearing in a conventional article.
Rather than relying on a crude published-versus-unpublished distinction, ask what type of report you have, what information it contains, whether it represents a unique study, and how it fits your eligibility criteria.
Conference abstracts can identify studies even when they are insufficient for final analysis
Conference abstracts deserve particular care.
An abstract may contain too little information to evaluate methods fully or extract the data needed for synthesis. That does not make it useless for searching.
It can alert you to the existence of a study.
You can then search for a later full publication, contact authors where appropriate, inspect a trial registry, or identify another report from the same investigation.
Cochrane's technical guidance treats conference abstracts and proceedings as potentially useful sources of studies. This reinforces the distinction between using a source to identify research and deciding whether the information available from that source is sufficient for inclusion or analysis.
Preprints create a similar distinction between discovery and appraisal
Preprints have become increasingly visible in several research fields. They can make findings available rapidly, sometimes months before journal publication.
Their advantage is timeliness. Their obvious limitation is that the version you retrieve may not yet have undergone formal journal peer review and may later be revised, corrected, withdrawn, or published in a different form.
If preprints fall within the eligibility criteria of a review, they should be handled deliberately. Check whether a peer-reviewed version has subsequently appeared, link multiple versions of the same study, and ensure that one investigation is not counted twice.
The question is not simply “Are preprints trustworthy?” It is “What report of this study am I looking at, what can I reliably learn from it, and is there a later or more complete report?”
Grey literature can contain information that journal articles omit
Peer-reviewed journal articles operate under practical constraints. Word limits, editorial priorities, and publication conventions can affect what is reported.
Other sources may provide additional information. Trial registers can reveal pre-specified outcomes. Dissertations may provide fuller descriptions of methods and analyses. Regulatory documents and clinical study reports can contain extensive study information. Conference materials can reveal that a study existed even when no full journal publication is found.
For systematic reviews, searching multiple report types can therefore help both study identification and interpretation.
This is also why PRISMA 2020 asks reviewers to specify all databases, registers, websites, organizations, reference lists, and other sources used to identify studies rather than treating bibliographic journal databases as the entire search universe.
Searching grey literature does not mean including everything you find
This point is critical.
Searching broadly and including indiscriminately are not the same methodological choice.
Search broadly
Allow potentially eligible studies to be identified across appropriate publication types and information sources.
Include according to criteria
Apply pre-specified eligibility rules and appropriate appraisal to determine which studies and information contribute to the review.
A dissertation can be retrieved and excluded because its study design is ineligible. A conference abstract can identify a study whose full report is later used. A government report can be included when it meets the criteria. A peer-reviewed journal article can also be excluded because its study does not address the review question.
Publication status should not substitute for well-defined inclusion and exclusion criteria.
Do not use peer review as a substitute for critical appraisal
If a study qualifies for inclusion, its methodological strengths and weaknesses still need to be assessed using methods appropriate to the type of evidence and review.
JBI describes critical appraisal as the process used in systematic reviews to evaluate methodological quality, internal validity, and risk of bias among studies meeting the inclusion criteria. This evaluation is conceptually separate from the publication venue.
A peer-reviewed study can have serious risk-of-bias concerns. A dissertation can have a strong design. A preprint can contain a rigorous study but still warrant additional caution because its reporting has not completed the journal review process.
These judgments require appraisal rather than labels.
The answer depends on what kind of literature search you are doing
Not every literature search needs the comprehensiveness of a systematic review.
| Search purpose |
Role of peer-reviewed literature |
Need for broader sources |
| Quick background search |
Peer-reviewed articles may provide a useful practical focus |
Broader searching may be unnecessary unless the topic requires it |
| Conventional thesis or research-paper literature review |
Peer-reviewed scholarship will often form much of the academic evidence base |
Reports, policies, dissertations, datasets, standards, and other sources may still be appropriate depending on the question |
| Scoping review |
Peer-reviewed studies may be important but need not define the entire evidence base |
Grey literature may be relevant depending on the review question and protocol |
| Systematic review |
Peer-reviewed journal reports are important sources of evidence |
Publication-status restrictions generally require justification because broader searching can reduce reporting and publication bias |
If your assignment specifically requires peer-reviewed journal articles, then following that requirement is reasonable. That is an assignment constraint, not a universal rule of literature searching.
Some research questions depend heavily on non-journal evidence
For certain questions, searching only peer-reviewed journals would be especially limiting.
A review of government policy implementation may need government reports. Research on doctoral education may benefit from dissertations. A review of ongoing clinical trials requires trial registers. Questions about emerging technologies may encounter important preprints and conference proceedings before the journal literature catches up.
The information sources should follow the evidence needed to answer the question.
This connects directly to the question of how many databases and other sources you actually need. Searching several journal databases does not necessarily substitute for searching a register, repository, dissertation database, or relevant organizational source when those contain evidence important to the topic.
More sources do not automatically mean a better review
The opposite mistake is to assume that rigorous searching requires collecting every conceivable form of grey literature.
That is not the goal either.
Information sources should be selected because they are plausibly useful for identifying eligible evidence. A review of randomized clinical trials has different priorities from a review of educational policy documents. Searching a dissertation database may be valuable for one and contribute little to another.
The objective is purposeful coverage, not maximal accumulation.
This is why searching more databases does not automatically improve a literature search. The same principle applies to grey-literature sources.
Peer-reviewed filters are not necessarily as simple as they appear
Some discovery systems and databases offer a “peer reviewed” checkbox. Before relying on it, determine what the platform means by that designation.
A database may classify journals rather than individual articles, apply publication-type metadata, or derive the designation from source lists. Editorials, letters, news items, corrections, and other material may appear within journals that otherwise use peer review.
Conversely, some databases do not offer a reliable peer-review limiter at all.
If peer-review status is an eligibility requirement, check the platform's documentation and verify ambiguous cases rather than assuming that a single interface filter perfectly operationalizes the criterion.
Publication-type filters can accidentally exclude relevant records
Researchers sometimes try to identify peer-reviewed evidence by limiting a search to “journal article” or another publication type.
This may not do what they intend.
Publication-type indexing varies by database, and newly added records may not yet have complete indexing. A relevant study may also appear under a different document type or have multiple reports.
Cochrane advises that searches should capture as many eligible studies as possible and that restrictions based on publication format or status should be justified. Its search guidance warns that publication-status restrictions can introduce publication bias.
Watch Out
Do not assume that selecting “journal article” is equivalent to selecting “methodologically reliable study.” Document type describes a report. It does not appraise the research.
Grey literature introduces its own challenges
Searching beyond journal literature is not cost-free.
Grey literature can be difficult to locate systematically. Search interfaces may be rudimentary. Metadata can be sparse. Full texts may disappear from websites. Duplicate reports can be difficult to identify. Citation information may be incomplete. Methodological reporting may be limited.
Unpublished studies that can be located may also be an unrepresentative subset of all unpublished studies. Cochrane explicitly acknowledges this limitation while concluding that excluding all unpublished studies poses a greater concern because of known reporting biases.
So the choice is not between a pristine peer-reviewed evidence base and a perfectly comprehensive grey-literature evidence base. Both have limitations. The task is to design a search that manages those limitations in relation to the review question.
Record the source and publication status as you work
If your search includes multiple report types, documentation becomes particularly important.
You may need to record:
- where each record was found;
- what type of report it is;
- whether a later or peer-reviewed version exists;
- whether several reports represent the same underlying study;
- which report supplied the information used in the review.
PRISMA 2020 requires systematic review authors to report all information sources searched and the full strategies, including filters and limits. Maintaining those records while searching is considerably easier than reconstructing them after screening.
The broader documentation requirements are covered in what you should record so the literature search can be reproduced.
