03 · What You Need to Know
Evaluate what was done, not merely where it was published
Peer review and research quality are related questions, not the same question
Journal peer review subjects a manuscript to scrutiny before publication, usually by editors and researchers with relevant expertise. That process can identify unclear reporting, methodological problems, unsupported claims, and other weaknesses. It is therefore reasonable to care whether a report underwent external review.
But peer review is a process, not a quality certificate. A peer-reviewed article can still contain important methodological limitations. Conversely, research conducted outside journal publishing can be carefully designed, transparently reported, and technically reviewed.
Peer-review status
Tells you something about the publication and scrutiny process through which the document passed.
Methodological quality
Concerns whether the design, data collection, measurement, analysis, and reporting support the study's findings and the inferences drawn from them.
For a non-peer-reviewed report, you therefore need to perform the appraisal rather than assuming that either the absence or presence of journal peer review settles the matter.
Start by identifying exactly what kind of document you have
The word report is extraordinarily broad. Before evaluating quality, establish what the document is trying to do.
It might be:
- an empirical research report;
- a program evaluation;
- a technical report;
- an analysis of administrative data;
- a survey report;
- an evidence synthesis;
- a policy report;
- a consultancy report;
- an expert or advisory report;
- an advocacy document;
- an organizational performance report.
These documents require different questions. You cannot appraise an expert policy argument as though it were a randomized trial, nor should you evaluate a national survey using criteria designed for qualitative interviews.
If the report contains original empirical research, identify the underlying study design first. That determines which methodological standards are relevant.
AACODS is useful for evaluating grey literature
One established framework for critically appraising grey literature is the AACODS checklist. It considers six domains: Authority, Accuracy, Coverage, Objectivity, Date, and Significance. The framework is specifically intended to help researchers evaluate sources that may not have passed through conventional publishing and peer-review systems.
| AACODS domain |
Questions to ask |
| Authority |
Who is responsible for the intellectual content? What expertise, institutional affiliation, or relevant experience do the authors and organization have? |
| Accuracy |
Are the aims, methods, data sources, analysis, references, and conclusions sufficiently documented and credible? |
| Coverage |
What population, evidence, period, or subject does the report cover, and are important boundaries or exclusions made explicit? |
| Objectivity |
Is the standpoint clear? Are competing evidence, uncertainty, limitations, and potential interests acknowledged rather than selectively ignored? |
| Date |
When was the report produced and when were the underlying data collected? Is that timing appropriate for the claim being made? |
| Significance |
Does the report contribute meaningful, relevant, or otherwise difficult-to-obtain evidence to your research question? |
AACODS can impose useful discipline on what might otherwise become a vague judgment that a source "looks credible." It should not, however, be treated as a universal replacement for methodological appraisal.
Authority: find out who is actually responsible for the evidence
Begin with authorship and provenance.
Ask:
- Are individual authors identified?
- What are their qualifications or relevant expertise?
- Which organization produced the report?
- Does that organization have responsibility or recognized expertise in the subject?
- Who commissioned the work?
- Who funded it?
- Was another organization responsible for collecting or analyzing the data?
- Are contributors, reviewers, or advisory groups identified?
A recognizable institutional logo is useful for establishing provenance but should not end the investigation. A government agency may commission a consultancy. A university center may prepare a report for an industry body. An advocacy organization may sponsor independent researchers. The relationships matter because they help you understand who controlled the research question, data, analysis, and publication.
If authorship is completely unclear, that does not automatically make every statement false, but it weakens accountability and should affect how confidently you rely on substantive claims.
Funding and commissioning deserve separate attention
Funding does not invalidate research. Nearly all substantial research requires resources from somewhere.
The relevant questions concern influence and transparency. Did the funder design the study? Could the commissioning organization prevent publication? Did researchers have access to the complete dataset? Was the analysis conducted independently? Are conflicts of interest disclosed?
These questions matter whether the funder is a government, company, charity, professional organization, advocacy group, or university.
Do not use a simplistic rule such as "industry-funded equals biased" or "government-funded equals independent." Examine the actual relationship between the sponsor and the research.
Accuracy begins with a surprisingly basic question: where did the numbers come from?
A polished report may contain charts, percentages, regression coefficients, quotations, and infographics while providing remarkably little information about their origins.
Trace important evidence backward.
If the report states that 62% of respondents hold a particular view, ask:
- Who were the respondents?
- How were they selected?
- How many participated?
- What question were they asked?
- When was the survey conducted?
- How was nonresponse handled?
- Were results weighted?
- Is uncertainty reported?
If a statistic comes from another organization, locate the original source when feasible. If the report performs a new analysis of an existing dataset, distinguish the original data from the report authors' analytical contribution.
Numbers do not become self-validating merely because someone has placed them in a very respectable table.
Look beyond the executive summary for the methodology
Executive summaries are written to communicate conclusions quickly. They are rarely sufficient for critical appraisal.
Search the complete report for:
- methods or methodology sections;
- technical appendices;
- sampling documentation;
- survey instruments;
- coding frameworks;
- data dictionaries;
- statistical notes;
- supplementary files;
- protocols;
- linked methodological reports.
Some institutional publications separate the accessible public report from a much more technical methodological document. Before judging a report as opaque, check whether that supporting documentation exists.
If it does not, methodological opacity is itself relevant evidence about how confidently the report can be evaluated.
Use a study-design-specific appraisal when the report contains empirical research
AACODS asks useful questions about the credibility of grey literature as a source. But suppose the report describes a cohort study, cross-sectional survey, qualitative study, quasi-experimental evaluation, diagnostic study, or systematic review. You still need to ask whether that particular study design was conducted appropriately.
JBI provides critical-appraisal tools tailored to several research designs, including cohort, case-control, analytical cross-sectional, qualitative, quasi-experimental, systematic review, and other forms of evidence. The purpose of such appraisal is to assess methodological quality and the extent to which bias may have affected study design, conduct, and analysis.
This leads to a useful two-level approach:
Source-level appraisal
Evaluate authority, provenance, transparency, objectivity, currency, coverage, and significance using a framework such as AACODS.
Study-level appraisal
Evaluate the empirical research using criteria appropriate to its actual design and the risks of bias relevant to that design.
A report can perform well at one level and poorly at the other. A highly transparent government report may openly describe a weak observational design. Conversely, a technically sophisticated analysis may be published in a document with unclear authorship or funding.
Coverage asks what the report represents and what it leaves out
Every report has boundaries. The important question is whether those boundaries are visible and appropriate to the claims.
Suppose an institutional survey is described as showing what "university students" think, but only students from one campus were invited. Or a government report discusses national employment trends while excluding informal workers. The statistics may be correct for the population actually observed while the report's language encourages a broader interpretation.
Look for:
- geographic boundaries;
- population eligibility;
- sampling coverage;
- excluded groups;
- time periods;
- missing data;
- outcomes not reported;
- evidence sources intentionally omitted.
A credible report should make important limitations reasonably visible. When it does not, narrow your own interpretation accordingly.
Objectivity does not mean the authors have no perspective
Complete neutrality is an unrealistic standard. Researchers make choices about questions, methods, comparisons, outcomes, and interpretations even in peer-reviewed studies.
For grey literature, the more useful question is whether the authors' standpoint and interests are sufficiently transparent and whether the evidence is handled fairly.
Look for signs that the report:
- distinguishes data from interpretation;
- acknowledges important limitations;
- discusses findings that complicate its preferred narrative;
- uses evidence consistently rather than selectively;
- identifies uncertainty;
- discloses relevant interests;
- supports important factual claims with traceable sources.
A report produced to advocate for a policy is not automatically unusable. Its purpose does, however, make it especially important to examine whether contrary evidence and uncertainty have been handled fairly.
Watch Out
Do not confuse a professional tone with objectivity. A report can be beautifully typeset, institutionally branded, and written in restrained language while still selecting evidence strategically. Appraise the evidence chain, not the aesthetics.
Check whether the conclusions outrun the design
A report may conduct its analysis competently and still overstate what the evidence means.
Watch for common inferential leaps:
- correlation described as causation;
- a convenience sample generalized to an entire population;
- self-reported behavior treated as objectively observed behavior;
- a short-term outcome interpreted as long-term impact;
- statistical significance presented as practical importance;
- descriptive trends used to claim program effectiveness without an appropriate comparison;
- one institution's experience generalized to a national context.
The appropriate question is not merely "Are the data real?" It is "What inference can these data and this design legitimately support?"
Read the limitations section, then look for limitations the authors did not mention
A good limitations section is encouraging because it demonstrates some recognition of uncertainty. It is not necessarily exhaustive.
Compare the authors' stated limitations with what you observe from the methods. For example, the authors may acknowledge a small sample but say nothing about severe selection bias. They may discuss missing responses but not explain whether the measurement instrument was validated. They may note that findings are observational but still use causal language throughout the conclusions.
Critical appraisal requires independent judgment rather than merely copying the report's self-assessment.
Check the date of the evidence, not only the date printed on the cover
A report published recently may analyze old data. A report published several years ago may remain the definitive source for a historical period.
Identify, where possible:
- publication date;
- data-collection dates;
- period covered by administrative records;
- date of the literature search if the report synthesizes evidence;
- version or update date.
Currency is relative to the claim. A 2018 report may be excellent evidence about conditions in 2017 and poor evidence about conditions today.
Significance asks whether the source adds something that matters
A report does not need to be perfect to be useful. The final AACODS domain asks whether the material makes a meaningful contribution in context.
A report may provide the only available national dataset, detailed local implementation evidence, an evaluation of a program that was never academically published, or information about a population absent from journal studies. Such evidence may be highly significant despite methodological limitations.
Significance should not be confused with validity. Unique evidence can still be weak evidence. Instead, ask what the review would lose if the report were omitted and whether its contribution justifies including it with appropriate qualification.
Do not turn AACODS into an arbitrary numerical quality score
Checklists can tempt researchers to count ticks: five yes answers must be better than four, and twelve must surely be twice as good as six.
That logic can be misleading. Different weaknesses have different consequences. An old publication date may be irrelevant for a historical research question, while an unidentified data source could fundamentally undermine a prevalence estimate. Similarly, unclear authorship and a minor coverage limitation should not necessarily receive equal evidential weight simply because each occupies one checklist item.
Unless your review methodology specifies and justifies a validated scoring approach, use appraisal domains to structure judgment rather than inventing a numerical threshold for "good" and "bad" reports.
Appraisal should affect interpretation, not merely produce paperwork
The point of evaluating a report is not to complete a checklist and file it away. Appraisal should influence what you do with the evidence.
A limitation may lead you to:
- exclude a report under predefined eligibility criteria;
- include it but qualify the conclusion;
- use it for descriptive context but not causal inference;
- avoid extracting a particular outcome;
- seek corroboration from another source;
- downgrade confidence in a finding;
- conduct a sensitivity analysis where appropriate;
- highlight uncertainty in the final synthesis.
In systematic reviews, the consequences of appraisal should be specified by the review methodology rather than improvised after seeing whether a report agrees with the emerging conclusion.
Apply comparable scrutiny to peer-reviewed and non-peer-reviewed evidence
It is reasonable to examine a non-peer-reviewed report carefully because no conventional journal review process can be assumed. But do not create an asymmetric standard in which reports must prove their innocence while journal articles are accepted at face value.
JBI's critical-appraisal guidance treats methodological appraisal as part of evidence synthesis for included research. Peer-reviewed papers still require assessment of methodological quality and risk of bias.
This is particularly relevant when deciding whether government and institutional reports can contribute research evidence. Publication venue is one characteristic of a source. It should not substitute for examination of the research itself.
