03 · What You Need to Know
A report can be authoritative without being strong research evidence
Government and institutional reports are a major form of grey literature
Government departments, public agencies, universities, research institutes, professional organizations, international bodies, nongovernmental organizations, and other institutions routinely produce material outside conventional commercial academic publishing. Such documents are commonly classified as grey literature.
JBI guidance explicitly recognizes government reports, surveys, administrative sources, and reports from experts among sources that may provide useful published or unpublished evidence for reviews. Methodological literature on grey literature similarly includes government, research, and committee reports among potentially relevant sources for systematic reviews.
The practical importance varies by discipline. Government agencies may possess national administrative datasets unavailable to ordinary researchers. International organizations may conduct large-scale surveys. Universities may publish institutional evaluations. Research institutes may produce commissioned studies intended primarily for policymakers or practitioners rather than journal audiences.
The fact that such work sits outside journal publishing tells you something about its dissemination route. It does not tell you, by itself, whether the underlying evidence is strong or weak.
First ask whether the report actually contains research
The word report covers documents with very different purposes. Some reports present original empirical research. Others summarize existing evidence, explain policy, communicate statistics produced elsewhere, recommend action, document organizational activity, or advocate a particular position.
Consider several hypothetical documents:
- a national household survey reporting its sampling design, questionnaire, response rate, weighting procedures, and statistical analysis;
- an evaluation of a government program using administrative records and interviews;
- an annual institutional report listing enrollment and financial statistics;
- a policy paper recommending changes based on selected previous studies;
- a technical report documenting laboratory or engineering tests;
- an advocacy report combining secondary statistics with organizational arguments.
All might legitimately be called reports. They do not represent the same kind of evidence.
Empirical research report
Generates or analyzes data using identifiable methods that can be evaluated in relation to a research or evaluation question.
Policy, guidance, or advocacy document
May contain useful facts, interpretations, recommendations, or citations but does not necessarily constitute an original empirical study.
Before treating a report as research evidence, identify which role it actually plays.
Official status establishes provenance, not methodological validity
A report from a national government, major university, or internationally recognized organization may have strong institutional authority. That matters. Provenance can help you establish who is responsible for the document, what expertise they possess, and whether the source is authentic.
But authority and methodological validity answer different questions.
A respected organization can publish a weak analysis. A small research institute can produce a rigorous one. An official report may use excellent data but make conclusions that extend beyond those data. Conversely, a document without journal peer review may provide unusually detailed methods, transparent datasets, and carefully qualified interpretations.
Source authority
Who produced the report, their expertise, institutional responsibility, reputation, and accountability for the content.
Evidence quality
Whether the design, data, measurement, analysis, reporting, and interpretation adequately support the claim being made.
Do not let the first answer substitute for the second.
Peer review is relevant, but it is not a binary quality switch
Many government and institutional reports do not undergo conventional journal peer review. Some nevertheless undergo internal technical review, external expert review, statistical quality assurance, stakeholder review, formal clearance procedures, or other forms of scrutiny.
The absence of journal peer review therefore deserves attention, but it does not establish that no quality-control process occurred. Likewise, peer-reviewed publication would not guarantee that a study is methodologically sound.
If review procedures matter to your appraisal, look for an acknowledgments section, methodological appendix, technical documentation, review statement, institutional publication policy, or other description of how the report was checked.
When that information is unavailable, record the review process as unclear rather than assuming either rigorous review or no review at all.
Evaluate the study design, not just the document type
If a government report contains a cross-sectional survey, evaluate it as a cross-sectional survey. If it reports a qualitative study, examine sampling, data collection, analytic procedures, reflexivity, and reporting appropriate to qualitative research. If it evaluates an intervention, assess the design and potential sources of bias relevant to that evaluation.
This is important because a generic grey-literature checklist and a study-design appraisal tool answer partly different questions.
Grey-literature appraisal can help you examine provenance, transparency, objectivity, currency, and significance. A design-specific appraisal asks whether the actual empirical methods support the inference.
For empirical reports, you may need both perspectives.
AACODS provides one framework for appraising grey literature
A commonly used framework for evaluating grey literature is the AACODS checklist. The acronym refers to six domains:
| Domain |
What you are asking |
| Authority |
Who produced the document, and do the authors or organization have appropriate expertise and credibility? |
| Accuracy |
Are the aims, methods, data, references, and conclusions sufficiently documented and internally credible? |
| Coverage |
What does the report cover, what does it leave out, and are the scope and limitations clear? |
| Objectivity |
Is the presentation balanced, and are interests, assumptions, competing evidence, or potential biases apparent? |
| Date |
When was the information produced, collected, or updated, and is it sufficiently current for your question? |
| Significance |
Does the source make a meaningful contribution to the question or evidence base? |
AACODS is useful because grey literature often raises questions that journal-oriented appraisal tools do not emphasize. It should not, however, become a mechanical stamp of approval. A report can perform well on institutional authority and still contain a study design with serious risk of bias.
Look for a methods section, even when it is hidden elsewhere
Government and institutional publications are not always organized like journal articles. The main report may contain only a brief methodology section while detailed methods appear in:
- a technical appendix;
- a separate methodology report;
- supplementary files;
- survey documentation;
- a data dictionary;
- a statistical methods document;
- a project webpage;
- a linked protocol or evaluation framework.
Before concluding that methods are absent, check the report's appendices and accompanying documentation.
For quantitative evidence, look for information about sampling, measurement, missing data, weighting, statistical procedures, uncertainty, and analytic decisions. For qualitative evidence, look for recruitment, data-generation procedures, analytic methods, researcher positioning where relevant, and evidence supporting the themes or interpretations.
If the methods remain opaque after reasonable investigation, that limitation should affect how much weight you place on the findings.
Ask where the data came from
A report may display dozens of tables without generating any original data. Determine whether the authors collected the data, analyzed an existing administrative dataset, combined several secondary sources, commissioned another organization to conduct the research, or simply reproduced statistics from elsewhere.
This matters for both appraisal and citation.
If a report quotes a national statistic that actually originates from a statistical agency, the original statistical release may be the better source for that specific figure. If the report performs a novel analysis of the same dataset, however, the report itself may be the relevant source for that analysis.
Follow the evidence chain far enough to know what you are actually citing.
Administrative data can be valuable without being designed for research
Governments and institutions hold administrative data generated through ordinary operations: enrollment records, hospital records, benefits claims, tax records, licensing databases, school attendance, program participation, or service utilization.
These datasets can be exceptionally large and valuable. Yet they were often created for administrative rather than research purposes.
That raises questions about:
- how variables were defined;
- whether data capture was consistent;
- who is included or excluded;
- changes in systems over time;
- missing or erroneous records;
- whether the available variables adequately represent the concepts being studied.
A very large dataset does not repair a poor measurement process. Sample size and validity remain different properties.
Government statistics are not automatically causal evidence
Suppose an official report shows that regions with higher program participation have lower unemployment. That may be a useful descriptive association. It does not establish that the program caused unemployment to fall.
The strength of the inference depends on the research design and analysis, not on the official status of the statistics.
This distinction is particularly important because institutional reports often inform decisions. Descriptive monitoring, program evaluation, causal inference, forecasting, and policy recommendation are different analytical tasks. A document may perform one well without supporting the others.
Watch Out
Do not upgrade an association into a causal conclusion because the numbers come from an official source. Government provenance can strengthen confidence that you have located an authentic dataset or report, but causal inference still depends on design, measurement, comparison, analysis, and plausible alternative explanations.
Consider why the report was produced
Every research output has a context. Journal articles are shaped by publication incentives, disciplinary conventions, funding arrangements, and authors' interests. Institutional reports have their own contexts.
A ministry may be evaluating its own program. A consultancy may have been commissioned by an organization with a preferred policy direction. An industry association may produce evidence relevant to regulation of its members. An advocacy organization may intentionally seek evidence supporting social change.
None of these circumstances automatically invalidates the evidence. They do make transparency about funding, commissioning, authorship, analytic independence, and objectives important.
Ask whether the report acknowledges competing interpretations, limitations, and evidence that does not support its preferred conclusion. A report that transparently describes uncertainty deserves different treatment from one that presents advocacy as though it were neutral empirical inference.
Policy recommendations and empirical findings are not the same thing
Reports frequently move from data to recommendations. Keep those layers separate.
A national survey may provide strong evidence about the prevalence of a problem. The recommendation that government should adopt a particular intervention involves additional judgments about effectiveness, cost, feasibility, equity, values, and policy priorities.
You can therefore accept the empirical findings as relevant evidence without necessarily accepting every recommendation derived from them.
Empirical claim
A statement about what was observed, measured, estimated, compared, or analyzed using data.
Policy recommendation
A judgment about what should be done, potentially informed by evidence but also involving values, priorities, feasibility, and other considerations.
Currency matters differently for different claims
An older report is not automatically obsolete. If you are studying a historical policy, the report's age may be precisely what makes it relevant. If you are citing the current number of schools using a technology, a decade-old institutional survey may be inappropriate even if its methods were excellent.
Check both the publication date and, where possible, the period during which the data were collected.
A report published this year could analyze data from several years earlier. Conversely, a longstanding statistical series may remain the authoritative source for historical trends.
Reports can provide evidence unavailable in journals
Government and institutional reports can sometimes provide scale, detail, or timeliness that conventional articles do not.
They may contain:
- national or regional statistics;
- large administrative datasets;
- program evaluations;
- technical specifications;
- implementation evidence;
- local or institutional data;
- survey instruments and methodological appendices;
- policy and regulatory context;
- findings that were never submitted for journal publication.
Methodological discussions of grey literature emphasize that these sources can improve comprehensiveness and provide evidence absent from commercial publications. This is particularly relevant when excluding grey literature could systematically narrow the visible evidence base.
Do not give every report equal evidential weight
Two reports may both satisfy your broad inclusion criteria while deserving very different levels of confidence.
One might provide a clearly defined question, representative sampling, validated measures, transparent analysis, uncertainty estimates, full appendices, declared funding, and explicit limitations. Another might provide several charts and strong conclusions without explaining where the data came from.
Calling both "government reports" obscures the difference that matters.
The unit of appraisal should therefore be the actual evidence and the claims being drawn from it, not merely the document category.
