Relevance vs. Quality: Does a Paper Need Both to Be Included?

If a Paper Is Relevant but Weak, Does It Still Belong in Your Review?

You find a study that addresses almost exactly the question your literature review is asking. The population fits. The topic fits. The outcomes are useful. Then you examine the methods more closely and notice serious limitations.

Should you exclude it?

The reverse can happen too. You may find an exceptionally rigorous study published in a respected journal, but its population, intervention, context, or research question differs substantially from yours. Should its methodological strength earn it a place in the review anyway?

These situations expose an important distinction in study selection: relevance and methodological quality are not the same judgment. They can both matter, sometimes considerably, but they answer different questions and should not automatically be collapsed into a single include-or-exclude decision.

A Paper Must Be Relevant, but Quality Does Not Always Determine Eligibility

Relevance and Quality Tell You Different Things About a Study

Relevance Asks Whether the Study Can Help Answer Your Question

Relevance concerns the relationship between a study and the purpose of your review. Does the study investigate the population, concept, intervention, exposure, phenomenon, context, outcome, or other elements that define what you are trying to understand?

In a structured review, this judgment is operationalized through eligibility criteria. Cochrane guidance for systematic reviews of interventions, for example, emphasizes defining eligibility criteria from the review question and specifying the types of studies that can address it. JBI similarly treats congruence between the review question and inclusion criteria as fundamental to evidence synthesis.

This is the logic behind deciding which papers actually belong in the review. A study does not become relevant because it is famous, highly cited, recent, sophisticated, or published in a prestigious journal. It is relevant because it addresses the evidence you need.

Quality Asks How Much Confidence You Can Place in the Study

Methodological quality concerns how well a study was designed, conducted, analyzed, and reported relative to the methodological standards appropriate to that kind of research. In many evidence-synthesis contexts, the more precise concept is risk of bias: whether features of the study could systematically distort its results.

The appropriate appraisal therefore depends on study design. Concerns relevant to a randomized trial are not identical to those relevant to a cohort study, diagnostic accuracy study, qualitative investigation, or systematic review.

Depending on the design, appraisal may consider issues such as allocation procedures, confounding, participant selection, measurement, missing data, analytic methods, selective reporting, or congruence between qualitative methodology and interpretation.

A study can therefore perform well on one dimension and poorly on the other.

A High-Quality Irrelevant Study Is Still Irrelevant

Imagine that your review examines the experiences of first-year university students using generative AI for formative feedback. You find a large randomized controlled trial evaluating an AI tutoring system among secondary-school students.

The trial may be meticulously designed. That does not make it evidence about first-year university students' experiences of generative AI feedback.

It could still be valuable background literature. It might help explain the technology, identify possible mechanisms, or inform the discussion. But methodological strength cannot compensate for failure to satisfy the substantive boundaries of the review.

This distinction is particularly important during screening. If a study clearly does not satisfy the eligibility criteria, there is usually little reason to perform a detailed methodological appraisal merely to determine whether it should be included. Relevance and eligibility logically come first.

A Relevant but Methodologically Weak Study Creates a Different Problem

Now reverse the situation. Suppose a study directly examines the population and phenomenon in your review, but the methods raise substantial concerns.

That study is still relevant. The question becomes what your review methodology says to do with evidence of limited credibility.

There is no universal rule that every methodologically weak study must be excluded. In some systematic reviews, eligible studies remain included regardless of their risk-of-bias assessment, with those assessments incorporated into the synthesis and interpretation. This preserves visibility of the available evidence while preventing weak evidence from being treated as though it were equally convincing.

Other review protocols may specify methodological features as eligibility criteria. A review might, for example, include only randomized trials because the question concerns intervention effects and the review was explicitly designed around that evidence. This is different from screening studies according to whether the reviewer personally considers them “good.” The eligible study designs were defined as part of the review's scope.

Study Design Eligibility Is Not the Same as Post Hoc Quality Filtering

This distinction is easy to miss.

Suppose a systematic review specifies before screening that only randomized controlled trials are eligible. Excluding a cohort study is then an eligibility decision. The cohort study does not satisfy the predetermined study-design criterion.

Now suppose an eligible randomized trial is critically appraised and found to have a high risk of bias. Excluding it solely because the result of the appraisal is unfavorable is a different action. That is a quality-based exclusion.

If critical appraisal will be used as a basis for exclusion, that rule should be determined prospectively where possible. JBI guidance, for example, indicates that when critical-appraisal scores or particular criteria will determine exclusion, the decision rule should be established before critical appraisal begins rather than improvised after researchers see which studies pass or fail.

Quality Is Not a Property You Can Reliably Infer From the Journal

Researchers sometimes use shortcuts: “It is in a Q1 journal, so the study must be high quality,” or “It is from an unfamiliar journal, so I should exclude it.” Neither is an adequate study-level appraisal.

Journal metrics, indexing status, citation counts, institutional reputation, and author prominence may tell you something about visibility or publication context. They do not establish whether the particular study handled confounding appropriately, measured its variables well, avoided important biases, or used an analysis suited to its design.

Peer review is likewise not a substitute for critical appraisal. Published research can contain methodological limitations, and those limitations vary from study to study.

Critical Appraisal Should Match the Kind of Evidence

Calling a paper “low quality” without explaining what was assessed can conceal more than it reveals. Different research designs are vulnerable to different methodological problems, so appraisal tools and criteria should be appropriate to the evidence being examined.

JBI provides design-specific critical appraisal tools for multiple forms of evidence. Cochrane systematic reviews of interventions use structured risk-of-bias approaches appropriate to the designs under review. Other established tools exist for particular evidence types and review purposes.

The important principle is not that every literature review must use the same checklist. It is that judgments about quality should be based on relevant methodological features rather than intuition, journal reputation, writing polish, or whether you happen to agree with the findings.

Not Every Literature Review Needs the Same Quality-Appraisal Procedure

The distinction also depends on what you mean by “literature review.”

Systematic reviews generally require explicit procedures for assessing methodological limitations or risk of bias according to the applicable review methodology. The assessment then contributes to interpretation of the evidence and, depending on the protocol, may affect synthesis or inclusion.

Scoping reviews have a different purpose. They often seek to map the extent, characteristics, or nature of evidence rather than determine an effect estimate or establish the certainty of a narrowly defined body of evidence. Critical appraisal may therefore be unnecessary for some scoping review objectives, although it can be conducted when it serves the purpose of the review.

Traditional narrative reviews are more heterogeneous. Formal risk-of-bias assessment may not always be appropriate or expected, but researchers should still evaluate the credibility and limitations of evidence rather than treating every publication as equally persuasive.

Four Papers Can Have Very Different Combinations of Relevance and Quality

Study B is the case that often causes confusion. Its weaknesses matter, perhaps substantially, but the protocol in this example does not define methodological quality as a threshold for eligibility. The appropriate response is therefore not to make the study disappear. Its limitations should remain visible when the evidence is analyzed and conclusions are drawn.

Studies C and D illustrate the opposite problem. Strong methods cannot answer a question the studies were not designed to address. They might still inform background or discussion, but they do not become eligible evidence simply because they are methodologically impressive.

Common Mistakes When Judging Relevance and Quality

Decide What Quality Will Do Before You Start Using It to Exclude Studies

The practical issue is not simply whether you should care about study quality. You should. The more important question is what role methodological quality is supposed to play in your particular review.

Start by separating eligibility from appraisal. First determine whether a study fits the review question and the predetermined inclusion criteria. Then, when required by the methodology, assess its methodological limitations using an appropriate approach.

If you find yourself applying your criteria differently depending on the paper, the problem may lie in the criteria rather than the study. This is where applying inclusion and exclusion criteria consistently becomes important.

And if screening exposes a genuine flaw in the criteria themselves, do not quietly make an exception. A justified amendment may sometimes be necessary, but changing inclusion criteria after screening has started requires careful consideration and transparent documentation.

Before Excluding a Paper Because of Its Quality

Questions About Relevance and Study Quality

Relevance Determines Fit; Quality Determines How Carefully You Trust the Evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.