What Should You Do When an Important Study Doesn’t Meet Your Original Criteria?

What If the Paper Seems Too Important to Exclude?

You establish your eligibility criteria, begin screening, and eventually encounter a paper that creates an uncomfortable problem.

It is influential. Perhaps it introduced the theory underlying your topic, is cited throughout the field, contains unusually informative data, or seems directly relevant to the argument you are developing. Yet when you apply the criteria you established for the review, the study does not qualify.

Maybe the participants fall outside your population. Perhaps the study predates your publication window, uses an ineligible design, examines a neighboring intervention, or was conducted in a setting outside the scope you defined.

Should you make an exception?

Usually, the study's importance alone is not enough. An influential paper can be extremely useful without becoming part of the formal body of evidence being synthesized. The harder question is whether the paper merely falls outside a defensible boundary or whether its exclusion exposes a genuine problem with the boundary itself.

Do Not Give a Study Special Eligibility Because It Seems Important

An Ineligible Study Can Still Matter to Your Review

Start by Separating Importance From Eligibility

The first question is not whether the study is important. It is whether the study meets the eligibility criteria established for the evidence being reviewed.

Those are different judgments.

A study can be highly important and still be ineligible. Conversely, an obscure study can satisfy every eligibility criterion.

This distinction follows directly from the purpose of systematic-review eligibility criteria. Cochrane describes prespecified eligibility criteria as a fundamental feature of systematic reviews. These criteria are derived from the review question and normally specify the population, intervention or exposure, comparator where applicable, study designs, and other justified boundaries that determine which evidence can answer the question.

Once those boundaries are established, applying the criteria consistently means that prominence should not quietly become an additional route to eligibility.

“But Everyone Cites This Study” Is Not an Inclusion Criterion

Highly cited studies can exert a peculiar gravitational pull on a literature review. Once a paper is recognized as foundational, excluding it can feel almost intellectually irresponsible.

But citation count measures scholarly attention, not fit with your review question.

A landmark randomized trial in adults does not become evidence about children because it has 10,000 citations. A foundational study conducted in hospitals does not become eligible for a review explicitly restricted to primary care. A classic paper published before a justified historical boundary does not automatically cross that boundary because later researchers regard it as important.

None of this makes the study irrelevant to scholarship. It means that relevance to the broader topic and eligibility for a particular synthesis are not identical.

Being Excluded From the Evidence Base Does Not Mean Being Erased From the Article

This distinction solves many apparently difficult cases.

A paper can serve several functions in a literature review. Some sources provide the evidence being formally synthesized. Others explain the intellectual, theoretical, methodological, or historical context in which that evidence should be understood.

The important requirement is transparency. Do not present an ineligible contextual paper as though it were one of the studies selected under the formal eligibility process.

This is the same distinction involved in deciding which papers actually belong in the evidence being reviewed. Not every paper you cite needs to play the same role.

Ask Why the Study Fails the Criterion

Before deciding what to do, identify the precise reason for ineligibility.

Suppose the study:

• uses a population outside the defined scope;
• examines a related but different intervention or phenomenon;
• uses a study design excluded by the protocol;
• falls outside a justified publication period;
• occurs in an ineligible setting;
• reports a publication type outside the review criteria; or
• fails another explicitly defined requirement.

Then ask a second question: Is that boundary still methodologically justified?

This is where the problem becomes more interesting. Sometimes the study simply does not belong. At other times, discovering it exposes a criterion that was poorly specified, arbitrary, or inconsistent with the question the review was intended to answer.

There Are Two Very Different Reasons You Might Want an Exception

Imagine that your systematic review includes university students but excludes other postsecondary learners. You discover an influential study conducted in a degree-granting polytechnic.

You might want to include it for either of two reasons.

Reason 1: The review objective has always concerned higher education generally, and you realize that limiting the eligibility criterion to institutions formally called universities was an accidental narrowing of the intended population.

Reason 2: The study reports an especially compelling finding, and you do not want to lose it.

Those motivations are methodologically different.

Cochrane guidance recognizes that unforeseen issues can require sensible post hoc eligibility decisions, but emphasizes that such decisions should remain faithful to the objectives of the review. It also states that changes to the protocol must not be based on study findings or synthesis results because doing so can introduce bias.

Do Not Create a One-Paper Exception

If a criterion is genuinely defective, correct the criterion. Do not merely exempt the paper.

Suppose your original criteria exclude studies from polytechnics, but after reconsideration you determine that recognized degree-granting higher-education institutions should qualify.

The methodological consequence is not:

“Include this important polytechnic study.”

It is:

“Revise the eligible setting to include recognized higher-education institutions, then apply that revised criterion to every relevant record.”

That distinction is crucial. A defensible amendment creates a general rule. A special exception privileges one study.

If the criterion is substantively revised, follow the same principles involved in changing eligibility criteria after screening has begun: document the amendment, revisit affected screening decisions, and consider whether the search itself needs to be revised.

Use the Counterfactual Test

One useful way to detect selective reasoning is to mentally change the study's findings.

Suppose the important study reports a dramatic positive effect. Ask:

Would I still argue that the eligibility criterion is wrong if this study had found no effect, or the opposite effect?

If yes, you may have a genuine scope problem.

If no, your desire to amend the criteria may be entangled with the result.

The same test works for authorship and prestige. Would you interpret the criterion the same way if the study came from an unfamiliar research team and appeared in an obscure journal?

This is not a formal statistical procedure. It is a useful discipline for identifying when methodological judgment may be drifting toward preferential treatment.

Do Not Confuse Quality With Eligibility Either

An important study may appear methodologically superior to the studies that satisfy your criteria. That still does not automatically make it eligible.

Suppose your review concerns qualitative experiences of patients, while a landmark randomized trial provides extremely strong evidence about treatment effectiveness. The trial may be methodologically excellent, but it answers a different question.

Conversely, an eligible study may contain serious methodological limitations. Whether those limitations affect inclusion, synthesis, or interpretation depends on the review methodology.

The broader distinction between relevance and methodological quality therefore remains important here. Neither quality nor prestige can compensate for a fundamental mismatch with the question being synthesized.

Older Landmark Studies Require the Same Reasoning

This dilemma frequently appears when reviews use publication-date restrictions.

Suppose you specify studies published from 2015 onward and then discover a foundational 2009 study that shaped the entire field.

The appropriate response depends on why 2015 was chosen.

If the boundary corresponds to a meaningful technological, policy, diagnostic, theoretical, or historical transition relevant to the question, the older study may legitimately fall outside the formal evidence base while remaining important contextual literature.

If 2015 was chosen merely because ten years sounded manageable, discovering the older study may expose a weakly justified restriction.

The issue is therefore not simply whether an old study should be excluded because of its age. It is whether the date criterion has a defensible relationship to the review question.

The Same Applies to Studies From Different Countries or Populations

An influential study may fail eligibility because it was conducted in another country or among a somewhat different population.

Again, ask why that characteristic was excluded.

Cochrane advises that restrictions based on population characteristics or settings should have a sound rationale. Arbitrary restrictions can unnecessarily discard evidence, while overly broad criteria can reduce applicability to the question being asked.

So if an important study from another country appears, do not automatically include it because it is important and do not automatically exclude it because it is foreign. Examine whether the geographical boundary itself is conceptually necessary.

The appropriate reasoning is explored further when considering whether research from a different country or population should be excluded.

Do Not Secretly Move an Ineligible Study Into the Synthesis

A particularly problematic compromise is to declare a study formally excluded but then use its results as though they contribute to the synthesis.

For example, a review might report that only 12 studies met the eligibility criteria but later state that “the evidence shows...” based partly on quantitative results from a thirteenth ineligible study.

This makes the actual evidence base difficult to identify.

If an excluded study is cited for context, make its role clear. Do not let contextual citation become a back door into the formal synthesis.

If You Change the Criterion, Reconsider More Than the One Study

A substantive amendment has consequences.

Suppose you broaden the eligible population specifically because an important paper reveals that the original population boundary was unnecessarily narrow.

You may then need to:

• rescreen previously excluded records affected by the new criterion;
• retrieve full texts that now appear potentially eligible;
• reconsider studies already classified;
• check whether the original search was capable of retrieving the newly eligible evidence;
• rerun or modify searches where necessary;
• update protocol or registration information where appropriate; and
• report the amendment transparently in the completed review.

This is why changing the rules should not be treated as an easy solution to one awkward paper. A real change in scope has review-wide implications.

An Important Excluded Study May Need to Be Explicitly Reported

For systematic reviews reported using PRISMA 2020, transparency extends to studies that readers might reasonably expect to have been included.

PRISMA 2020 item 16b asks reviewers to cite studies that might appear to meet the inclusion criteria but were excluded, and to explain why they were excluded. The expanded checklist also calls for reporting the flow from identified records through screening and full-text assessment, including primary reasons for exclusion at the relevant stage.

This can be particularly useful for a well-known paper. Rather than bending the criteria because readers will notice its absence, explicitly showing why it was ineligible may make the review more transparent.

The detailed mechanics of recording reasons for study exclusion become especially important in these cases.

Systematic and Narrative Reviews Handle This Problem Differently

The dilemma is sharpest in systematic reviews because the included evidence is defined through explicit eligibility criteria.

A narrative review has greater freedom to select literature according to theoretical importance, historical influence, methodological contribution, or relevance to an argument. A landmark study outside a particular population or period may therefore still be central to the discussion.

But even narrative reviews benefit from distinguishing different functions of evidence. A historical study should not be presented as contemporary evidence without qualification. A study of a different population should not silently be generalized to the target population. A methodological paper should not be treated as empirical evidence for an effect.

The principle is broader than systematic-review procedure: use important papers for the contribution they can legitimately make, rather than changing their evidential role because they are important.

A Landmark Study Falls Outside Your Population Criterion

Now imagine instead that the review objective repeatedly refers to “learners across formal education,” while the protocol accidentally restricts eligibility to university students. The same paper now exposes a different problem: the population criterion may not represent the intended review question.

In that case, the appropriate response would not be to make an exception for the landmark study. The researchers would need to reconsider the population criterion itself, document any amendment, and apply the revised scope to all potentially affected evidence.

Common Mistakes When an Important Study Is Ineligible

Decide Whether the Problem Is the Study or the Criterion

When an important study fails an eligibility criterion, resist the immediate temptation either to force it into the review or dismiss it without reflection.

Instead, diagnose the situation.

One practical test is to write two short sentences before making the decision:

“This study is important because...”

“This study is eligible because...”

If you find yourself using the same answer for both, examine your reasoning. Importance and eligibility may overlap, but they are not interchangeable.

For a systematic review, also preserve a clear exclusion record when a potentially eligible or conspicuous study is rejected. PRISMA 2020 specifically asks authors to cite studies that might appear eligible but were excluded and explain why. That is often a better solution than distorting the eligibility rules merely to prevent readers from wondering where a famous study went.

Before Making an Exception for an Important Study

Questions About Important Studies That Fail Eligibility Criteria

An Important Study Can Matter Without Becoming Eligible

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.