How Strictly Should You Apply Inclusion and Exclusion Criteria?

Should Your Criteria Be Treated as Absolute Rules?

You established your inclusion and exclusion criteria before screening. Then the real literature arrived.

One study is almost exactly what you wanted, except its participants are slightly outside your stated age range. Another uses an intervention that fits conceptually but has a different label. A third combines eligible and ineligible participants. Then you find an influential study that seems too important to leave out even though it technically falls outside one of your criteria.

How strictly should you apply the rules?

For a systematic or similarly structured review, eligibility criteria need to be applied consistently enough that the studies are selected because they fit the review question, not because the reviewer likes their findings, recognizes their authors, or considers them too important to exclude. At the same time, applying criteria consistently does not mean interpreting ambiguous wording mechanically when an unforeseen case exposes a genuine problem in how a criterion was originally defined.

The challenge is knowing the difference between applying judgment within your criteria and changing the criteria to accommodate a particular study.

Apply the Criteria Consistently, but Do Not Confuse Consistency With Rigidity

Consistency Matters More Than Mechanical Rule-Following

Eligibility Criteria Define the Boundaries of the Review

In a systematic review, inclusion and exclusion criteria are not administrative details added after the research question has been written. They operationalize the scope of the review.

Cochrane describes prespecified eligibility criteria as a fundamental feature of systematic reviews. For intervention reviews, these criteria commonly arise from the population, intervention, comparator, and eligible study designs, with other restrictions added where justified. JBI likewise emphasizes that eligibility criteria should be scientifically defensible and detailed enough for reviewers to make reliable study-selection decisions.

In practical terms, your criteria translate a conceptual question into decisions about actual studies.

This relationship matters because the criteria should serve the question. A criterion that no longer represents what the review is genuinely trying to investigate may need examination. But a paper that simply fails a valid criterion does not acquire an exemption because excluding it feels uncomfortable.

“Strict” Should Usually Mean Consistent

Researchers sometimes imagine strict screening as literal rule enforcement: if the protocol says participants aged 18 years and older, a study with a mean age of 19 but a small number of 17-year-olds must automatically disappear.

That may or may not be the appropriate interpretation.

The more useful question is whether your eligibility criteria already specify how mixed or borderline populations are handled. Perhaps studies are eligible if at least 80% of participants satisfy the population criterion. Perhaps subgroup data must be separately extractable. Perhaps any inclusion of minors makes the study ineligible. Each rule could be defensible in the right review.

The important issue is that the same interpretation should apply to comparable studies.

Consistency therefore means that Study A and Study B should not receive different treatment merely because Study A is famous, produces an expected result, or seems especially useful to your argument.

Do Not Add Unwritten Criteria During Screening

Selection criteria can quietly become stricter as reviewers encounter real papers.

Imagine that your protocol includes empirical studies conducted in higher education. During screening, you begin excluding studies from community colleges because you had universities in mind when writing the question. Later, you exclude studies conducted entirely online because you had imagined campus-based institutions. Neither restriction appears in the original criteria.

You are no longer simply applying the criteria. You are introducing new eligibility rules.

Sometimes screening reveals that a missing distinction genuinely matters. If so, the issue should be handled explicitly and consistently rather than becoming an unwritten rule applied record by record.

Do Not Create Exceptions for Studies You Want to Keep

The opposite problem is equally serious.

Suppose your review explicitly includes university students but excludes secondary-school populations. You discover a highly cited study of Grade 12 students that is conceptually fascinating and closely related to your topic.

Its importance does not change its population.

You may cite the study as background if that is appropriate, but including it in a body of evidence defined as university research would require a methodological justification beyond “this paper is too important to leave out.”

This is particularly important because post hoc decisions can become entangled with study findings. Cochrane requires changes to eligibility criteria to be justified and explicitly warns that protocol changes must not be made on the basis of study findings or synthesis results because doing so can introduce bias.

If a paper that falls outside the criteria nevertheless appears essential, the appropriate question is what role an important but formally ineligible study should play, not how to quietly make it eligible.

Borderline Cases Are Not the Same as Exceptions

A borderline case arises when it is genuinely uncertain how an existing criterion applies. An exception occurs when the criterion clearly applies but you decide not to enforce it for a particular study.

This distinction prevents methodological flexibility from becoming selective flexibility.

Criteria Need Operational Definitions

Many screening disagreements arise because a criterion is conceptually reasonable but operationally vague.

Consider:

Include studies of young adults.

What counts as a young adult? Is eligibility based on every participant's age, the mean age, an age range, or the way the original investigators define their sample?

Or consider:

Include studies of online learning.

Does that include blended learning? Emergency remote teaching? Courses with one online component? Synchronous videoconferencing? Self-paced digital modules?

The problem is not that researchers need to become more severe. The problem is that the criteria need enough conceptual and operational precision to support reproducible decisions.

JBI guidance emphasizes that eligibility criteria should contain adequate conceptual and operational definitions to enable reliable inclusion decisions. Cochrane similarly recommends pilot-testing eligibility criteria using clearly eligible, clearly ineligible, and doubtful reports before full selection proceeds.

Pilot Screening Is Where Useful Strictness Is Built

Before screening hundreds of records, test your criteria against a small but varied set of studies.

Cochrane suggests pilot testing eligibility criteria on several reports that include clearly eligible, clearly ineligible, and doubtful cases. The purpose is to refine ambiguous criteria, train reviewers, and ensure that different people can apply the rules consistently.

This is much safer than discovering after 700 abstracts that one reviewer interpreted “higher education” as universities only while another included vocational colleges.

One Failed Essential Criterion Can Be Enough

You do not necessarily need to evaluate every criterion after a study has clearly failed one essential eligibility requirement.

Cochrane recommends assessing eligibility criteria in an order of practical importance so that the first “no” can serve as the primary reason for exclusion. The order can differ among reviews.

Suppose a study clearly involves an ineligible population. There may be little value in spending another ten minutes determining whether its intervention, comparator, outcomes, and study design also fail.

This does not mean choosing whichever exclusion reason is easiest to report. It means designing a screening sequence that produces consistent, interpretable decisions without unnecessary work.

Do Not Treat Every Criterion as Equally Self-Evident

Some criteria are straightforward. Others involve legitimate judgment.

Publication year, for example, is usually unambiguous once the applicable date rule is established. Whether a complex intervention falls within a conceptual definition may require considerably more interpretation.

Similarly, a geographical restriction may appear simple until a multinational study includes both eligible and ineligible countries. A population restriction becomes complicated when subgroup data are available. A language restriction raises different questions from a methodological-design restriction.

Consistency therefore does not mean pretending judgment has disappeared. It means structuring that judgment so that similar cases are handled similarly and the reasoning can be explained.

Criteria Should Not Be Stricter Than the Review Question Requires

Restrictions narrow the evidence base, sometimes in ways that affect what conclusions can be drawn.

Cochrane advises that restrictions on population characteristics or settings should have a sound rationale. It also requires justification for excluding studies based on publication status. JBI similarly emphasizes that participant characteristics used for inclusion or exclusion should be justified rather than based on personal preferences.

This principle applies beyond health research. If you restrict a review to one country, one language, one publication type, one methodological design, or an arbitrary publication period, ask what feature of the research question makes that restriction necessary.

A criterion can be applied perfectly consistently and still be poorly conceived.

Outcomes Require Particular Care

It may seem logical to exclude every study whose abstract or paper does not report the outcome you want. In many systematic reviews, that can be problematic.

Cochrane generally advises against making outcome reporting itself an eligibility criterion because a study may have measured an outcome without reporting usable results. Excluding studies according to which results were reported can create vulnerability to reporting bias. There are legitimate exceptions when measurement of a particular outcome is fundamental to the review question, but these should be specified and justified.

So a rule such as “exclude studies that do not report a statistically significant effect on academic performance” would be especially problematic. It makes eligibility depend directly on findings.

Changing a Criterion Is Different From Clarifying It

Suppose your protocol defines eligible participants as “postsecondary students.” During pilot screening, reviewers disagree about whether vocational students qualify. You consult the intended scope of the review and clarify that accredited postsecondary vocational education is included.

Now consider a different situation. The protocol explicitly states “students enrolled in bachelor's degree programs,” but after screening begins you decide to include vocational students because several interesting studies would otherwise be lost.

The first situation may be clarification of an ambiguous operational definition. The second is a substantive expansion of eligibility.

Both should be documented appropriately, but they are not methodologically identical. If a substantive change is necessary, changing inclusion criteria after screening has started requires justification and transparent reporting.

The Findings Must Not Decide How Strict You Become

This is the boundary that matters most.

Cochrane's guidance is explicit that protocol changes should not be made on the basis of study findings or synthesis results. More broadly, systematic review methods are prespecified precisely because methodological decisions made after researchers know how those decisions affect the results are more susceptible to bias.

Imagine two studies that both narrowly fail the same criterion. One reports a dramatic effect supporting your hypothesis. The other reports no effect. If you find yourself searching for a reason to include the first while confidently excluding the second, the problem is no longer merely interpretive ambiguity.

Not Every Literature Review Requires Protocol-Level Formality

Systematic reviews rely particularly heavily on prespecified eligibility criteria because they make claims based on a deliberately identified body of evidence. Other review types may permit greater interpretive flexibility.

A narrative review, for example, might intentionally discuss seminal theoretical papers, methodological critiques, contemporary empirical evidence, and opposing perspectives even though these sources would not satisfy one common set of formal eligibility criteria.

That does not mean selection can be arbitrary. The researcher should still be able to explain why particular literature was included and avoid selectively presenting only evidence favorable to a preferred argument.

The stricter procedural expectations described here therefore apply most directly to systematic and similarly structured evidence syntheses. The broader principle, however, travels well: define what evidence you need, apply those boundaries consistently, and make exceptions visible rather than invisible.

What Should You Do With a Study That Almost Fits?

Now suppose the paper reports a spectacular improvement in student performance. That finding should not determine whether 150 of 180 eligible participants is acceptable. The same eligibility interpretation should have been reached if the study had reported no improvement at all.

This is the practical meaning of applying criteria strictly enough to protect the review from selective decision-making while remaining responsive to legitimate methodological problems that could not reasonably have been anticipated.

Common Mistakes When Applying Eligibility Criteria

Be Firm About the Principle and Careful About the Interpretation

A useful way to approach eligibility decisions is to separate three situations: clear application, genuine ambiguity, and substantive change.

When possible, resolve foreseeable ambiguities during protocol development and pilot screening. Define how you will handle mixed populations, overlapping age ranges, ambiguous settings, multi-component interventions, multiple reports, and other situations likely to occur in your evidence base.

You will not anticipate everything. Systematic reviewing would be much easier if the literature had read your protocol first. When an unforeseen case does arise, the standard is not perfect foresight. It is transparent, defensible, and consistent judgment.

Before Making a Borderline Eligibility Decision

Questions About Applying Inclusion and Exclusion Criteria

Apply the Same Rules to Comparable Studies, but Leave Room for Defensible Judgment

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.