Should You Decide Your Inclusion and Exclusion Criteria Before Searching?

Do You Need to Know What You Will Include Before You Start Searching?

You are about to search the literature, but several decisions are still unsettled. Should you include only undergraduate students or all university students? Only empirical studies? Only journal articles? Studies from the last ten years? English-language publications? What about conference papers, dissertations, mixed populations, or studies that measure your outcome but do not report it in the way you expected?

One approach is to search first and decide later. You retrieve the literature, see what is available, and then determine which studies seem suitable.

For exploratory searching, some flexibility is entirely reasonable. For a systematic review or another evidence synthesis intended to use explicit and reproducible study-selection methods, however, deciding eligibility after seeing the results creates a methodological problem. Your criteria can begin to reflect the studies you happened to find rather than the question you intended to answer.

The important distinction is between preliminary searching used to understand and refine the review and the formal search and selection process used to identify the evidence that will actually be included.

For Systematic Reviews, Define Eligibility Before the Formal Search and Screening

Eligibility Criteria Should Guide the Search, Not Be Invented From Its Results

Inclusion and exclusion criteria define which studies belong in your evidence base

Eligibility criteria are the rules used to determine whether a study or report belongs within the scope of a review. Inclusion criteria specify characteristics that eligible evidence must satisfy. Exclusion criteria identify characteristics that make evidence ineligible.

The terminology is sometimes used as though these are two completely separate lists, but they are often two sides of the same decision.

JBI guidance notes that explicit inclusion criteria may sometimes be sufficient because exclusion can simply represent the opposite of inclusion. Explicit exclusion criteria become particularly useful when they clarify boundaries that might otherwise be ambiguous.

For example, saying that a review includes empirical studies involving university students already implies that purely conceptual papers are not eligible. You do not necessarily need an exhaustive catalogue of every imaginable publication that will not qualify.

Systematic reviews require criteria to be specified in advance

Pre-specification is a defining methodological feature of systematic evidence synthesis.

The Cochrane Handbook states that systematic review authors should pre-specify the criteria used to include and exclude studies. Its methodological standards describe predefined, unambiguous eligibility criteria as a fundamental prerequisite for a systematic review.

There is a straightforward reason for this. If you decide what counts only after seeing the available studies, your decisions can be influenced by their characteristics or findings.

Imagine planning a review of an educational intervention. After searching, you discover that most studies involve postgraduate students, although your original interest was undergraduates. You might be tempted to broaden the population because otherwise the review will contain few studies.

That change might ultimately be defensible. But it is methodologically different from having defined university students broadly from the outset. The decision now arises from knowledge of the retrieved evidence and therefore needs to be considered and reported as a change to the original plan.

Pre-specification does not mean you must avoid looking at the literature beforehand

This is where the advice is often misunderstood.

You do not need to formulate a review question and eligibility criteria in an evidence vacuum.

Before finalizing a systematic review protocol, researchers commonly need to understand whether relevant literature exists, how concepts are defined, what study designs have been used, whether another review already addresses the question, and whether the proposed review is feasible.

JBI guidance explicitly recommends preliminary searching when developing a review protocol, including searching for existing reviews on the same topic. It also expects the protocol to justify the review and its inclusion criteria in the context of what is already known.

The problem is therefore not “searching before setting criteria.” The problem is allowing the formal evidence base to determine eligibility opportunistically without a transparent methodological rationale.

Your research question should drive the eligibility criteria

A well-formulated review question guides the boundaries of the review. Cochrane notes that the review question influences eligibility criteria, searching, data collection, synthesis, and presentation of findings.

For intervention reviews, eligibility criteria are commonly organized around elements such as population, intervention, comparator, outcomes, and study design. Other types of reviews may use different frameworks appropriate to their questions.

Depending on the review, eligibility criteria may address:

• population or participants;
• phenomenon, intervention, exposure, or topic;
• comparison conditions where relevant;
• outcomes or phenomena of interest where appropriate;
• study design;
• setting or context;
• publication or report characteristics;
• date or time period;
• language;
• publication status or document type.

Not every review needs a restriction for every category. A criterion should exist because it helps operationalize the scope of the question, not because a template contains an empty box.

Criteria need to be operational enough for another person to apply them

“Include relevant studies” is not an eligibility criterion.

Neither is “include recent research,” “exclude low-quality studies,” or “include studies related to university students” unless those phrases are defined sufficiently for consistent application.

Eligibility criteria should make the intended boundary interpretable.

Cochrane recommends that criteria be sufficiently broad to encompass relevant diversity while sufficiently narrow to allow a meaningful answer. That balance is familiar from searching itself: boundaries need to be useful rather than merely detailed.

Eligibility criteria and search criteria are not the same thing

This distinction is essential.

Suppose your inclusion criteria require undergraduate students, an educational intervention, a particular outcome, and a randomized design.

It does not follow that every one of those elements should become a mandatory search block.

Cochrane advises that searches should be motivated by eligibility criteria but also notes that searching every aspect of the review question may be unnecessary or undesirable. Comparators and outcomes, for example, may not be described reliably in titles, abstracts, or indexing.

A characteristic can therefore be mandatory for inclusion without being mandatory for retrieval.

This is why deciding which concepts should actually become search terms remains a separate methodological decision.

Think of the search as a retrieval net and screening as the eligibility test

The formal search usually aims to retrieve candidate records broadly enough that eligible studies are unlikely to be missed. Screening then applies the eligibility criteria more precisely.

Suppose only studies involving undergraduate students are eligible. A search might nevertheless retrieve studies described simply as involving university students. During full-text screening, you can determine whether the participants were actually undergraduates.

If you had required the exact population characteristic during retrieval, the study might never have reached screening.

This helps explain why Cochrane advises designing searches to capture as many studies as possible that meet the eligibility criteria while avoiding unnecessarily restrictive search concepts.

Decide major eligibility restrictions for methodological reasons, not because they make screening easier

A large search result can make restrictions suddenly attractive.

You retrieve thousands of records and begin considering a five-year date limit. Perhaps English-only studies would be easier. Maybe excluding conference abstracts would reduce the workload. Perhaps restricting to one study design would make the project manageable.

Those decisions may sometimes be defensible. But reducing screening workload is not automatically a methodological justification for changing the scope of the review.

PRISMA 2020 expects systematic review reports to specify eligibility criteria, including relevant report characteristics such as year, language, and publication status. Its explanation and elaboration also recommends providing rationales for notable restrictions.

Questions about what date range should apply, whether to restrict by language, and whether to include only peer-reviewed literature should therefore be decided on substantive and methodological grounds.

Do not confuse “outcome not measured” with “outcome not reported”

Outcome-based eligibility requires particular care.

PRISMA 2020 recommends clearly distinguishing studies that are ineligible because an outcome of interest was not measured from studies in which the outcome was measured but the relevant result was not reported.

Cochrane guidance similarly cautions against automatically using reported outcome data as the basis for inclusion. Excluding a study simply because the result you want is unavailable can create problems related to selective reporting.

The appropriate treatment depends on the review question and methodology, but the broader lesson is important: define precisely what your criterion refers to. “No relevant outcome” can conceal several very different situations.

Inclusion and exclusion criteria should be piloted

A criterion can sound perfectly clear when written and become surprisingly ambiguous when applied to real papers.

Does a study with 80% undergraduates and 20% postgraduate students qualify? What if the results are not reported separately? Does a blended-learning intervention count as online learning? What if a conference paper later became a journal article?

Cochrane recommends pilot testing eligibility criteria on a small sample that includes clearly eligible, clearly ineligible, and uncertain reports. The purpose is to refine the criteria, train reviewers, and ensure that more than one person can apply them consistently.

Piloting is not the same as redefining the review after seeing which studies produce attractive findings. It is an operational test of whether the planned criteria can actually be applied.

Sometimes criteria legitimately need to change

Pre-specification does not require pretending that researchers can foresee every situation.

A full search or screening process may expose an ambiguity that was not apparent during protocol development. You may discover an unexpected study design, a population that straddles the eligibility boundary, or a publication format that the protocol did not anticipate.

A change can be justified.

What matters is transparency.

JBI guidance expects review reports to describe deviations from the methods specified in the a priori protocol. More generally, amendments should be documented with enough detail to distinguish a reasoned methodological revision from a post hoc decision influenced by desirable results.

Exploratory literature reviews require a different level of pre-specification

Not every literature search is a systematic review.

If you are exploring a topic for a thesis proposal, learning the vocabulary of a field, developing a research question, or preparing a conventional narrative literature review, you may not need a protocol containing rigid study-selection criteria before the first search.

Indeed, early searching may be necessary precisely because you do not yet understand the literature well enough to define sensible boundaries.

You should still make deliberate decisions about scope. Otherwise, it becomes easy to include studies because they support an emerging argument and ignore others because they do not.

The appropriate degree of pre-specification therefore depends on what you are claiming about the review.

The important point is not to borrow the language of systematic review methodology while selecting evidence opportunistically.

Your criteria should be settled before the formal search is treated as final

In practice, review development is often iterative:

This sequence allows the literature to inform good methodological planning without allowing the final eligibility rules to drift silently in response to whichever studies happen to appear.

Using a Preliminary Search Without Letting It Decide the Final Criteria

This example illustrates the difference between eligibility and retrieval. The review can have a precise population criterion while the search uses broader population terminology.

If the formal search later retrieves far more material than expected, that is not a reason to rewrite the eligibility criteria simply to reduce the workload. Instead, diagnose whether the search itself is unnecessarily broad.

Common Mistakes When Setting Inclusion and Exclusion Criteria

Decide What Counts Before You Start Deciding Which Studies Count

If you are conducting a systematic review, scoping review, or another evidence synthesis with explicit selection methods, treat eligibility criteria as part of the research design rather than an administrative step performed after searching.

Use preliminary searches to make the criteria better. Then establish the rules that will govern the formal selection process.

Once the criteria are established, they should inform the search without being copied mechanically into it. Your next methodological decisions include how far back the evidence needs to extend, which information sources cover it, and how comprehensively the search must retrieve potentially eligible studies.

Before Finalizing Your Eligibility Criteria

Questions About Setting Inclusion and Exclusion Criteria

Use Preliminary Searching to Refine the Rules, Then Apply the Rules to the Formal Search

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.