When Should You Search the Reference Lists of Included Studies?

Once You Have Included Studies, Should You Search Every Reference List?

You have completed your main database search, screened the records, read the full texts, and identified studies that genuinely meet your inclusion criteria. Each included paper may contain dozens, sometimes hundreds, of references.

Should you now inspect those references for additional eligible studies?

Often, yes. Included studies are particularly useful starting points for citation searching because you already know they belong within your evidence base. Their references may lead to earlier studies that your database strategy missed because of different terminology, incomplete indexing, historical vocabulary, or other retrieval problems. But the value of reference list checking varies by topic, and a rigorous search requires more than casually scanning references that happen to look interesting.

Included Studies Are Strong Starting Points for Backward Citation Searching

Why Included Studies Make Particularly Useful Seed References

An Included Study Has Already Passed Your Relevance Test

Citation searching needs a starting point. Current methodological guidance calls these starting publications seed references.

You could choose seeds because they are famous, highly cited, recent, or recommended by an expert. But for a systematic review, included studies have an important advantage: they have already passed your predefined eligibility process.

You know that they address the question closely enough to belong in the review. Their citation relationships therefore provide a defensible place to search for additional literature.

The TARCiS statement recommends that supplementary backward and forward citation searching generally use all records meeting the review's inclusion criteria after full-text screening of the primary search results as seed references.

Reference List Checking Is a Specific Form of Backward Citation Searching

Terminology can become confusing here because researchers use expressions such as reference checking, hand searching references, citation chasing, citation tracking, and snowballing somewhat inconsistently.

The TARCiS statement recommends more precise terminology. Backward citation searching means retrieving and screening references cited by a seed reference. Reference list checking is the sub-method in which you do this by manually reviewing the reference list.

This is different from forward citation searching, which identifies later publications that cite the seed paper. The distinction between backward and forward citation searching matters because reference lists can only lead you into cited literature.

Why Wait Until After Full-Text Screening?

There is a practical reason to use included studies rather than every remotely relevant paper discovered during the search.

Before full-text screening, you may have hundreds or thousands of records that appear potentially relevant from their titles and abstracts. Searching every reference list would create a very large secondary screening workload, much of it originating from papers that eventually prove ineligible.

After full-text screening, the seed set is more defensible. These papers actually meet the inclusion criteria.

This does not mean researchers can never use other seeds. TARCiS recognizes circumstances in which additional pertinent records, such as relevant systematic reviews, may be justified. The important point is that seed selection should follow a rationale rather than convenience.

Reference Lists Can Reveal Earlier Studies Your Keywords Missed

Database searching relies heavily on searchable words and indexing. Reference lists do not.

An included paper can cite an earlier study even when the older paper uses terminology absent from your search strategy. That makes backward citation searching particularly useful when terminology has changed over time or differs among research traditions.

If an included contemporary paper cites an older publication that seems relevant, retrieve the original and inspect its terminology. It may provide the vocabulary needed to find still more older research written before modern terminology became established.

They Can Also Reveal Studies That Were Poorly Indexed

A relevant study may fail to appear in your search even when its terminology seems appropriate.

Perhaps it is not indexed in the database you searched. Perhaps its title and abstract provide little searchable detail. The database may classify the document unexpectedly, or your search fields and limits may not capture it.

A citation from an included study provides another retrieval route.

When reference checking uncovers an eligible paper your primary search missed, investigate why. The answer may expose a search weakness that affects more than one publication.

Reference Checking Can Test the Sensitivity of Your Primary Search

This is one of the more useful diagnostic roles of reference list checking.

For topics that are relatively easy to search and have already been addressed with a highly sensitive search, TARCiS does not explicitly recommend supplementary backward and forward citation searching. Instead, it notes that checking the reference lists of included records can be used to confirm the sensitivity of the primary search.

If the references reveal no additional relevant unique studies, that finding may provide some reassurance that the primary strategy was sufficiently sensitive. If they reveal eligible studies that the primary search missed, you have evidence that the original retrieval was not exhaustive.

This is not a formal mathematical sensitivity test unless you have an appropriate reference standard. It is a practical diagnostic check on how the search behaved.

Difficult-to-Search Topics Are Stronger Candidates for Citation Searching

Some research questions translate neatly into database concepts. Others do not.

Terminology may be inconsistent. Concepts may overlap with unrelated topics. Relevant studies may describe an intervention or phenomenon indirectly. Older research may use obsolete language. The topic may cross several disciplines.

For such difficult-to-search topics, TARCiS recommends seriously considering backward and forward citation searching as supplementary techniques.

Reference lists of included studies are particularly valuable in these situations because the citation relationships can connect relevant research without requiring every publication to share the vocabulary of your database query.

Search All Included Records When Feasible

For systematic searching, selectively checking only the reference lists that “look promising” can introduce an avoidable layer of researcher judgment.

TARCiS recommends using all records included after full-text screening of the primary database results as seed references for supplementary backward and forward citation searching.

The rationale is straightforward: using more relevant seed references increases the opportunity to identify additional unique relevant records, and current evidence does not provide a reliable rule for predicting in advance which included studies will be the most productive seeds.

If your review contains 12 included records, searching all 12 may be manageable. If it contains 1,200, the workload becomes a different problem.

A Very Large Evidence Base May Justify Sampling

TARCiS recognizes that searching every included record may become impractical when the seed set is very large.

In such circumstances, a defined sample of included records may be used. The departure from searching all included records should be justified, and the sampling method should be reported. The guidance gives random sampling as one example.

This is different from informally choosing the papers you personally consider most important. A predefined sampling approach makes the decision more transparent and less dependent on assumptions about which studies are likely to produce useful references.

A Very Small Evidence Base Can Create the Opposite Problem

If your primary search identifies only one or two eligible studies, their reference lists may provide a limited view of the literature.

In this situation, TARCiS notes that further pertinent records can sometimes be used as additional seed references. A closely relevant systematic review identified during screening is one possible example.

This can help broaden the citation network without pretending that a paper meets your inclusion criteria when it does not. The distinction should be documented clearly.

Do Not Screen References by Title Alone When Better Information Is Available

Manual reference lists often provide only bibliographic information. A title can be enough to recognize an obviously irrelevant paper, but many titles provide poor evidence about eligibility.

TARCiS recommends that backward citation searching ideally involve screening titles and abstracts of cited references as provided by a citation index. Manual title screening from reference lists can still be performed, but access to abstracts may improve the screening process.

In practical terms, a promising reference should usually be retrieved or searched in a bibliographic source so that you can evaluate enough information to make a defensible eligibility decision.

Every Retrieved Reference Still Needs Normal Screening

Included papers cite publications for many reasons.

Their reference lists contain background sources, theoretical papers, methods, measures, previous reviews, contextual evidence, and studies involving different populations or outcomes. The fact that an eligible paper cited another work does not make that work eligible.

Apply the same inclusion and exclusion criteria you use elsewhere unless your protocol specifies another defensible process.

Deduplicate Before Creating More Screening Work

Reference checking can quickly reproduce studies you already found through database searches or other seed papers.

Before screening citation-search results, TARCiS recommends deduplicating them. In an iterative search, new records can also be deduplicated against everything retrieved previously.

This matters especially when included studies belong to a tightly connected literature. Ten seed papers may share dozens of the same references, and screening every duplicate separately adds work without increasing coverage.

Newly Included Studies Can Become New Seeds

Suppose reference list checking identifies three additional studies that satisfy your inclusion criteria.

Should you inspect their references too?

Potentially, yes. TARCiS recommends considering another iteration of citation searching when supplementary searching identifies additional eligible records. Those newly included studies become new seed references.

This turns one round of reference checking into iterative citation searching or snowball searching.

Iteration can improve retrieval, but each round adds screening work. The decision to continue should consider how productive previous rounds were and the demands of the review.

Reference Lists Only Show the Literature That Came Before

A paper cannot cite research that had not yet been published when its reference list was finalized.

Reference checking is therefore inherently backward-looking. If you also want research published later than an included study, you need forward citation searching or updated database searches.

This is particularly important when included studies are several years old. Their references may provide excellent historical coverage while telling you nothing about the literature that subsequently developed around them.

Reference Checking Does Not Replace the Primary Database Search

Citation searching is powerful because it exploits relationships that keyword searches can miss. It also has structural limitations.

A relevant paper outside the citation networks of your seeds may never appear. Authors can fail to cite relevant work. Separate disciplines may form weakly connected citation communities. Recent papers may have limited citation connections.

TARCiS therefore recommends against standalone citation searching when the literature search aims for completeness of recall.

For systematic and scoping reviews, reference checking is better understood as a supplementary route into the evidence base rather than a reason to avoid a sufficiently comprehensive primary search.

What Happens When Included Studies Reveal Research Your Search Missed?

The value of reference checking is not merely that it adds two papers. It also tells you something about the behavior of the original search.

One missed study exposes a terminology problem. The other exposes a database-coverage problem. Those discoveries may justify broader changes to the search rather than simply inserting two references at the end.

Common Mistakes When Checking References of Included Studies

Use Included Studies as a Structured Check on the Search You Already Conducted

Reference list checking is most useful when it is connected to a clear search purpose.

You may be trying to recover studies that terminology-based searching missed, trace older literature, improve the sensitivity of a systematic search, or test whether your primary strategy appears to have captured the relevant evidence adequately.

Investigate Every Genuine Miss

An eligible paper found only through reference checking is diagnostic evidence.

Check whether the paper was indexed in your databases. Examine its terminology. Look at its subject headings. Review any date, language, publication-type, or field restrictions. Determine whether a search translation error or database-specific syntax problem affected retrieval.

Sometimes there is nothing reasonable you could have changed. A paper may simply fall outside the databases you searched. In other cases, one missed study reveals a terminology problem that could affect an entire branch of the literature.

Keep Reference Checking Separate From Casual Reading

Researchers naturally notice interesting references while reading. There is nothing wrong with following them.

For a systematic search, however, distinguish this exploratory behavior from the planned citation-searching procedure. A reproducible method needs to explain which seed references were searched, what direction was used, how references were retrieved and screened, and whether iterations occurred.

Record Enough Information to Report the Search Transparently

TARCiS recommends reporting the seed references, direction of citation searching, search dates where applicable, number of iterations, citation indexes or tools used, relevant deduplication procedures, and screening method. PRISMA-S also provides reporting guidance for citation searching.

Manual reference list checking does not require a citation-index search date in the same way an electronic forward citation search does, but the method should still be described clearly enough for readers to understand what was checked.

Know When Another Iteration Is Worth the Effort

Finding one or more new eligible studies creates a legitimate reason to consider another round. It does not mean the citation network must expand forever.

Successive iterations often produce increasing duplication and fewer unique eligible studies. The review topic, sensitivity of the primary search, objective of the literature search, previous yield, and screening burden all matter.

The broader question of how far citation chaining should continue before it becomes unproductive becomes especially important once newly included studies begin generating new seeds.

Before Checking the References of Included Studies, Confirm Your Method

Questions About Searching Reference Lists of Included Studies

Included Studies Are Some of Your Strongest Citation-Searching Seeds

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.