03 · What You Need to Know
Think About Replication as an Investment in Reducing Uncertainty
The Value of Replication Depends on What the New Evidence Could Change
Replication is sometimes discussed as though every published finding ought to be independently repeated. In principle, independent evidence is fundamental to cumulative science. In practice, however, researchers cannot replicate everything.
The number of potential replication targets greatly exceeds the resources available to investigate them. Researchers therefore face a selection problem: which claims are worth testing again?
One useful starting point comes from Nosek and Errington's account of replication. They define a replication in terms of whether possible outcomes would provide diagnostic evidence about a claim from prior research. This shifts attention away from simply repeating procedures and toward what the new evidence could tell us.
That idea has an important consequence. If obtaining either a similar or a different result would barely change what anyone believes about the claim, the replication may have limited informational value. If either outcome could substantially alter confidence in an important claim, the case for replication becomes stronger.
Importance and Uncertainty Are Two Central Considerations
Research on replication target selection has proposed several ways of deciding which studies deserve scarce replication resources. There is no universally accepted formula, but two considerations recur: how important the claim is and how uncertain the existing evidence remains.
Importance
How consequential the claim is for theory, subsequent research, professional practice, policy, intervention, or other decisions relevant to the field.
Uncertainty
How much reasonable doubt remains about the claim given the quantity, quality, consistency, precision, and independence of the available evidence.
Isager and colleagues formalized a related concept as replication value, in which the usefulness of replicating a finding depends on its importance and uncertainty. Other work on replication target selection has identified a broader set of considerations, including value or impact, uncertainty, study quality, and the costs and feasibility of conducting the replication.
These should be treated as decision aids rather than a mechanical scoring system for every research project. What counts as important or sufficiently uncertain can depend on disciplinary priorities and the purpose of the research.
Replication Becomes More Useful When an Important Claim Has Little Independent Support
A finding can become influential surprisingly quickly. One study proposes an interesting relationship, later papers cite it, researchers incorporate it into theoretical arguments, and eventually the claim begins to sound more established than the underlying evidence warrants.
This is precisely where independent replication can be valuable.
If a consequential claim depends primarily on one original study or several closely related studies from the same research group, another independent test may provide information that the literature currently lacks. Publication and citation do not themselves constitute independent confirmation.
This does not mean the original study is presumed unreliable. The rationale is epistemic rather than adversarial: claims become more credible when they survive informative tests using new evidence.
The distinction also helps when deciding whether to replicate an existing claim or pursue a different research question. If the unresolved credibility of the existing claim is itself the important knowledge problem, replication may deserve priority.
Replication Can Be Particularly Valuable When Other Research Depends on the Finding
Some findings function as foundations. Researchers use them to justify new hypotheses, construct interventions, select variables, develop models, or make methodological decisions. The consequences of uncertainty therefore extend beyond the original paper.
Imagine that dozens of subsequent studies assume a particular effect exists but very few independently test the effect itself. If that assumption is unreliable, later research may be building on unstable evidence.
In such cases, replication can have substantial downstream value. Confirmatory evidence may strengthen the basis for further research. Evidence inconsistent with the original claim may encourage researchers to reconsider assumptions, identify boundary conditions, or redirect resources.
This is why citation count alone is an inadequate measure of replication priority. Citations can indicate influence, but they do not tell you whether the cited claim is actually central to later work, whether it remains uncertain, or what would happen if confidence in it changed.
Conflicting Evidence Can Make Another Replication More Informative
Suppose several studies investigate the same broad claim but produce inconsistent results. One finds a substantial effect, another a small effect, and another little evidence of the predicted relationship.
Another study may be useful, but simply adding one more estimate to the disagreement is not necessarily enough.
A stronger replication asks why the existing evidence conflicts and designs the new study accordingly. Perhaps previous studies differed in statistical power, measurement quality, participant characteristics, implementation fidelity, or theoretically meaningful contextual conditions. The replication can then be designed to discriminate among plausible explanations.
At that point, the project may involve more than repeating the original procedure. Depending on what is changed and what claim is being tested, you may need to determine whether the design is a direct or conceptual replication, or whether it has moved toward an extension.
A Surprising or Counterintuitive Finding May Deserve Replication, but Surprise Is Not Enough
Unexpected findings naturally attract attention. A result that challenges a well-supported theory, reverses conventional expectations, or suggests an unexpectedly large effect may reasonably motivate further testing.
But "this result is surprising" is not a complete replication rationale.
You still need to ask how important the claim is, how much evidence supports it, and whether another study can meaningfully reduce the uncertainty. A surprising result with little theoretical or practical consequence may be less valuable to replicate than an ordinary-looking result on which substantial research or practice depends.
Weak Evidence Can Increase the Need for Replication, but a Weak Study Is Not Automatically Worth Replicating
A study with a small sample, imprecise measurement, questionable analytical choices, or incomplete reporting may leave considerable uncertainty. That might seem to make it an obvious replication target.
Sometimes it does. Sometimes it does not.
If the underlying claim is important, a rigorous new test may be highly useful. But if the original research question itself has little scientific value, faithfully reproducing a weak study may simply reproduce its weaknesses. In other cases, the better project is a substantially improved test of the underlying claim rather than a close procedural repetition.
The decision therefore requires separating weaknesses in the original study from the value of the claim it attempted to test. This distinction becomes central when considering whether a weak study is actually worth replicating.
Practical or Policy Consequences Can Raise the Value of Independent Verification
The stakes associated with a claim also matter. If research is being used to shape an educational intervention, clinical practice, organizational policy, public program, or another consequential decision, uncertainty about the underlying evidence may carry real costs.
A replication can be especially valuable when decision-makers would act differently depending on whether the claim receives further support.
That does not mean every applied finding must be replicated before anyone can use it. Evidence requirements depend on the decision, risks, existing literature, alternatives, and costs of being wrong. Rather, practical consequences should be part of the judgment about whether reducing uncertainty is worth the investment.
Generalizability Questions Can Make Replication Useful Even After a Finding Has Recurred
A finding may have been replicated and still leave important questions unresolved. Perhaps most studies involve similar participants, institutions, countries, age groups, instruments, or experimental conditions.
Consistent findings within a narrow set of conditions do not establish that the effect generalizes everywhere.
Testing a claim under a meaningfully different condition can therefore provide new evidence about its boundaries. Nosek and Errington emphasize that successful replication across conditions that inevitably differ from the original study provides evidence relevant to generalizability, while inconsistent outcomes can indicate that the reliability of a finding is more constrained than previously understood.
This is also why an already successful replication does not necessarily close the matter. Whether another test is useful depends on what uncertainty remains, an issue that becomes particularly relevant when deciding whether to replicate a finding that has already replicated successfully.
Feasibility Matters Because Research Resources Have Opportunity Costs
A theoretically valuable replication can still be a poor practical choice if it cannot be conducted well.
Replication consumes participants, researcher time, funding, equipment, access to materials, and analytical effort. Those resources could have been used for another study. This opportunity cost is one reason replication target selection cannot be based solely on the importance of a finding.
Ask whether you can obtain an adequate sample, implement the relevant procedures competently, access necessary materials, measure the constructs appropriately, and conduct an analysis capable of providing an informative result. If essential features of the original work cannot be reproduced, determine whether the resulting differences still permit a meaningful test of the claim.
When exact reproduction is impossible, the problem is not automatically fatal. It does, however, require careful reasoning about what can change without making the replication uninterpretable.
The Best Replication Target Is Not Necessarily the Most Famous Finding
Famous studies are attractive replication targets because their influence is visible. But visibility and scientific priority are not synonymous.
A highly publicized finding may already have been examined extensively, while a less celebrated finding may quietly support an important body of work without much independent verification. Replication resources may generate more useful knowledge when directed toward the latter.
The relevant question is not "Which study will people recognize?" It is "Where would additional evidence have the greatest value?" This distinction is particularly important when choosing between a famous finding and one that genuinely matters to your field.
