Does Preregistration Automatically Make a Study More Rigorous?

If a Study Is Preregistered, Should You Trust It More?

“Preregistered” increasingly appears in abstracts, methods sections, journal badges, and discussions of open science. Because preregistration is intended to address problems such as undisclosed analytical flexibility and retrospective presentation of hypotheses, the label can easily acquire a broader meaning: preregistered research is rigorous research.

That conclusion goes too far.

A researcher can preregister a weak hypothesis, an inadequate sample, a poorly validated measure, an inappropriate analysis, or a design incapable of answering the research question. Recording a decision before seeing the results tells you something important about when the decision was made. It does not tell you that the decision was good.

Preregistration may contribute to rigor, but rigor still has to be earned through the research itself.

Preregistration Can Support Rigor, but It Does Not Create It Automatically

What Preregistration Can Improve and What It Cannot

Preregistration Primarily Adds Information About Decision Timing

The clearest contribution of preregistration is temporal transparency.

Researchers make decisions about hypotheses, outcomes, sampling, exclusions, transformations, covariates, statistical models, stopping rules, and reporting. Some choices are made before the relevant results are known. Others arise after researchers have learned something from the data.

A time-stamped preregistration can provide evidence about which decisions were specified prospectively. That can help readers distinguish prediction from post hoc explanation and planned analyses from later exploration.

This is an important contribution, but it is narrower than a general certification of research quality.

You need both questions.

A Bad Research Question Does Not Improve When It Is Preregistered

Suppose researchers preregister a hypothesis built on weak theory or a poorly defined construct. The time stamp establishes that the hypothesis existed before the outcome was known. It does not establish that the hypothesis is meaningful.

Likewise, preregistration cannot transform an unimportant research question into an important one or repair a mismatch between the stated question and the design used to answer it.

Prospective specification concerns the chronology of the claim. The substantive quality of that claim remains a separate matter.

A Weak Design Can Be Preregistered Perfectly

Imagine a causal claim investigated using a design that cannot adequately support causal inference. Every methodological detail could be preregistered before data collection, and the fundamental design limitation would remain.

The same applies to confounding, inappropriate comparison groups, inadequate controls, poor randomization procedures, serious attrition, or other threats to validity.

Preregistration may make the planned design transparent. It does not determine whether that design is capable of supporting the intended inference.

Preregistration Cannot Rescue Poor Measurement

A construct measured badly remains measured badly whether the instrument was selected before or after registration.

If a study uses an unreliable, invalid, poorly operationalized, or inappropriate measure, preregistration does not repair the measurement problem. At most, it can show that the measure was selected prospectively rather than chosen after researchers discovered which measure produced a favorable result.

Those are different virtues.

Advance selection can reduce one source of ambiguity while leaving the validity of the measurement itself entirely open to evaluation.

An Underpowered Study Can Be Preregistered

Researchers can preregister a sample size that is too small to estimate the effect of interest with useful precision.

Indeed, preregistration can document the sample-size rationale and stopping rule, but it cannot guarantee that the rationale is sound.

There is some empirical evidence that preregistration is associated with better planning in this area. Van den Akker and colleagues compared 193 preregistered psychology studies with 193 related non-preregistered studies and found that the preregistered studies more often reported power analyses and generally had larger samples.

That finding is interesting, but it should not be interpreted as proof that preregistration itself caused the difference. The authors note that researchers self-select into preregistration and may differ from non-preregistering researchers in other ways.

A Preregistered Analysis Can Still Be Statistically Wrong

Advance specification does not validate an analytical method.

A researcher can preregister an incorrect statistical test, inappropriate model specification, unjustified covariates, problematic treatment of missing data, or decision threshold that does not answer the intended question.

If the error becomes apparent later, researchers should not knowingly retain it merely to achieve perfect adherence. As discussed when considering deviations from a preregistered study, a justified methodological correction can be preferable to following a flawed plan.

What matters is that the change is visible and its consequences are considered.

Preregistration Can Encourage Better Planning

Although preregistration does not guarantee rigor, the act of writing a detailed plan may improve parts of the research process.

Researchers may discover before data collection that they have not clearly identified the primary outcome, connected a hypothesis to a specific analysis, established an exclusion rule, defined a stopping criterion, or decided how variables will be constructed.

Resolving these ambiguities early can improve planning.

This benefit depends on the quality of the preregistration process. Completing a vague form immediately before data collection may contribute far less than carefully thinking through the decisions that need prospective specification.

Preregistration Can Make Some Questionable Practices More Visible

Suppose a paper reports one primary outcome, but the preregistration identifies three. Or a hypothesis appears in the publication but not in the preregistration. Or the published exclusion criteria differ from those specified in advance.

The registration gives readers a reference point against which those discrepancies can be detected.

This is useful, but again it should not be overstated. Preregistration does not physically prevent selective reporting or undisclosed deviations.

In a study comparing 459 preregistrations with their corresponding psychology publications, van den Akker and colleagues found that more than half contained omitted preregistered hypotheses or added hypotheses, while approximately one fifth contained hypotheses whose direction changed between preregistration and publication.

The finding illustrates an important point: creating a preregistration and faithfully reporting its relationship to the final study are separate practices.

Empirical Evidence About Preregistration Is More Nuanced Than the Rhetoric Sometimes Suggests

It is tempting to infer that preregistration must reduce p-hacking, HARKing, statistical errors, and inflated effects because it is designed partly to constrain or reveal research flexibility.

The empirical evidence is not that simple.

Van den Akker and colleagues' comparison of preregistered and non-preregistered psychology studies did not find robust evidence that the preregistered studies had fewer positive results, smaller effect sizes, or fewer statistical inconsistencies. They did find larger samples and more frequent power analyses among preregistered studies.

Other meta-research has reported different patterns depending on the samples and study types examined. Some earlier comparisons that included Registered Reports found substantially lower proportions of positive findings among prospectively registered research than conventional publications.

These studies differ in design and cannot be collapsed into a simple verdict that preregistration either “works” or “does not work.” Ordinary preregistration, Registered Reports, researcher self-selection, registration quality, adherence, disciplinary practice, and publication processes can all matter.

The defensible conclusion is narrower: preregistration has plausible transparency and planning benefits, but broad claims that it automatically produces more rigorous or unbiased research exceed what the label itself establishes.

Specificity Matters

A preregistration that says “appropriate statistical analyses will be conducted” provides little constraint on later analytical choices.

A more specific registration may identify the primary outcome, exclusion criteria, model, covariates, inferential criteria, and relevant contingencies. Readers can then determine whether the final analysis corresponds to the advance plan.

Even then, specificity should not be confused with correctness. A highly specific but methodologically inappropriate plan may be worse than a well-justified change to that plan.

The best preregistration is not necessarily the longest or most rigid one. It is sufficiently specific to clarify consequential advance decisions while remaining methodologically defensible.

Adherence Matters Too

A rigorous preregistration process requires more than registration.

Researchers need to compare the completed study with the advance plan and report consequential deviations transparently.

If a preregistration specifies one analysis but the paper quietly reports another, the existence of the registration does not by itself solve the transparency problem.

Readers evaluating a preregistered paper should therefore examine the relationship between the registration and publication rather than stopping at the presence of a preregistration statement or badge.

Registered Reports Add a Different Kind of Safeguard

A Registered Report adds prospective peer review. The Stage 1 plan is evaluated by a journal before the main results are known, allowing reviewers to identify methodological weaknesses while researchers may still be able to correct them.

This can provide a stronger prospective quality-control mechanism than preregistration alone.

Even so, peer review does not guarantee correctness. A reviewed design can still contain limitations, implementation problems can emerge, and interpretation can remain contestable.

The difference between preregistration and a Registered Report is therefore relevant when evaluating what kind of prospective scrutiny the study actually received.

Rigor Is Multidimensional

A useful evaluation of rigor considers the entire chain from question to conclusion.

Preregistration touches several dimensions. It fully determines none of them.

Two Preregistered Studies Can Differ Dramatically in Rigor

Misconceptions About Preregistration and Research Quality

Use Preregistration as One Piece of Evidence About Research Quality

If you are conducting a study, preregister because prospective specification serves a useful methodological or transparency purpose, not because the word “preregistered” will make the study rigorous by declaration.

The useful question is not “Was it preregistered?” followed by a verdict. Ask instead: what uncertainty about the research process does the preregistration reduce?

That framing gives preregistration appropriate credit without asking it to solve problems it was never capable of solving.

How to Evaluate the Rigor of a Preregistered Study

Frequently Asked Questions About Preregistration and Rigor

Preregistration Can Strengthen the Process Without Certifying the Study

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.