03 · What You Need to Know
How to Tell Legitimate Flexibility From Result-Driven Analysis
Pre-Specification Is Not the Same as Predicting the Future
Advance analysis planning asks you to make decisions using the information reasonably available before the relevant results are known. It does not require pretending that future data characteristics are already known.
You can define the research question, primary outcome, principal comparison, variable roles, unit of analysis, and intended analytical strategy before collecting data. You may also know the broad structure of the observations, such as whether measurements will be repeated or participants will be clustered within sites.
Other features may remain uncertain. You may not know how much outcome data will be missing, whether a numerical algorithm will converge, whether a rare category will contain enough observations for a planned analysis, or whether an unforeseen data-quality problem will arise.
The analysis plan should specify what can reasonably be decided before collection while acknowledging genuine uncertainty rather than concealing it.
Core Confirmatory Decisions Usually Deserve Greater Advance Specification
The more consequential a decision is to the study's main confirmatory claim, the stronger the reason to specify it before the relevant results can influence the choice.
Depending on the study, this may include the primary outcome, principal research question or hypothesis, analytical population, primary comparison or estimand, main statistical model, treatment of important covariates, multiplicity strategy, and major missing-data assumptions.
Formal requirements vary by field. In clinical trials, ICH E9 states that the principal features of the statistical analysis should be described in the protocol and that the principal statistical analysis should be specified before breaking the blind. The precise regulatory standard should not be transplanted mechanically into every discipline, but the underlying logic is useful: decisions carrying the main evidential burden should not ordinarily be selected according to which result they produce.
Some Choices Can Be Specified as Contingencies
A decision does not need to be completely fixed to be planned.
Suppose a proposed model could fail to converge under particular circumstances. The analysis plan can identify the primary model and describe a defensible alternative if convergence cannot be achieved. If a measurement has known limits of detection, the plan can describe how values below those limits will be handled. If missingness exceeds a meaningful threshold or particular patterns arise, additional analyses may be planned.
The key is to define the reason for switching methods rather than leaving the researcher free to choose after comparing which result is more favorable.
Planned contingency
An alternative procedure linked in advance to a methodological condition or problem that may arise.
Result-driven choice
Selecting among analyses because one produces a more desirable effect estimate, p-value, confidence interval, or substantive conclusion.
Assumption-Dependent Decisions May Need Some Flexibility
Analytical methods rely on assumptions, and some relevant data characteristics cannot be fully evaluated before observations exist. This can justify conditional decisions.
However, “we will test assumptions and choose an appropriate analysis” may still leave too much discretion. The plan should identify which assumptions matter, how they will be assessed, and what kinds of departures would meaningfully challenge the primary method.
It is also unwise to let one preliminary significance test mechanically dictate the entire analysis. Assumption assessment may require graphical diagnostics, substantive reasoning, knowledge of a procedure's robustness, and consideration of the consequences of departures.
The objective is not to eliminate judgment. It is to prevent the observed substantive result from becoming the hidden decision rule.
Data Cleaning Requires Rules but Cannot Anticipate Every Error
No analysis plan can enumerate every typo, impossible date, duplicate record, coding error, or data-entry problem that might occur.
It can nevertheless establish principles for data cleaning. Researchers can define valid ranges, identify logically impossible combinations, describe duplicate-handling procedures, document corrections, preserve raw data, and specify consequential exclusion rules where possible.
Unexpected anomalies can then be resolved using those principles and documented. Flexibility is appropriate because the exact errors are unknowable, not because researchers should be free to remove observations that make the results inconvenient.
Missing Data Often Require Both Advance Planning and Adaptation
You may know that missing data are possible without knowing their eventual amount, pattern, or causes.
A plan can specify procedures to reduce missingness, how missingness will be summarized, the primary analytical approach, important assumptions, and sensitivity analyses relevant to those assumptions. Once the data are available, additional investigation may be warranted if the observed pattern differs materially from what was anticipated.
CONSORT 2025 asks trial reports to describe methods used to handle missing data, and its explanatory guidance emphasizes the assumptions underlying those methods and sensitivity analyses where appropriate. Other designs may require different approaches, but the broader principle remains: uncertainty about future missingness justifies planning contingencies, not ignoring the issue until the end.
Sensitivity Analyses Are a Legitimate Form of Planned Flexibility
A primary analysis may rely on assumptions that cannot be verified conclusively from the observed data. Sensitivity analyses allow researchers to ask whether the substantive conclusion changes under other reasonable assumptions or analytical choices.
This is different from searching through analyses for the most attractive answer.
A useful sensitivity analysis has a methodological purpose. It should challenge an assumption or choice important enough that a different conclusion would alter how the evidence is interpreted. When more than one analysis could answer the same research question, defining the primary and sensitivity roles in advance can make the resulting evidence much easier to interpret.
Exploratory Analysis Can Remain Open
Not every analysis needs to be pre-specified. Exploration is a legitimate part of research.
Researchers may discover unexpected patterns, formulate new hypotheses, investigate subgroups, examine alternative functional forms, or identify relationships that were not anticipated. Those analyses can generate valuable scientific ideas.
The crucial issue is how they are represented. An analysis inspired by observed data should not quietly acquire the status of a preplanned confirmatory test. Labeling analyses according to their role preserves the distinction between testing a prediction and discovering something worth investigating further.
Confirmatory analysis
Evaluates a question, hypothesis, or principal analytical target specified independently of the relevant observed results.
Exploratory analysis
Investigates patterns, questions, or hypotheses that may emerge through engagement with the observed data.
Qualitative Research May Require More Methodological Flexibility
In qualitative research, flexibility may be built into the methodology rather than treated as an exception.
Early interviews may shape later probes. Emerging categories may influence sampling. Coding frameworks may evolve. Researchers may deliberately pursue contradictory cases or developing theoretical ideas.
This does not mean that everything remains unspecified. A qualitative analysis plan can specify the analytical approach, data preparation, documentation, reflexivity, team roles, and logic of iteration while leaving substantive codes, categories, themes, or interpretations open where the methodology requires emergence.
The relevant distinction is between flexibility that serves the methodological approach and arbitrary changes that lack a defensible rationale.
Sequential Mixed-Methods Designs Also Need Conditional Decisions
Mixed-methods studies provide another clear example of legitimate flexibility.
In an explanatory sequential design, researchers may plan to select interview participants according to important or unexpected quantitative patterns. They cannot know exactly which pattern will emerge before the quantitative analysis occurs.
The mixed-methods integration plan can instead define how quantitative findings will guide qualitative sampling or questioning and what criteria will inform those decisions.
Again, the plan specifies the logic of adaptation rather than the unknowable outcome of that adaptation.
A Methodological Error Can Justify Changing a Prespecified Analysis
Pre-specification does not turn a mistake into a good method.
You may discover that the planned model is incompatible with the data structure, that a variable was incorrectly defined, that a statistical assumption was misunderstood, or that the proposed procedure does not estimate the quantity the research question actually asks about.
In such cases, knowingly retaining an inappropriate analysis merely because it was written earlier would prioritize procedural consistency over methodological validity.
Correct the problem. Then make the correction visible. Explain what the original plan specified, why it became inappropriate, what replaced it, and whether the change occurred before or after examining relevant outcome information.
Timing Matters When You Describe a Change as “Prespecified”
“Before analysis” can be ambiguous. A decision made after researchers have examined outcome distributions, group differences, or preliminary models is not equivalent to one made before those results were available.
When analytical chronology matters, document when important amendments were made and what information was available at the time. Registries, dated protocols, statistical analysis plans, version histories, or other records can help establish that sequence in contexts where formal documentation is appropriate.
The goal is not paperwork for its own sake. It is to allow readers to understand whether the data being evaluated could have influenced the method chosen to evaluate them.
Flexibility Becomes More Credible When It Is Transparent
A study does not become weak merely because the final analysis differs from the original plan. Real research encounters problems.
What undermines interpretability is silently changing outcomes, exclusions, models, transformations, subgroups, or hypotheses and then presenting the final configuration as though it had always been intended.
Transparent deviation reporting lets readers evaluate whether the change was methodologically necessary, exploratory, or potentially influenced by the observed results. That is considerably more informative than either pretending nothing changed or insisting that a flawed plan must be followed forever.
