Should You Test Your Planned Analysis on Pilot Data?

If You Have Pilot Data, Should You Run the Analysis Planned for the Main Study?

A pilot can produce a dataset that looks remarkably similar to the one expected from the main study. The variables are there. The outcome can be calculated. The statistical model can be fitted. Software will happily produce coefficients, confidence intervals, and p-values, whether or not the study was designed to make those numbers meaningful.

So should you run the planned analysis?

Often, yes, if your purpose is to test whether the analytical workflow works. Pilot data can expose coding errors, unexpected data structures, scoring problems, missingness, linkage failures, impossible model specifications, or assumptions about the dataset that do not survive contact with actual observations.

That is quite different from treating the resulting pilot estimates as an early answer to the main research question.

Test the Analytical Workflow, Not the Main Hypothesis by Default

There Are Two Very Different Meanings of “Testing the Analysis”

First Distinguish the Analysis Pipeline From the Research Hypothesis

Researchers can mean two different things when they say they want to test an analysis.

The first is procedural: can the data produced by the study be transformed into the variables, scores, datasets, models, and outputs specified in the analysis plan?

The second is substantive: do the pilot data support the hypothesis or show an effect?

The first can be an important pilot objective. The second generally requires much greater caution when the pilot was designed primarily to assess feasibility.

Test Whether the Variables Arrive in the Form You Expected

An analysis plan is usually written before the final dataset exists. Researchers make assumptions about variable names, coding, ranges, response categories, time points, identifiers, and file structures.

Pilot data can show whether those assumptions are correct.

A binary variable expected to contain 0 and 1 may arrive as text labels. Dates may be stored inconsistently. A questionnaire may export each response as a separate string. A repeated-measures system may produce data in wide format when the planned analysis expects long format. Different sites may code the same category differently.

None of these problems is statistically glamorous, but each can derail an analysis remarkably efficiently.

Test Scoring and Derived Variables

Many analyses depend on variables that are not recorded directly. Questionnaire scores may require reverse coding and summation. Composite outcomes may combine several measurements. Dates may be converted into durations. Categories may be derived from multiple responses.

Run these procedures on preliminary data and inspect the results. Check minimum and maximum possible scores, missing-item rules, reverse-coded items, units, impossible values, and whether derived variables behave as expected.

A scoring error identified after ten pilot records is inconvenient. The same error embedded throughout a definitive analysis is considerably less charming.

Test Whether Data From Different Sources Can Be Linked

Modern studies often collect information from several systems: surveys, electronic records, learning platforms, laboratory systems, devices, intervention logs, or administrative databases.

The analysis may depend on combining them correctly.

Pilot data can reveal inconsistent identifiers, duplicate records, mismatched time stamps, incompatible formats, missing linkage fields, or unexpected one-to-many relationships. Testing linkage is therefore part of determining whether the data-collection and management workflow is practical.

Use Pilot Data to Discover Missingness Before It Becomes a Full-Study Pattern

Analysis plans often specify how missing data will be handled, but pilot data can reveal where missingness actually arises.

You may discover that one questionnaire item is routinely skipped, a particular time point has poor completion, a device fails to transmit certain observations, or a branching rule accidentally prevents a subgroup from seeing an essential question.

At this stage, the most important response may be to fix data collection rather than immediately develop a more sophisticated missing-data model.

Missing data can be an analytical problem, but in a pilot they may first be evidence of a procedural problem.

Test Whether the Planned Model Can Be Fitted

Running the planned model can expose mismatches between the analysis plan and the data structure. Variables may have insufficient variation. Categories may be extremely sparse. A repeated-measures structure may differ from what was anticipated. Clusters may be too few for a planned approach. Software syntax may fail because the data are formatted incorrectly.

These findings can help researchers refine coding, data-management procedures, software scripts, or aspects of the analysis plan before the definitive dataset arrives.

However, failure to fit a complex model in a very small pilot does not necessarily mean the model will fail in the much larger main study. The pilot may simply contain too few observations or events. Distinguish a structural problem from a small-sample limitation.

Test Analysis Code Before You Need It

A pilot offers an opportunity to develop and debug analysis code early.

Researchers can test data import, validation, recoding, exclusions, scoring, transformations, model fitting, diagnostics, tables, figures, and export procedures. Reproducible scripts can then be revised and documented before the main analysis.

Simulated data can complement pilot data here. Simulation is especially useful for testing scenarios that the small pilot does not contain, such as rare categories, particular patterns of missingness, boundary values, or larger datasets.

Pilot data provide realism; simulated data provide controlled coverage. Used together, they can expose different weaknesses.

Check Whether the Analysis Plan Requires Information You Forgot to Collect

This is one of the most valuable reasons to test the workflow early.

You may discover that a planned adjustment requires a variable absent from the case-report form, that longitudinal records cannot be ordered because timestamps are missing, that a clustering variable is not retained, or that the analysis requires distinguishing events that the database records under one code.

If the omission is discovered before the main study, the collection system may still be corrected. If discovered after data collection, no statistical technique can reliably recreate information that was never recorded.

Do Not Let the Pilot Become an Unplanned Interim Effectiveness Test

Once researchers fit the main model, they will inevitably see the estimated effect. That creates a temptation to decide whether the definitive study is promising based on the pilot result.

For a feasibility-focused pilot, this is generally inappropriate. The pilot sample is usually selected to answer feasibility questions, not to estimate the substantive effect with sufficient precision or test the main hypothesis with adequate power.

A statistically significant result can be unstable. A non-significant result may reflect insufficient information rather than absence of an effect. The analytical software does not know why the sample size was chosen.

Interpretation should therefore respect what the pilot study can and cannot legitimately establish.

Be Careful About Using Pilot Effect Estimates for Main-Study Sample Size

A common reason for analyzing pilot outcomes is to obtain an effect estimate for the definitive sample-size calculation. Small pilot estimates can be highly uncertain, making direct use of the observed effect problematic.

An unusually large pilot effect can lead to an underpowered definitive study if the calculated sample becomes too small. An unusually small effect can produce an unnecessarily large target or discourage worthwhile research.

Where possible, sample-size planning should incorporate the best relevant external evidence and a scientifically meaningful target effect. Pilot data may inform other design parameters, but uncertainty should be considered explicitly.

Changes to the Analysis Plan Need to Be Documented

Testing the analytical workflow may reveal a legitimate reason to modify the planned analysis. Perhaps a variable is structured differently than expected, a scoring procedure needs clarification, or the original model cannot represent the data-generating structure appropriately.

Changes made because of pilot findings should be documented, justified, and finalized before the definitive analysis where possible. The implications depend partly on whether pilot data remain separate from the main study or contribute to it.

If pilot observations may later be included in the definitive dataset, using those same data to make outcome-informed analytical decisions raises additional concerns. The distinction between an external pilot and an internal pilot becomes important, as does the question of whether pilot data will contribute to the main analysis.

Running the Planned Analysis Without Treating It as the Final Answer

The pilot analysis has been highly useful without answering the main substantive question. Its value lies in preventing the same analytical and data problems from reaching the definitive dataset.

Common Mistakes When Analyzing Pilot Data

Use Pilot Analysis as a Stress Test for the Data Pipeline

Run enough of the intended analysis to determine whether the path from raw data to final output works. The objective is to discover what would otherwise surprise you after the main study has finished collecting data.

Testing analysis early is part of a broader principle: study components can be tested before full implementation when doing so resolves consequential uncertainty. The analysis pipeline should not be exempt merely because it occurs near the end of the research process.

Before the Main Study, Stress-Test Your Analysis Workflow

Frequently Asked Questions About Analyzing Pilot Data

Use Pilot Data to Debug the Analysis, Not to Pretend the Main Study Has Already Happened

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.