03 · What You Need to Know
A Pilot Problem Is Evidence, Not an Embarrassment
First Determine What Actually Failed
“The pilot did not work” is too vague to guide a redesign.
Break the result into specific feasibility problems. Was recruitment too slow? Were too few people eligible? Did eligible participants decline? Did participants enroll but fail to complete follow-up? Was the intervention unacceptable? Could staff not deliver it consistently? Did measurement fail? Did the data-management system produce unusable records?
Different failures require different responses.
If recruitment is the problem, examine the recruitment pathway and identify where potential participants are being lost. If data collection is the problem, investigate which part of the data-collection workflow is impractical.
Do not redesign the entire study until you know which assumption has actually failed.
Separate a Fixable Procedure From a Fundamental Design Problem
Some pilot problems are relatively local. Instructions may need clarification. A questionnaire may be unnecessarily long. Appointment scheduling may need more flexibility. Staff may need additional training. A data-export setting may be incorrect.
Other problems reach much deeper. The eligible population may be far smaller than assumed. The intervention may be unacceptable to the intended participants. The primary outcome may not be collectable under realistic conditions. The comparison condition may be impossible to implement. Required resources may exceed what the research program can obtain.
Procedural problem
A problem that may be corrected while preserving the essential research question, population, intervention or exposure, outcomes, and design logic.
Fundamental design problem
A problem whose solution may require substantial changes to what is being studied, who is being studied, how the research question is answered, or whether the study remains viable.
This distinction is not always obvious, but it helps determine whether simple refinement is enough or whether the study requires more substantial reconsideration.
Ask Why the Problem Occurred Before Choosing a Solution
Suppose recruitment reaches only half the expected rate. Adding more recruitment sites might appear to be the obvious solution.
But what if the real problem is that most screened participants fail an unnecessarily restrictive eligibility criterion? Or eligible participants decline because the study requires too many visits? Additional sites may reproduce the same bottleneck rather than solve it.
Likewise, poor adherence to an intervention could reflect unclear instructions, excessive burden, technical problems, low acceptability, or an intervention that simply does not fit the intended context.
Use pilot process data, participant feedback, staff observations, and qualitative evidence where appropriate to understand the mechanism behind the feasibility problem.
Do Not Lower the Standard Simply to Make the Study “Feasible”
Once a pilot misses a feasibility target, researchers may be tempted to redefine success.
For example, a recruitment rate initially considered inadequate may suddenly be described as acceptable because abandoning or redesigning the study would be inconvenient. A high missing-data rate may be rationalized because “most participants completed something.”
This is one reason progression criteria can be useful when specified before pilot results are known. They make explicit what evidence would support proceeding, proceeding with modifications, or reconsidering the study.
Progression criteria should not be applied mechanically, but neither should they be rewritten casually after an unfavorable result.
Some Problems Can Be Solved Without Changing the Scientific Question
Many pilot findings lead to procedural refinement rather than a new study.
Researchers might simplify participant instructions, change reminder timing, improve staff training, revise appointment scheduling, repair a data system, alter recruitment materials, add an accessible data-collection mode, or redistribute tasks among research staff.
If these changes improve implementation without materially changing the population, intervention or exposure, comparison, outcomes, or underlying design logic, the research may remain substantially the same study.
The changes should nevertheless be documented, particularly when they arise from empirical pilot evidence.
Other Solutions Can Change What the Study Is Actually Asking
Suppose poor recruitment leads researchers to remove eligibility criteria. That may be reasonable, but it could alter the target population. Suppose participants cannot tolerate the intervention, so its intensity is cut in half. The revised intervention may no longer be the intervention originally proposed. Suppose the primary outcome is replaced because it is difficult to collect. The scientific question may shift accordingly.
Changes should therefore be evaluated for their scientific consequences, not merely their logistical convenience.
At some point, enough modification can transform the original study into a substantially different one. Determining how much a study can change after piloting before it becomes a different study requires looking at what those changes do to the research question and inferential structure.
Do Not Sacrifice Validity to Rescue Feasibility
A design can be made easier in ways that undermine its ability to answer the research question.
Removing difficult measurements may reduce burden but eliminate a necessary outcome. Relaxing eligibility criteria may improve recruitment but create a population inconsistent with the original question. Shortening follow-up may improve retention but make it impossible to observe the phenomenon of interest. Simplifying an intervention may make implementation easier while changing its active components.
The appropriate modification should improve feasibility while preserving enough scientific integrity for the study to remain worth conducting.
Watch Out
“Feasible” does not mean “easy to conduct.” A methodologically weak design can often be made operationally convenient. The goal is a study that is both workable and capable of producing evidence relevant to its research question.
Decide Whether the Revised Procedure Needs Another Test
Researchers sometimes identify a pilot problem, make a substantial modification, and proceed immediately to the main study. That assumes the solution works without testing it.
Minor corrections may not require another formal pilot. If an obvious software configuration error is repaired and can be verified directly, further participant testing may add little.
More consequential changes deserve greater caution. If recruitment strategy, intervention delivery, assessment sequence, participant burden, or another important process changes substantially, further preliminary testing may be justified.
The question is whether the modification introduces a new uncertainty capable of threatening the main study.
You May Need to Repilot Only the Changed Component
Further testing does not always mean repeating the entire pilot.
If recruitment worked but participant instructions failed, researchers might revise and retest the instructions while retaining evidence that the recruitment process is workable. If the data pipeline failed but all participant-facing procedures worked, simulated and preliminary data may be sufficient to test the corrected system.
This targeted approach follows the broader principle of testing the parts of the study that contain consequential uncertainty.
Sometimes the Correct Decision Is Not to Proceed
This can be the most difficult conclusion, particularly after substantial intellectual, financial, or institutional investment.
A pilot may show that no credible modification can make the proposed study workable within available resources. The population may be inaccessible. The intervention may be unacceptable. Essential measurements may be impossible. The necessary modifications may undermine the research question itself.
In such cases, stopping or substantially reconceptualizing the project may be more scientifically responsible than conducting a definitive study that is unlikely to produce interpretable evidence.
The resources already invested are sunk costs. They should not determine whether additional participants and resources are committed to an unworkable design.
An Unfavorable Pilot Result Can Be Scientifically Valuable
Pilot and feasibility findings can help other researchers understand barriers, refine future interventions, avoid impractical procedures, and develop better study designs. Reporting such findings transparently can therefore contribute to cumulative methodological knowledge.
Publication and reporting should not be reserved only for pilots that support progression. A well-designed pilot that demonstrates important infeasibility may answer its research question very successfully.
Document What Changed and Why
When a pilot leads to modification, preserve the decision trail.
Record the original procedure, the relevant pilot evidence, the interpretation of the problem, the modification, and the rationale for believing the change addresses it. Where another test is conducted, document its results as well.
This record improves transparency and helps distinguish evidence-informed refinement from undocumented protocol drift.
It also becomes particularly important if researchers are considering whether pilot participants or data can still contribute to the definitive study after the changes.
