03 · What You Need to Know
Distinguish Refinement From a Change in the Scientific Study
Changing a Study After a Pilot Is Often the Point
A pilot that identifies problems but leaves researchers unwilling to modify anything has limited practical value. Pilot and feasibility research is intended to inform whether a future study should proceed and what modifications may be needed before it does.
Progression decisions are therefore not necessarily binary. Researchers may proceed without important changes, proceed after modifying the design, undertake additional preliminary work, or decide not to proceed. Methodological guidance commonly recognizes this middle ground rather than treating every departure from the pilot protocol as failure.
Changes after piloting are consequently not suspicious by themselves. The relevant questions are why the changes were made, what they affect, and whether the revised design remains scientifically continuous with the study originally proposed.
Minor Operational Refinements Usually Do Not Create a New Study
Some modifications change how efficiently or clearly the protocol operates without materially changing the scientific question.
Examples might include clarifying participant instructions, improving reminder wording, changing appointment spacing, correcting software errors, strengthening staff training, improving recruitment materials, simplifying administrative forms, or reorganizing data-management procedures.
If participants remain drawn from essentially the same target population, receive the same relevant intervention or exposure, provide the same substantive outcomes, and contribute to the same underlying research question, such modifications may reasonably be understood as refinements of the same study.
That does not make them invisible. Important modifications should still be documented and, where required, reflected in protocol amendments, ethics approvals, registrations, operating procedures, or other study documentation.
Ask Whether the Research Question Has Changed
The research question is one of the strongest anchors of study identity.
Suppose a pilot reveals that one questionnaire is too burdensome, so researchers replace it with another validated measure of the same construct collected at the same intended time point. That may be a consequential measurement change, but the substantive question might remain largely intact.
Now suppose the researchers abandon the original outcome entirely and instead study a different construct because it is easier to measure. The research may no longer be asking the same question.
When evaluating continuity, compare what the original and revised studies are intended to establish. If a positive result from each version would support materially different scientific conclusions, the change is more than administrative refinement.
Changes to the Population Can Change the Meaning of the Study
Pilot recruitment problems often prompt reconsideration of eligibility criteria. Some criteria may turn out to be unnecessarily restrictive and can be revised without undermining the research question.
Other eligibility changes can redefine the target population. Expanding a study from first-year university students to all university students, from adults with one specific condition to a broader clinical population, or from one professional group to several may change who the findings are intended to describe.
The important issue is not simply that an inclusion criterion changed. Ask whether the revised population represents the same scientific target and whether the expected mechanisms, risks, measurements, intervention effects, or interpretation could differ materially.
If recruitment prompted the modification, return to the evidence about where the recruitment pathway actually failed before broadening eligibility simply to increase enrollment.
Changes to the Intervention or Exposure Can Be Fundamental
Pilot studies often refine intervention delivery. Researchers may adjust training, scheduling, reminders, delivery procedures, or other implementation features to improve fidelity and acceptability.
But changing the intervention's essential content, intensity, duration, mechanism, or mode of delivery may alter what is being evaluated.
Consider an eight-week intervention that participants find too burdensome. Reducing unnecessary administrative requirements while preserving the intervention may be a procedural refinement. Reducing the intervention from eight weeks to two weeks and removing several core components could create a meaningfully different intervention.
There is rarely a universal line separating the two. Researchers need to identify the intervention's essential components and determine whether the revised version still represents the treatment or program specified by the research question.
Changes to the Comparator Can Alter the Contrast Being Tested
The comparison condition helps define what a study can conclude.
Comparing an intervention with usual practice asks a different question from comparing it with an active alternative. Likewise, changing a wait-list control to an attention-matched condition may alter the interpretation of any observed difference.
A pilot may legitimately reveal that the original comparator is impractical or unacceptable. If it is replaced, however, researchers should recognize that the scientific contrast may have changed even if the intervention remains identical.
The study should therefore be evaluated as a system rather than by asking whether most protocol sections remained unchanged.
Changing the Primary Outcome Deserves Particular Scrutiny
A pilot can reveal that the intended primary outcome is difficult to collect, poorly completed, insensitive under the proposed conditions, or otherwise impractical.
Changing it may be necessary. The implications depend on what replaces it.
A different instrument measuring essentially the same construct may preserve much of the scientific question, although measurement properties and comparability still need consideration. Replacing the construct itself may change what constitutes success in the main study.
Timing also matters. Measuring the same outcome immediately after an intervention and six months later can support different conclusions about persistence or longer-term effects.
Outcome changes become particularly sensitive when researchers have inspected substantive pilot results before choosing the replacement. A change motivated by feasibility is methodologically different from selecting the outcome that happened to produce the most favorable pilot effect.
Changes to Follow-Up Can Change the Question Even When the Outcome Stays the Same
Researchers sometimes shorten follow-up because retention is poor. This may improve feasibility, but it can also alter the phenomenon being studied.
If the original question concerns whether an intervention has a sustained effect at twelve months, measuring only immediate post-intervention outcomes does not answer the same question more conveniently. It answers a different temporal question.
Before shortening follow-up, determine whether the original time horizon is scientifically essential. If it is, a more appropriate response may be to improve retention procedures rather than redefine the endpoint around what participants are easiest to retain for.
Changing Data Collection Can Be Operational or Scientific
Switching from paper to a well-tested electronic form may be primarily operational if the substantive measurement remains equivalent. Removing an essential assessment because it is difficult to administer is a different matter.
The distinction again depends on what the modification does to the evidence. Researchers should consider whether the revised procedure measures the same construct, uses compatible timing and conditions, and preserves the information required by the analysis.
When a pilot reveals practical problems, first determine which part of the data-collection workflow is actually causing them. A targeted correction may preserve far more of the original design than replacing the measurement strategy wholesale.
Changes to the Analysis Plan Can Range From Technical Corrections to New Questions
A pilot may expose legitimate analytical problems. Variables may require different coding, the dataset may have a structure different from what was anticipated, or the planned model may not appropriately represent repeated or clustered observations.
Correcting such problems does not necessarily change the study's substantive question.
By contrast, inspecting pilot outcomes and then choosing different subgroups, endpoints, models, or hypotheses because they produce more favorable results can change the inferential process and introduce bias.
This is why testing whether the analytical workflow works should be distinguished from using pilot outcomes to search for the analysis you prefer.
Several Small Changes Can Matter Collectively
Study identity should not be assessed one amendment at a time.
A slight expansion of eligibility may seem minor. So might a shorter follow-up, a revised intervention schedule, a different outcome instrument, and a new recruitment setting. Considered separately, each may appear defensible. Together, they may produce a study whose participants, intervention experience, measurement, and context differ substantially from the original proposal.
After a series of modifications, compare the complete original and revised protocols side by side. Ask whether a reader would reasonably regard them as investigating the same substantive question under sufficiently comparable conditions.
There Is No Universal Percentage of Acceptable Change
Counting changed protocol elements is a poor way to decide whether the study remains the same.
One change can be fundamental. Replacing the intervention may transform the study even if every other protocol section remains untouched. Ten administrative corrections may leave the scientific design virtually unchanged.
Operational continuity
The study procedures have been refined, but the core scientific question and inferential structure remain substantially intact.
Scientific discontinuity
The modifications materially change what is being studied, in whom, against what comparison, over what time, or what conclusion the resulting evidence can support.
The distinction requires judgment. That judgment should be transparent and tied to the scientific consequences of the changes rather than an arbitrary numerical threshold.
Major Changes May Need Further Preliminary Testing
Even a scientifically defensible modification can introduce new feasibility uncertainty.
If researchers substantially broaden the population, the revised recruitment process may behave differently. If intervention delivery changes, acceptability and fidelity may need reassessment. If the primary outcome is replaced, completion and data-management procedures may need testing.
The question is therefore not only whether the revised design remains conceptually related to the original. It is whether the new or modified components have sufficient evidence to support full-scale implementation.
When the original pilot has already shown that the design needs meaningful revision, focused retesting or a further pilot may be more defensible than assuming the revised solution will work.
The Extent of Change Affects Whether Pilot Data Can Still Be Used
Changes after piloting also matter if researchers hope to include preliminary participants in the definitive analysis.
If pilot participants experienced materially different eligibility rules, interventions, outcomes, timing, or procedures, treating their observations as interchangeable with main-study observations becomes harder to justify.
This is particularly relevant for external pilots, which are conducted separately from the definitive study and allow greater freedom to modify the future protocol. Internal pilots are different because initial participants are prospectively embedded within the definitive study and permitted adaptations should be considered in advance.
Before pooling anything, examine whether pilot participants or data remain compatible with the main study.
Approvals, Registrations, and Documentation Need to Follow the Revised Study
Whether a modification requires ethics review, protocol amendment, trial registration updates, funder approval, regulatory review, or other formal action depends on the study and governing requirements.
Researchers should therefore verify the applicable requirements with the responsible bodies rather than assume that describing something as a “pilot refinement” exempts it from review.
At minimum, maintain a clear record of what changed, why it changed, what pilot evidence motivated the change, when the change occurred, and whether further testing was conducted.
Watch Out
Do not use “the pilot showed we needed to change it” as a blanket justification for any modification. Pilot evidence can justify revision, but the revised design must still be scientifically defensible. A change that makes the study easier while preventing it from answering the intended research question solves the wrong problem.
