Manuel B. Garcia

Manuel B. Garcia serves as the Senior Director for Educational Technology and Digital Learning at FEU Institute of Technology, Manila, Philippines. Read More

Contact Info

1607, FEU Tech Building,
P. Paredes St, Sampaloc,
Manila, Philippines
mbgarcia@feutech.edu.ph

Follow Me

Part 4 · Guide 181 of 217

Which Study Decisions Should Be Fixed Before Data Collection Begins?

Before data collection begins, researchers should usually settle decisions that determine who or what will be studied, what evidence will be collected, and how the primary questions will be answered. The aim is not to eliminate legitimate flexibility but to prevent important choices from being influenced unnecessarily by emerging data.

By Manuel B. Garcia
181
Decisions to Fix Before Data Collection Guide 181 of 217
01 · The Question

What Needs to Stop Being Provisional Before Data Collection?

Early study planning is supposed to be flexible. You may compare sampling strategies, revise instruments, pilot procedures, reconsider variables, or debate several analytical approaches. At some point, however, the study moves from planning into evidence generation.

That transition matters. Once data begin accumulating, researchers can become aware of distributions, patterns, group differences, unexpected observations, or preliminary results. Decisions that were methodologically neutral beforehand may then become entangled with knowledge of what the data appear to show.

Which decisions should therefore be settled before collection begins? The answer depends on the design, but the strongest candidates are decisions that define the study's population, evidence, procedures, primary outcomes or questions, and planned approach to analysis.

02 · The Short Answer

Fix Decisions That Define the Evidence and Its Primary Interpretation

In Brief

Before data collection begins, researchers should generally settle the consequential decisions that determine who or what enters the study, what will be measured or observed, how data will be collected, which outcomes or questions are primary, and how the main research questions will be analyzed or interpreted.

Not every decision must be permanently fixed at the same moment. Some methodologies legitimately require adaptation, and unforeseen circumstances can justify amendments. The key is to prespecify decisions that could otherwise be influenced by emerging data and to document any later changes transparently.

03 · What You Need to Know

The Decisions That Usually Need to Be Settled Early

Fix the research questions, objectives, and primary hypotheses

The study needs a stable statement of what it is principally trying to answer. If the central research question changes after researchers begin seeing the data, the resulting analysis may still generate a useful finding, but it should not automatically be presented as though that had been the original question.

Where hypotheses are appropriate, specify the primary hypotheses before relevant results are known. Exploratory questions can coexist with confirmatory ones, but the distinction should remain visible.

This is also the point at which the research team should confirm that the protocol actually matches the research questions, objectives, and hypotheses. A beautifully prespecified analysis is not helpful if it answers a different question.

Define who or what is eligible for the study

Eligibility criteria determine the population or material from which the evidence is drawn. In participant research, this may include inclusion and exclusion criteria. In other studies, it may involve rules for selecting records, documents, schools, organizations, observations, specimens, cases, or datasets.

These criteria should generally be established before researchers know which inclusion rules produce more favorable findings. Otherwise, apparently methodological exclusions can become a way of reshaping the dataset after its patterns are visible.

This does not mean that every unforeseen eligibility problem is prohibited from correction. If a criterion proves unsafe, infeasible, ambiguous, or scientifically inappropriate, an amendment may be warranted. The important issue is transparency about what changed and why.

Set the sampling and recruitment strategy

Decide how eligible units will be identified and selected or how participants will be approached and recruited. Where relevant, specify the sampling frame, sampling technique, strata or clusters, recruitment channels, target sample size, allocation ratios, stopping rules, or other design-specific features.

Sample-size justification should also be completed when required by the methodology. For some quantitative designs, this may involve power or precision calculations. Other designs use different principles for determining an appropriate sample.

Sampling cannot be planned independently of measurement and analysis. The sampling, measurement, data collection, and analysis plans need to operate as a coherent system.

Define the primary outcomes, variables, constructs, or phenomena

Decide what evidence will represent the concepts in the research question. In quantitative studies, this may require identifying primary and secondary outcomes, predictors, exposures, covariates, and operational definitions. In qualitative research, it may involve specifying the phenomenon of interest, sampling logic, domains of inquiry, and initial data-generation strategy without prematurely fixing findings that should emerge through analysis.

The distinction between primary and secondary outcomes is especially important when several plausible outcomes could be emphasized. SPIRIT 2025 requires randomized trial protocols to define primary and secondary outcomes, including the specific measurement variable, analysis metric, aggregation method, and time point for each outcome.

Choosing which outcome matters most after discovering which one produces the strongest result undermines the informational value of calling it "primary."

Choose or define the measurement approach

Before collecting substantive data, identify how key variables or constructs will be measured. This can include instruments, scales, tests, observation systems, interview procedures, devices, administrative records, coding schemes, or laboratory methods.

Important details may include which instrument version will be used, how scores are calculated, who administers the measure, when it is administered, what language or adaptation is used, and what procedures ensure consistency.

Pilot testing can legitimately lead to revisions before the main study. That is one reason pilot work should be distinguished from the definitive data collection it is intended to prepare.

Set the main data collection procedures

The protocol should establish what participants or study units will experience and how observations will be generated. Relevant decisions can include procedure sequence, intervention or exposure conditions, assessment schedule, interview format, follow-up intervals, instructions, recording methods, and responsibilities of research personnel.

If several people or sites collect data, greater procedural specification may be needed to avoid systematic differences in implementation. This is where an adequately detailed research protocol becomes operational rather than merely descriptive.

Decide how important exclusions and problematic observations will be handled

Researchers frequently encounter incomplete questionnaires, protocol deviations, failed measurements, duplicate records, ineligible cases, attrition, missing observations, and unusual values. Not every situation can be anticipated, but foreseeable decision rules should be considered before researchers know how those rules affect the results.

For example, if a questionnaire contains several multi-item scales, determine whether partially completed scales will be scored and under what conditions. If an analysis excludes participants who fail to complete follow-up, consider in advance whether that exclusion is methodologically defensible and what alternative handling of missing data may be required.

Prespecify the primary analytical approach

The analysis plan should explain how the main research question will be answered using the data that will actually be collected. The necessary specificity varies by methodology and study design.

For quantitative research, consequential decisions can include the primary statistical model or test, outcome definition, analysis population, covariates, treatment of missing data, transformations, multiple-testing adjustments, subgroup analyses, and sensitivity analyses where applicable. SPIRIT 2025 guidance emphasizes that prespecifying analyses can reduce the risk of selectively reporting the most interesting results from multiple possible analytical approaches.

For qualitative research, prespecification does not mean determining themes before reading the data. It means documenting the methodological and analytical approach, such as how material will be prepared, coded, compared, interpreted, and reviewed, while preserving the forms of responsiveness that the methodology legitimately requires.

Prespecifying the analytical process Deciding in advance how evidence will be approached, processed, and evaluated.
Predetermining the findings Deciding in advance what patterns, themes, relationships, or conclusions the data are supposed to produce.

The former supports transparency. The latter is not the purpose of a protocol.

Set ethical and data-protection procedures before participants or sensitive data are involved

Decisions affecting informed consent, privacy, confidentiality, access to identifiable information, participant burden, foreseeable risks, compensation, withdrawal, data security, and other protections should be addressed before the relevant research activity begins.

Where ethics committee or institutional review board approval is required, the approved study documents govern what researchers are authorized to do. For randomized trials, SPIRIT 2025 states that the full protocol must be submitted for research ethics committee or institutional review board approval before participants are enrolled.

Applicable requirements differ across jurisdictions, institutions, and study types, so researchers should verify the rules governing their own work.

Decide how protocol changes will be recognized and documented

Prespecification loses much of its value if the original plan can simply be overwritten. Establish how protocol versions, amendments, dates, approvals, and reasons for changes will be recorded.

SPIRIT 2025 describes randomized trial protocols as living documents that may be formally amended and recommends a transparent audit trail across versions. The same general documentation principle is useful more broadly whenever changes to important methodological decisions need to be distinguished from the original plan.

A clear protocol version and change history preserves both plans rather than forcing researchers to pretend that the first plan was perfect.

Not everything belongs on the fixed side of the boundary

Some decisions genuinely depend on what happens during the study. Qualitative sampling may evolve in response to emerging analytical needs. A recruitment strategy may need adjustment if an intended channel becomes unavailable. A planned statistical model may require modification if its assumptions are seriously violated.

The important distinction is between justified responsiveness and result-driven flexibility. Instead of trying to eliminate all adaptation, identify which research decisions can legitimately remain flexible and define the conditions governing that flexibility where possible.

Watch Out

"We will decide after looking at the data" is not a neutral plan when the decision could determine which result becomes the headline finding. If a choice can reasonably be made before relevant results are known, prespecifying it usually provides a clearer record of the study's original intent.

04 · A Practical Example

What Should Be Settled Before Launching a Student Survey?

Hypothetical Example

Generative AI use and academic writing performance

A researcher plans to investigate whether generative AI use is associated with academic writing performance among undergraduate students. Before launching the survey, several plausible choices remain.

Research question Define whether the study concerns any generative AI use, frequency of use, particular academic uses, or another exposure.
Participants Specify which undergraduate students are eligible, how they will be recruited, and the intended sampling strategy and sample size.
Primary measures Define the primary measure of generative AI use and the primary indicator of academic writing performance before seeing which operationalization produces the strongest relationship.
Data quality rules Establish how duplicate, incomplete, ineligible, or otherwise unusable responses will be identified and handled.
Primary analysis Specify the main model or analytical procedure and any planned covariates that are justified by the research question and design.
Secondary exploration Identify additional analyses that may be exploratory rather than allowing them to replace the primary analysis after results are known.

Suppose the researcher later discovers an unexpected association between one specific type of AI use and performance. That finding can still be explored and reported. The protocol simply allows readers to see that this was a later exploratory observation rather than the study's original primary test.

05 · What Researchers Often Get Wrong

Common Misunderstandings About Fixing Study Decisions

Misconception

Everything Must Be Irreversibly Fixed Before the First Data Point

No. Some decisions are legitimately adaptive, and unexpected circumstances can require protocol amendments. The objective is not inflexibility. It is to identify decisions whose timing matters and prevent knowledge of emerging results from silently determining choices that could have been made beforehand.

Misconception

Only the Statistical Test Needs to Be Prespecified

No. Eligibility, sampling, outcomes, measurements, exclusions, data collection procedures, analysis populations, and missing-data rules can all affect results. Prespecifying a test while leaving every preceding decision open does not create a fully specified study.

Misconception

A Decision Is Prespecified as Long as It Was Made Before Analysis

Not always. Researchers may learn important information while collecting, cleaning, or inspecting data before formal analysis begins. The relevant timing depends on what information could influence the decision. For some choices, documentation before data collection is appropriate; for others, the critical boundary is before access to particular outcomes or comparative results.

Misconception

Changing a Prespecified Decision Automatically Invalidates the Study

No. A justified amendment can improve a study. The problem is not change itself but unexplained or result-contingent change that is presented as though it were original. Record what changed, why, when, and whether relevant data were already available.

Misconception

Prespecification Prevents Exploratory Research

It does not. Researchers can conduct exploratory analyses and follow unexpected findings. Prespecification helps distinguish those discoveries from analyses designed in advance to test the study's primary questions. Both can be valuable when reported according to what they actually are.

06 · What This Means for You

Ask Whether Seeing the Data Could Change the Decision

A practical way to decide what should be fixed early is to ask whether knowing something about the emerging data could tempt, consciously or otherwise, a different choice. The greater the potential for that choice to alter the study's principal result or interpretation, the stronger the case for deciding and documenting it beforehand.

A simple decision framework

If the decision determines who or what enters the dataset
Set the eligibility, sampling, recruitment, or selection rule before applying it to substantive study data whenever feasible.
If the decision determines what counts as the primary evidence
Define the primary outcome, variable, measure, phenomenon, or time point before seeing which option produces the preferred finding.
If the decision changes how the primary question will be analyzed
Prespecify the main analytical approach and important decision rules before relevant results are known.
If the methodology genuinely requires adaptation
Document the planned source of flexibility and the reasoning or criteria that will guide later choices.
If circumstances require a change after the protocol has been finalized
Preserve the original plan and document the amendment rather than silently replacing it.

Before launching the study, it can also be valuable to have the protocol examined by the people whose expertise is needed to identify unresolved decisions. The appropriate reviewers of a study protocol will depend on its design, risks, and institutional context.

07 · A Quick Checklist

Before Data Collection Begins, Check What Has Been Settled

Before collecting substantive study data, check:
The primary research questions, objectives, and hypotheses where applicable are clearly stated.
Eligibility, sampling, recruitment, selection, and sample-size decisions are sufficiently specified for the design.
Primary outcomes, variables, constructs, phenomena, and relevant time points are defined.
Key instruments, measures, data sources, intervention procedures, or data-generation methods have been selected and documented.
The main data collection sequence and responsibilities are sufficiently clear for consistent implementation.
Foreseeable rules for exclusions, incomplete observations, missing data, and other important data-quality issues have been considered.
The primary analytical approach corresponds to the research questions and the evidence that will actually be collected.
Ethical, privacy, consent, data-protection, and approval requirements are addressed before the relevant activities begin.
Any decisions intentionally left flexible are identifiable and methodologically justified.
A process exists for recording later protocol amendments without erasing the original plan.
08 · Frequently Asked Questions

Questions About Prespecifying Study Decisions

Does everything have to be decided before data collection starts?

No. The appropriate boundary depends on the methodology and decision. Some research legitimately evolves during data collection. The priority is to settle decisions that should not depend on knowledge of emerging results and to identify legitimate flexibility explicitly.

Can I change my research question after data collection begins?

You may discover new questions worth investigating, but distinguish them from the question that originally motivated and structured the study. A later question can support valuable exploratory analysis without being presented as though it had been prespecified.

Should I decide all covariates before collecting data?

For a prespecified primary analysis, important covariate decisions should generally be established in advance or governed by predefined criteria. The appropriate strategy depends on the design and analytical framework. Additional exploratory models can still be conducted and identified as such.

What if my planned statistical analysis turns out to be inappropriate?

Do not use an inappropriate analysis merely because it was prespecified. Document why the original approach became unsuitable, when that became apparent, what alternative was used, and whether the decision was made with knowledge of relevant results. Prespecification supports transparency; it does not override sound methodology.

Does this apply to qualitative research?

Yes, but not by forcing qualitative inquiry into a rigid quantitative model. Researchers can specify the methodological approach, sampling logic, initial data-generation procedures, analytical framework, and researcher role while preserving forms of adaptation that are legitimate within the chosen methodology.

Is a written protocol enough to prove that a decision was made in advance?

A dated protocol creates useful documentation, but stronger time-stamped mechanisms may be appropriate when establishing prior intent is especially important. Depending on the study, these can include ethics submissions, registries, repositories, preregistration, or controlled protocol-version records.

09 · The Bottom Line

Settle the Decisions That Should Not Depend on the Results

The Bottom Line

Before data collection begins, researchers should usually fix the consequential decisions that define the study population, primary questions, evidence, measurements, procedures, and main analytical approach, especially when those choices could otherwise be influenced by emerging data.

Prespecification does not prohibit justified adaptation or exploration. It creates a clearer record of what was planned, what remained legitimately flexible, and what changed later, allowing the study to adapt without rewriting its methodological history.

10 · Sources and Further Reading

Authoritative Guidance on Prespecified Study Decisions

11 · Cite this Guide

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.

Has the Field Guide helped your research?

If a guide helped clarify a question, inform a research decision, or move your work forward, I would love to hear about your experience. Your story may also help other researchers discover the Field Guide.

Share Your Experience
Takes only a few minutes