03 · What You Need to Know
Research Questions Do Not Usually Come With One Predetermined Study Attached
A research question constrains the kinds of evidence that can answer it, but it does not necessarily dictate one unique design. Methodological guidance repeatedly emphasizes that study-design selection should begin with the research question while also considering participants, resources, setting, ethics, feasibility, and the strengths and limitations of available designs.
Different designs may therefore address the same broad question from different angles or with different inferential strength. The important task is to identify exactly what you need to know and then compare credible ways of obtaining that evidence.
First Ask What Kind of Answer the Question Requires
Consider four superficially similar questions about an educational technology:
- How many students use the technology?
- How do students experience using the technology?
- Is technology use associated with academic performance?
- Does providing the technology improve academic performance?
These questions concern the same topic but require different kinds of evidence.
A descriptive survey might estimate patterns of use. Interviews or another qualitative approach might investigate students' experiences and interpretations. An observational design might examine an association between use and performance. A well-designed experiment may be appropriate when the aim is to estimate the causal effect of providing an intervention under conditions where randomization is ethical and feasible.
Changing the study can therefore change the answer you are capable of producing.
Before comparing designs, finish this sentence:
“To answer this research question, I need evidence that can show __________.”
If the blank is unclear, the research question may need further refinement before methodological optimization begins.
Do Not Confuse a Topic With a Research Question
“Artificial intelligence and student learning” is a topic. It does not tell you whether you need prevalence estimates, experiences, associations, causal effects, implementation evidence, longitudinal change, or something else.
A broad topic can support dozens of legitimate studies.
Methodological decisions become much easier once the question specifies what is being investigated, among whom or in what context, and what kind of relationship, description, comparison, process, or explanation is sought.
This is one reason frameworks such as FINER emphasize that research questions should be feasible, interesting, novel, ethical, and relevant. A question that cannot be investigated with available participants, expertise, resources, time, or methods may require refinement before a study is selected.
Compare Designs by Their Inferential Capability
Suppose you want to know whether an instructional intervention improves examination performance.
A cross-sectional survey asking students whether they believe the intervention helped them can answer a useful question about perceptions. It cannot, by itself, provide the same evidence about intervention effects as a design that compares outcomes under appropriately constructed intervention and comparison conditions.
Likewise, a retrospective analysis may be much easier than prospective follow-up, but the available records may lack important confounders, temporal information, or consistent measurement.
The easier study is useful only if it still answers the question you actually care about.
Methodological simplification
Removing unnecessary complexity while preserving evidence adequate for the research question and intended inference.
Methodological compromise
Making the study easier by removing information or design features that are necessary for the conclusion you still intend to make.
Ask Whether You Need to Collect New Data at All
Researchers often begin by imagining participants and data collection. Sometimes the relevant evidence already exists.
Suppose you want to estimate changes in university enrollment patterns over the last decade. If reliable institutional or national records already contain the necessary information, a new student survey may add burden without improving the answer.
Likewise, if the question concerns the overall state of evidence about an intervention and numerous relevant studies already exist, an evidence synthesis may be more informative than another small primary study.
This possibility deserves particular attention when substantial evidence may already exist and another primary study could add relatively little.
Before recruiting anyone, ask what existing datasets, records, publications, registries, archives, or other evidence could already address the question.
A Secondary-Data Study Can Be Easier, but Only If the Data Fit the Question
Existing data can dramatically reduce collection time and participant burden. They can also create a seductive methodological trap: allowing available variables to redefine the research question after the fact.
Suppose you want to study student engagement, but an institutional dataset contains only login frequency. Using those records may be efficient, but login frequency does not automatically represent the broader construct of engagement.
A secondary-data study becomes preferable when the available data adequately represent the population, variables, timing, comparisons, and other information the question requires.
If those conditions are uncertain, examine what poor-quality or unsuitable data would do to the proposed study before choosing efficiency over measurement quality.
A Cross-Sectional Study May Be Enough for a Descriptive Question
Researchers sometimes add longitudinal follow-up because it sounds methodologically stronger. But if the question is simply to estimate the prevalence of a characteristic at a particular time, repeated measurements may contribute little to the primary objective.
Conversely, if the question concerns change, incidence, temporal order, or developmental trajectories, a single cross-sectional observation may be inadequate.
Study-design guidance emphasizes matching design features to the question domain. There is no reason to incur the cost and attrition of longitudinal research when time is irrelevant to the question. There is equally little justification for choosing a cross-sectional design merely because longitudinal data are inconvenient when temporal evidence is essential.
A Retrospective Study May Sometimes Replace Prospective Follow-Up
Suppose you want to investigate whether an exposure predicts a later outcome. Prospective follow-up might initially seem necessary.
If high-quality historical records already document the exposure, relevant covariates, subsequent outcomes, and necessary timing, a retrospective design may answer the question much more efficiently.
But retrospective convenience comes with dependencies. Records were not necessarily collected for your research purpose. Variables may be missing or inconsistently measured. Relevant confounders may be absent. Selection into the dataset may create bias.
The alternative is therefore not automatically better. It is better only when its evidential limitations remain acceptable for the intended inference.
An Experiment Is Not Automatically the Best Design for Every Question
Randomized experiments can provide strong evidence for many causal questions because random allocation can reduce systematic baseline differences between comparison groups. That does not make randomized trials the universal “best” design.
They may be irrelevant to descriptive, diagnostic, prognostic, experiential, or many other questions. They can also be unethical, impractical, excessively costly, or poorly suited to interventions whose effects depend strongly on real-world context.
Methodological literature on alternatives to randomized trials emphasizes that important questions should not simply be abandoned because an RCT cannot be implemented. Instead, researchers should choose the strongest design that can feasibly answer the question while addressing plausible threats to validity.
The hierarchy should therefore be question first, design second.
A Qualitative Study May Answer Some Questions More Directly Than a Survey
Suppose you want to understand why doctoral students discontinue an online research-support program.
A large questionnaire might estimate how frequently predefined reasons are selected. But if the important reasons are poorly understood, a qualitative design may be better suited to discovering how participants interpret their experiences, what processes led to withdrawal, and which issues the research team did not anticipate.
Conversely, if you already understand the relevant categories and need an estimate of how common each is in a defined population, a survey may be more appropriate.
The choice is not qualitative versus quantitative as competing methodological identities. It is which evidence best answers the question.
Mixed Methods Should Earn Its Additional Complexity
Combining qualitative and quantitative approaches can be valuable when integration answers something that neither component could adequately answer alone.
For example, a quantitative component might estimate whether an intervention changes an outcome, while qualitative evidence helps explain how participants experienced implementation and why effects varied across settings.
But adding a second methodological component means additional sampling, data collection, expertise, analysis, integration, reporting, time, and often participant burden.
If the qualitative and quantitative components merely produce parallel descriptions without serving an integrated research purpose, the extra complexity may add little.
Mixed methods should therefore be selected because the question requires integration, not because two methods appear more rigorous than one.
A Smaller Study May Be Better Than a Larger Study With Unnecessary Objectives
Research projects often expand during planning.
A primary question acquires several secondary questions. Those questions introduce more variables. More variables require additional instruments. Additional instruments increase participant burden. The researcher then adds interviews to explain the survey, subgroup analyses to investigate heterogeneity, and follow-up assessments because longitudinal evidence might be useful someday.
Each addition may sound individually defensible. Collectively, they can produce a study that is expensive, difficult to recruit for, analytically fragmented, and no longer centered on its most important question.
A narrower design that answers one consequential question rigorously may be more useful than an ambitious design that answers six questions weakly.
Sometimes a Pilot or Feasibility Study Should Come Before the Main Study
Perhaps the definitive study is theoretically appropriate, but you do not yet know whether participants can be recruited, the intervention can be delivered, the procedures are acceptable, or the required measurements can be collected reliably.
In that situation, the easier study may not be a simplified version of the main study. It may be a feasibility study addressing the uncertainties that determine whether the main study should proceed.
Feasibility research is particularly useful when important design parameters cannot be established adequately from existing evidence. The purpose is to learn whether and how the future study can be conducted, not to obtain an underpowered preview of its main substantive result.
Sometimes the Best Alternative Is Not Another Primary Study
Before planning additional data collection, consider whether the research question is really asking about the totality of existing evidence.
If numerous studies have already investigated whether an intervention works, another small local comparison may be less informative than a systematic review. If the practical question is which intervention should be implemented, evidence synthesis may need to be combined with information about costs, feasibility, acceptability, implementation, or local context rather than another efficacy study.
Likewise, a methodological question may be answerable through simulation, reanalysis of existing datasets, or validation work rather than recruiting a new sample.
The unit of research progress is not the number of new datasets collected.
The Easiest Study Is Not Always the Most Efficient Study
Efficiency should be evaluated against information gained.
A quick survey that cannot answer the central question may cost less than a longitudinal study but still waste the resources it consumes. Conversely, an elaborate experiment may provide only marginally more information than a carefully designed analysis of existing high-quality data.
A useful comparison therefore considers both burden and evidential value.
| Question to compare |
Why it matters |
| Can the design answer the primary research question? |
A convenient study is not efficient if it answers a different question |
| What inference can the design support? |
Different designs may support descriptive, associational, causal, experiential, diagnostic, prognostic, or other claims differently |
| What new assumptions does it require? |
A simpler design may depend more heavily on measurement, confounding, missing data, or selection assumptions |
| Can existing data answer the question? |
New data collection may be unnecessary when suitable evidence already exists |
| How much participant burden is involved? |
More measurements and follow-up should have a clear scientific purpose |
| What expertise and resources are required? |
A theoretically ideal design that cannot be executed rigorously is not an effective choice |
| How long will the study take? |
Additional time should produce information necessary to the question |
| What important information is lost by simplifying? |
This distinguishes genuine efficiency from methodological compromise |
The goal is not minimum effort. It is a defensible relationship between the evidence you need and the resources required to obtain it.
06 · What This Means for You
Compare at Least One Serious Alternative Before Committing to the Study
Once a study design becomes detailed, changing it becomes psychologically and practically expensive. You have already imagined the sample, written the procedures, selected measures, perhaps even learned the analysis. That is precisely why alternative designs are best considered early.
Take your primary research question and identify at least one genuinely different way it might be answered. Do not construct an obviously inferior alternative merely so the preferred design wins.
Then compare the studies by what they would allow you to know.
A simple decision framework
If a simpler design provides the evidence required for the same primary inference
Prefer the simpler design unless the additional complexity serves another clearly justified purpose.
If a simpler design answers only a weaker or different question
Do not call it equivalent. Decide whether the narrower question is still worth answering or retain the stronger design.
If suitable existing data already contain the required evidence
Consider secondary analysis before imposing the burden and cost of collecting equivalent data again.
If the definitive study depends on major unresolved feasibility assumptions
Consider targeted feasibility work before committing to the full design.
If several methods answer complementary parts of a question and their integration matters
A mixed-methods design may be justified, but specify what integration adds beyond conducting two studies in parallel.
If no feasible design can produce evidence adequate for the intended claim
Revise the research question rather than lowering the evidential standard while keeping the original claim.
Remove Components One at a Time
A useful way to simplify a proposed study is to ask what happens if each component disappears.
Do you need the third measurement occasion? Do you need all six outcome measures? Does the interview component answer a question that matters to the primary objective? Are three comparison groups necessary? Does collecting another demographic variable change any planned analysis or interpretation?
If removing a component changes nothing important about the answer, that component may not be earning its place in the design.
This exercise can reduce participant burden as well as researcher workload. It may also improve data quality because shorter, more focused procedures can be easier to implement consistently.
Ask What the More Complex Study Buys You
Sometimes complexity is justified.
Longitudinal follow-up may establish temporal information unavailable cross-sectionally. Multiple sites may test whether findings generalize beyond one institution. Qualitative evidence may explain an implementation process that outcome estimates alone cannot reveal. Repeated measurements may characterize trajectories rather than simple before-and-after differences.
State the additional information explicitly.
If you cannot explain what a more complex design buys you, it may be buying mostly complexity.
Consider Feasibility as Part of Methodological Quality
A theoretically elegant design that cannot recruit enough participants, obtain the required data, maintain follow-up, or be completed with available expertise is not automatically superior to a less ambitious design that can be executed rigorously.
Research-question frameworks such as FINER explicitly treat feasibility as a characteristic of a good research question. Feasibility includes adequate participants, technical expertise, time, money, equipment, and institutional support.
If the proposed design depends on several fragile conditions, examine what could make the research idea fail once the study encounters real-world constraints.
Do Not Let Prior Investment Decide Which Design Wins
Perhaps you have already created the questionnaire, learned structural equation modeling, negotiated access to a site, or spent weeks writing the original protocol. Those investments make redesign inconvenient. They do not make the original design scientifically preferable.
If another study now appears to answer the question more directly, compare the designs based on future evidential value and costs rather than the effort already spent.
Otherwise, the decision risks becoming a question of whether you are preserving the study because it is good or because you have already invested in it.
Watch Out
Simplification should remove unnecessary research burden, not necessary evidence. If the easier study cannot measure the relevant construct, establish the required timing, represent the intended population, create the necessary comparison, or support the intended inference, it is not an efficient version of the same study. It is a different study.
