03 · What You Need to Know
A Method Is Appropriate Only in Relation to the Question It Must Answer
Methods are tools for producing and analyzing evidence. Calling a method quantitative, qualitative, experimental, observational, longitudinal, cross-sectional, ethnographic, survey-based, or mixed does not establish that it fits a particular question.
The same method can be appropriate for one question and poorly suited to another.
Begin with the evidentiary requirement, not your preferred technique
Suppose you are comfortable with surveys. That familiarity may make surveys efficient and attractive. But your question may concern how a complex social process unfolds during collaborative work.
A questionnaire might capture participants' retrospective perceptions of that process. Whether those perceptions are sufficient depends on the question. If you need evidence of moment-to-moment interaction, additional or different methods may be necessary.
This is why it helps to establish what evidence the research question actually requires before deciding how to collect it.
Methodological fit concerns capability, not prestige
Methods do not form a simple hierarchy in which more complex techniques are automatically better.
A randomized experiment can address some causal questions exceptionally well but may be unsuitable, infeasible, or unethical for other inquiries. In-depth interviews can provide rich evidence about experiences and meanings but may not support a population prevalence estimate. A cross-sectional survey can characterize patterns at a particular period but may be poorly positioned to observe individual developmental trajectories directly.
The relevant question is: Can this design and method generate evidence capable of supporting the answer I seek?
A descriptive method cannot automatically answer an explanatory question
One common mismatch occurs when the research question demands explanation but the method produces only description.
Imagine asking, “Why do faculty members discontinue using a learning management system after initial adoption?” but collecting only system logs showing login frequencies.
The logs can reveal when use declined, how frequently users logged in, and perhaps which features they accessed. They do not necessarily reveal why faculty discontinued use.
The data might contribute to an explanatory study, but additional evidence would likely be needed to investigate motivations, constraints, organizational circumstances, or decision processes.
Cross-sectional evidence does not automatically demonstrate change
Questions containing words such as develop, change, increase, decline, or evolve deserve particular scrutiny.
A cross-sectional study observes different units at a particular time or over a limited observation window. Differences among groups can sometimes inform hypotheses about developmental or temporal processes, but cross-sectional differences are not automatically equivalent to within-person change over time.
If the question specifically concerns trajectories or development, a longitudinal design may provide a more direct evidentiary structure. Alternatively, a cross-sectional study can ask a question that accurately reflects what its evidence can establish.
Association is not automatically evidence of causation
Perhaps the most consequential mismatch occurs when a question asks whether X causes, improves, reduces, leads to, or results in Y, but the design establishes only that X and Y are associated.
Causal inference requires more than observing that two variables move together. Alternative explanations, confounding, selection, temporal ordering, measurement, and other design considerations affect whether a causal interpretation is warranted.
This does not mean observational research can never contribute to causal inference, nor that every causal question requires a randomized trial. It means that causal language creates evidentiary obligations that must be addressed explicitly by the design and its assumptions.
Watch Out
Changing “is associated with” to “affects” in the title or discussion does not strengthen the evidence. It strengthens the claim. If the design has not changed, the evidentiary basis has not changed either.
Self-report methods cannot silently become behavioral measures
Suppose teachers complete a questionnaire asking how frequently they use learner-centered strategies. The resulting data provide evidence about teachers' self-reported practices.
If the research question explicitly concerns self-reported practice, the method may fit well.
If the study claims to determine what teachers actually do in classrooms, the alignment is less straightforward. Classroom observation, artifacts, student reports, or other evidence might be relevant depending on the methodological purpose.
Again, self-report is not inherently weak. The mismatch occurs when the construct claimed in the question differs from the phenomenon actually represented by the evidence.
A method can produce part of the answer without producing all of it
Some research questions contain several evidentiary demands.
Consider: “How and why does participation in an interdisciplinary research program influence early-career researchers' collaborative practices over time?”
Interviews may provide evidence about participants' interpretations of how and why their practices changed. Network data may document changing patterns of collaboration. Repeated observations or records may reveal additional aspects of behavior over time.
No single source necessarily has to do everything. A multimethod or mixed-methods design may be justified when different forms of evidence address genuinely different parts of the question.
The important point is not to add methods for sophistication. Add them when the question requires evidence that one method cannot adequately provide.
Methodological congruence matters particularly in qualitative inquiry
Qualitative research involves more than selecting interviews as a data-collection technique. The research question, philosophical assumptions, methodology, sampling, data collection, analysis, and interpretation should work coherently together.
Methodological congruence is therefore broader than matching a question with an instrument. A phenomenological question, ethnographic question, grounded-theory inquiry, case study, narrative inquiry, or other qualitative approach carries different assumptions about what is being investigated and how knowledge about it can be developed.
A mismatch can occur even when the researcher conducts technically competent interviews if the way participants are sampled, questions are asked, data are analyzed, or claims are made does not fit the methodology claimed by the study.
Sometimes the method is fixed and the question should change
Researchers do not always design studies from scratch.
You may be working with an existing dataset, archival collection, institutional database, previously collected survey, natural experiment, or restricted source of evidence. In these situations, the evidence may be relatively fixed.
It can be entirely legitimate to formulate a research question around what those data can defensibly answer.
The danger lies in retaining a question that requires unavailable evidence because it sounds more important. A modest question that your evidence can answer is methodologically stronger than an ambitious question that your design cannot address.
Sometimes the question is non-negotiable and the method should change
The reverse also occurs.
If the central purpose of the project is to understand how students make decisions while solving complex problems, and a multiple-choice achievement test cannot reveal those decision processes, narrowing the question to test scores may destroy the reason for conducting the study.
In that situation, changing or supplementing the method may be the better solution.
The decision depends on the intellectual purpose of the study, feasibility, ethics, available expertise, resources, access, and the strength of the claims you need to make.
Do not wait until analysis to discover the mismatch
A mismatch identified before data collection is a design problem. A mismatch identified after data collection can become a much more expensive problem.
This is one reason an alignment matrix can be useful. Mapping each research question to the evidence required, its source, collection method, and planned analysis can expose missing connections before fieldwork begins.
It is also possible for a study to be technically well executed yet conceptually misaligned. Good execution cannot compensate for answering a different question from the one the study claims to investigate.
