Does the Research Question Require Evidence Your Study Could Never Produce?

Could You Complete the Entire Study and Still Not Have the Evidence You Need?

Imagine asking, “Why do university students drop out?” and then discovering that your available dataset contains only age, sex, degree program, and enrollment status. You have data about students. You can identify who dropped out. You may even find statistical patterns associated with dropout.

But do those data contain evidence about why students left?

This is one of the most consequential mismatches in research design. Researchers sometimes begin with an important question, obtain data related to its topic, and assume that analysis will somehow bridge the remaining gap. Yet a dataset can be relevant without containing the evidence needed for the particular answer the question demands.

Work Backward From the Answer to the Evidence

The Evidence Must Match the Question, Not Merely the Topic

Answerability and feasibility are established criteria for evaluating research questions. The FINER framework asks whether a question is feasible given matters such as participant availability, expertise, time, funding, and manageable scope. Other research-question guidance similarly emphasizes that a study must have access to the people, documents, observations, variables, and other evidence needed to reach its conclusions.

The deeper issue, however, is alignment. Even a perfectly feasible data-collection plan may be inadequate if it generates the wrong kind of evidence for the question.

Start With the Claim the Question Would Require You to Make

Suppose your question is: “How does generative AI use improve students' critical thinking?”

A convincing answer would need to establish more than the coexistence of AI use and critical-thinking scores. The wording presupposes improvement and asks about a causal process. Depending on how “improve” and “how” are interpreted, you may need evidence of change over time, a relevant comparison condition, credible measurement of critical thinking, and evidence bearing on the mechanism through which AI use produces that change.

If your study consists only of a one-time self-report survey asking students whether they use AI and whether they believe it improves their thinking, the evidence may answer a different question: what students report about their use and perceptions.

The first discipline, then, is to state the claim before choosing the data.

Distinguish Data About a Topic From Evidence for a Claim

A database may contain thousands of records related to your topic. That does not make every question about that topic answerable.

Administrative enrollment records can describe retention patterns. They may identify characteristics associated with withdrawal. Unless they include appropriate information about students' reasons, circumstances, experiences, or relevant explanatory variables, they may not support a question asking why individual students leave.

This distinction is particularly important in secondary-data research, where the variables already available can tempt researchers to formulate claims broader than the dataset was designed to support.

Ask What Evidence Would Count as an Answer Before Naming the Method

When asked what evidence would answer their question, researchers sometimes reply, “I will use a survey,” “I will conduct interviews,” or “I will run regression.” Those are methods, not answers to the evidentiary question.

Instead, describe what you would need to observe or establish. Would you need evidence that two groups differ? Evidence that a phenomenon changed after an intervention? Participants' accounts of how they experienced a process? Repeated observations showing temporal development? Documentary evidence of institutional decisions?

If you cannot specify what evidence would count as an answer to the question, it is too early to decide which method should generate it.

Some Questions Require Temporal Evidence

Questions about change, development, trajectories, persistence, long-term outcomes, or temporal ordering require evidence with an appropriate time structure.

A one-time cross-sectional measurement can provide a snapshot. It cannot directly observe within-person change over several years. Asking participants at one moment to remember how they changed may produce retrospective evidence, but that is not equivalent to prospectively observing the change.

If your question asks how something develops over time, ask whether your study actually observes the relevant time process or merely asks participants to reconstruct it.

Causal Questions Require Evidence Beyond Covariation

If the question asks about an effect, impact, influence, or consequence, you may be asking a causal question. Observing that X and Y are associated does not by itself establish what would happen to Y if X were changed.

A causal question therefore requires a design and assumptions capable of addressing the relevant causal contrast. Depending on the problem, this may involve randomization or carefully designed observational approaches, temporal information, appropriate comparators, measurement of relevant confounders, and explicit causal assumptions.

Before proceeding, determine whether the wording contains a causal commitment that your evidence must support.

Mechanism Questions Require Evidence About the Mechanism

Finding that an intervention and outcome are related does not automatically explain how the effect occurs.

Suppose an instructional intervention improves examination performance. A researcher asks whether the improvement occurs because students receive more immediate feedback. Evidence showing a performance difference between intervention and comparison groups establishes neither that feedback changed nor that this change explains the outcome.

Mechanism questions require evidence about the proposed process itself. Exactly what that entails depends on the theoretical model and design, but the mechanism cannot simply be inferred from the existence of an outcome difference.

Experience Questions Require Access to Experience

If the question asks how participants perceive, interpret, understand, or experience a phenomenon, behavioral records alone may not be sufficient.

Learning-management-system logs can show recorded interactions with a platform. They do not directly reveal why a student acted, how the student interpreted the experience, whether the interaction was frustrating, or what meaning the student attached to it.

Conversely, interviews about experience do not automatically establish actual behavioral frequency or performance. Evidence should match the phenomenon being claimed.

Historical Questions Depend on Evidence That Survived

Some questions concern events, decisions, practices, or experiences that occurred before the study was conceived. The necessary records may never have been created, may have been destroyed, or may omit the information required.

Researchers can sometimes triangulate archival records, documents, retrospective accounts, and other sources. Yet some historical evidence is genuinely unrecoverable. A question should not promise a level of reconstruction that the surviving record cannot support.

Ethics Can Make Obtainable Evidence Unobtainable for Your Study

A form of evidence may be technically possible to generate while being ethically unacceptable.

Researchers cannot deliberately expose participants to serious harm merely to create a clean comparison. Sensitive personal data may be legally or institutionally restricted. Deception, privacy intrusion, withholding beneficial treatment, or accessing confidential records may be impermissible depending on the context.

Ethical feasibility is therefore part of answerability. FINER explicitly includes ethics alongside feasibility when evaluating research questions.

Access Restrictions Can Be Methodologically Decisive

You may know exactly what evidence is needed and know that it exists, yet lack permission to obtain it. Institutional records, proprietary platform data, confidential peer-review files, corporate analytics, protected health information, or private communications may be inaccessible.

If access is uncertain, verify it before finalizing a question whose answer depends on those data. “I think the university probably has that information” is not yet a data-access strategy.

Measurement Can Be the Missing Evidence

A study may have participants and data-collection opportunities but no defensible way to represent the central construct.

If your question concerns “deep learning” but the only available outcome is student satisfaction, the problem is not simply that the measure is imperfect. Satisfaction may represent a different construct altogether.

Ask whether the variables can actually be represented by defensible measurements before assuming that any available indicator will suffice.

The Evidence May Exist but Be Too Weak for the Strength of the Claim

Answerability is not binary. Sometimes a study can produce evidence relevant to the question but not strong enough for the wording of the conclusion.

A small exploratory qualitative study may provide rich insight into how particular participants understand a phenomenon while offering limited grounds for estimating population prevalence. A cross-sectional observational study may estimate associations while offering weaker grounds for some causal claims. A convenience sample may reveal patterns in the sampled group without justifying broad population estimates.

The appropriate response may be to narrow the inference rather than discard the study.

More Data Do Not Repair the Wrong Evidence

If the study lacks the information needed for the claim, increasing sample size does not necessarily solve the problem.

Ten thousand observations of the wrong variable remain observations of the wrong variable. A million cross-sectional records do not directly become longitudinal observations. More respondents reporting perceptions do not automatically transform perceptions into objective performance measures.

Sample size can improve precision. It cannot substitute for evidentiary relevance.

When a Convenient Dataset Cannot Answer the Question

The revised question is narrower, but it accurately describes the evidence the study can produce. It avoids turning statistical predictors into a complete explanation of why students leave.

Common Mistakes When Matching Evidence to a Research Question

Build an Evidence Map Before You Build the Study

Write your research question, then write one or two sentences describing what a convincing answer would need to establish. Under that, list the observations, measurements, comparisons, records, participant accounts, time points, or other evidence necessary to justify that answer.

Only then compare the list with what your proposed study can actually produce.

Check Whether Your Study Can Produce the Evidence You Need

Questions About Whether a Study Can Produce the Required Evidence

Your Study Needs the Right Evidence, Not Merely More Evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.