Would You Know What Evidence Counts as an Answer to the Question?

If the Study Were Finished Tomorrow, What Would Convince You That the Question Had Been Answered?

Researchers often move quickly from a research question to a method. A question about students leads to a survey. A question about experiences leads to interviews. A question involving two variables leads to correlation or regression. Once an instrument or analytical technique has been identified, the study can feel methodologically concrete.

One step may still be missing.

What would the evidence actually have to show for you to answer the research question?

That question is more fundamental than choosing a method. A questionnaire can generate responses, an interview can generate transcripts, and a database can provide thousands of observations. None of those facts establishes that the resulting information will constitute evidence for the particular answer your research question requires.

Specify the Evidence Before You Decide How to Collect It

Translate the Research Question Into an Evidence Requirement

A research question is not simply a sentence describing an interesting topic. It creates an evidentiary obligation. If the question asks how common something is, the study needs evidence capable of supporting an appropriate estimate. If it asks about experience, it needs evidence that represents participants' experiences. If it asks whether variables are related, the relevant variables need to be represented within a design capable of examining that relationship. If it asks about causation, the overall design must support a causal inference rather than merely document covariation.

Research-question guidance emphasizes that the question should shape the study design and identify the information needed for an answer. FINER similarly treats feasibility as requiring an ability to investigate the question with adequate participants, expertise, time, resources, and manageable scope.

Do Not Answer “What Evidence?” With the Name of a Method

Suppose the question is: “How do doctoral students experience the use of generative AI during dissertation writing?”

“I will conduct interviews” does not yet specify the evidence. Interviews are a means of generating evidence. The evidentiary requirement might instead be articulated as: accounts from doctoral students describing how they use generative AI during dissertation work, how they interpret that use, and the meanings, benefits, difficulties, or tensions they associate with the experience.

Once the requirement is clear, interviews may indeed be a defensible method. But now the method has been selected because it can generate the evidence rather than because interviews seemed customary for the topic.

Start With the Main Intellectual Task in the Question

Different question forms imply different evidentiary tasks.

These are broad categories rather than prescriptions. The appropriate evidence depends on the precise construct, population, context, design, and intended inference.

Separate the Phenomenon From Its Indicator

Suppose the research question asks about student engagement. A learning-management-system login count may be available. Is that the evidence the question needs?

Perhaps, if the question specifically concerns platform access behavior. But if “engagement” refers to cognitive investment, emotional involvement, active participation, or a multidimensional construct, login frequency may represent only a limited behavioral trace.

This is why evidence specification comes before instrument selection. You first identify the phenomenon the question requires you to represent, then evaluate whether the proposed indicator actually represents it.

If that connection is uncertain, examine whether the variable can actually be measured in a defensible way.

Distinguish Evidence of Perception From Evidence of the Phenomenon Being Perceived

Suppose students strongly agree with the statement, “Using generative AI improves my learning.” That is evidence about students' perceptions of AI and learning.

It is not, by itself, evidence that learning actually improved.

This distinction recurs across research. Employees' perceptions that remote work increases productivity are evidence about perceived productivity effects. Researchers' beliefs that a policy improves research quality are evidence about their beliefs. Participants' confidence in their own competence is evidence about self-assessed competence.

Perceptions may be exactly what the question seeks. Problems arise when evidence about perceptions is silently used to answer a question about performance, behavior, effectiveness, or causation.

Evidence of Association Is Not Automatically Evidence of Causation

If students who use AI more frequently have higher grades, the observed relationship is evidence of an association under the study's conditions. Whether it supports a causal conclusion depends on substantially more.

A causal question asks about what would happen under alternative exposure or intervention conditions. The design must therefore address the relevant causal contrast and assumptions rather than simply demonstrate that X and Y covary.

If the evidence specification requires an effect, check whether the research question contains a causal commitment that the design must be capable of supporting.

Questions About Change Need Evidence of Change

If the question asks how students' critical thinking develops over a degree program, measuring different first-year and fourth-year students at one moment is not the same as observing how the same individuals develop over time.

A cross-sectional comparison may still answer an important question about differences between cohorts. It should not quietly be treated as direct evidence of individual developmental change.

Evidence should correspond to the temporal claim being made.

Mechanism Questions Need Evidence About the Proposed Process

Suppose an experiment establishes that an instructional strategy improves performance. The next question asks, “How does the strategy produce that improvement?”

Knowing that an effect exists does not by itself reveal the mechanism. The study needs evidence about the intermediate process or processes proposed to connect the intervention to the outcome.

Mechanisms may be investigated through mediating variables, process observations, qualitative evidence, experimental manipulation, temporal data, or other approaches depending on the theoretical problem. The essential point is that a mechanism cannot be established merely by redescribing the outcome.

Comparative Questions Need Evidence That Is Actually Comparable

Suppose the question asks whether students at two universities differ in academic performance. Both institutions provide GPA data. At first glance, the evidence requirement appears satisfied.

If the institutions use substantially different grading systems, assessment structures, or grade distributions, the numerical values may not have sufficiently comparable meanings. The problem is not the absence of data but whether the evidence supports the intended comparison.

That is why comparative questions should also be checked for whether they rely on a comparison whose resulting difference can actually be interpreted.

Evidence Must Come From the Right Source

Ask who or what is in a position to provide the evidence.

If the question concerns students' experiences, students are an obvious source. If it concerns institutional policy, official documents and decision-makers may be more relevant. If it concerns actual platform behavior, system logs may provide evidence unavailable from memory alone.

No source is inherently superior. The source should be appropriate to the claim.

Sometimes Several Sources Are Needed

Some questions cannot be represented adequately by one source. A study of how a new learning technology is implemented might need policy documents, teacher accounts, student experiences, and classroom observations if each addresses a distinct part of the implementation process.

Using several sources should follow an evidentiary rationale rather than a desire to make the study appear more sophisticated. Three weak proxies do not automatically become one strong measure simply because they are triangulated.

The Evidence Requirement Also Sets the Limits of the Answer

Suppose the best available evidence allows you to estimate an association but not a causal effect. That does not make the evidence worthless. It tells you what answer is defensible.

Research questions and claims should remain proportionate to the evidence. When the question demands substantially more than the study can produce, you may need to strengthen the design or revise the question rather than hoping that analysis will close the gap.

This becomes especially important when the question requires evidence the proposed study could never produce.

Write an Evidence Specification Before Choosing the Instrument

The questionnaire was not inherently a poor method. It was simply evidence for a different question.

Common Mistakes When Deciding What Counts as Evidence

Write One Sentence Beginning With “To Answer This Question, I Need Evidence Of...”

Before selecting an instrument, dataset, sample, or analytical procedure, complete that sentence. Avoid methodological vocabulary initially. Describe the phenomenon, relationship, difference, experience, process, change, or effect that would have to be evidenced.

Then identify where that evidence could come from and what observations could represent it. Only after those decisions should you select the method capable of generating the required information.

Specify What Would Count as an Answer Before Collecting Data

Questions About Evidence and Research Questions

You Should Know What Would Count as an Answer Before You Decide How to Find It

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.