What Kind of Evidence Does Your Research Question Actually Require?

What would you actually need to observe or know to answer your question?

Researchers often move quickly from a research question to an instrument: write the question, find a questionnaire, conduct interviews, download a dataset, or start extracting records. A more fundamental step can disappear in between.

What evidence would actually allow you to answer the research question?

This matters because a plausible source of data is not necessarily evidence for the claim you want to make. Student satisfaction is not student learning. Website visits are not necessarily engagement. A person's intention to behave in a particular way is not the behavior itself. A publication count does not directly measure research quality. Before deciding how to collect data, you need to determine what would count as evidence of the phenomenon in the first place.

Translate the question into an explicit evidence requirement

Move from the wording of the question to the evidence needed to answer it

Every research question creates an evidentiary obligation

A research question commits you to finding particular information. If the question asks how common something is, you need evidence that permits an appropriate estimate. If it asks how people experience a phenomenon, you need evidence capable of representing those experiences. If it asks whether two variables are related, both variables must be represented appropriately. If it asks whether an intervention caused an outcome, measuring the outcome alone is not enough; the overall research design must also support the causal inference.

This is why question, design, and data need to align. Authoritative methodological guidance on data-source selection similarly emphasizes that the research question should determine the required data rather than researchers attempting to make an available dataset answer a question for which it lacks essential information.

Thinking in terms of evidence therefore creates a useful bridge between the question and the eventual choice of data collection method.

First identify what the question is asking you to establish

Research questions can require very different forms of answers. Look at the main intellectual task embedded in the question.

The final row is particularly important. “Why” questions can hide very different evidentiary demands. Asking why participants say they left a program is not the same as estimating which factors causally increased the probability of leaving. Participants' explanations may be exactly the evidence required for the first question but insufficient for the second.

Distinguish the construct from its indicator

Researchers frequently study constructs that cannot be observed directly. Motivation, engagement, anxiety, trust, learning, socioeconomic status, organizational climate, and research impact are examples. The researcher therefore needs indicators or measurements that represent the construct.

This is where apparently sensible studies can become misaligned. Suppose the question asks whether a new learning platform increases student engagement. The researcher might use login frequency because it is readily available. Yet logging in is not identical to engagement. A student may log in repeatedly because the platform is confusing, while another may download the material once and study it offline.

An indicator does not have to reproduce the construct perfectly to be useful. Many important constructs require indirect measurement. The important methodological question is whether there is a defensible relationship between the indicator and what you claim it represents.

Ask whether you need perceptions, behavior, performance, or records of events

Researchers sometimes treat different forms of evidence as interchangeable because they concern the same topic. They are not.

If you ask teachers whether they feel confident using artificial intelligence, you have evidence about reported confidence. If you ask them how frequently they use AI, you have reported behavior. If you inspect platform records, you may obtain evidence of recorded use on that platform. If you evaluate their performance on a task, you have evidence about performance under the specified conditions.

Each could be relevant to a study of AI use, but each answers something different. In particular, self-report and measures obtained independently of participants' reports should not be substituted for one another without considering what the research question actually targets.

Consider how direct the evidence is

Evidence can vary in how closely it represents the phenomenon of interest. Suppose a researcher wants to know whether students read assigned articles. Asking students whether they read them provides self-reported evidence. A platform record showing that the file was opened provides behavioral trace evidence. Neither necessarily establishes that the article was carefully read or understood.

If the actual question concerns comprehension, an assessment designed to capture comprehension may provide more relevant evidence. This illustrates why researchers should examine whether they are measuring the phenomenon of interest or relying on an indirect indicator.

Indirect evidence is not automatically weak evidence. Sometimes the target construct is inherently latent or otherwise inaccessible to direct observation. The problem arises when researchers forget the inferential step between indicator and construct and begin writing as though the proxy and phenomenon were identical.

Decide whose or what evidence you need

After specifying the evidence, identify the source capable of providing it. Sources may include participants, observers, documents, institutional records, databases, physical measurements, tests, digital systems, artifacts, or previously collected research data.

Different sources can provide different views of the same phenomenon. A university's policy document can establish what the formal policy says. Faculty interviews can reveal how instructors understand that policy. Classroom observation may show how practices unfold. Administrative records might document implementation-related events. None automatically substitutes for the others.

The choice may also determine whether you need new data or whether suitable evidence already exists. The distinction between primary and secondary data matters here, but existing data should be selected because they contain the necessary information, not merely because obtaining them is convenient.

Evidence must have enough specificity for the question

A dataset can be broadly relevant to a topic yet still lack a critical variable, level of detail, time period, population, or measurement needed to answer the question. Methodological guidance on data-source selection recommends identifying minimum data requirements before committing to an existing source. This reduces the temptation to force available data into a question they cannot adequately address.

Suppose you want to investigate whether feedback timing is associated with subsequent student performance. A dataset containing final grades and whether feedback was provided is not necessarily enough. If the timing of feedback is central to the question, the data must represent timing with sufficient precision. A variable that merely indicates “feedback received” does not satisfy that evidentiary requirement.

The required evidence depends on the claim you want to make

One of the most useful ways to evaluate evidence is to work backward from the sentence you hope to write in the conclusion.

If you want to conclude that “participants perceived the intervention as useful,” participant reports may provide suitable evidence. If you want to conclude that “the intervention improved performance,” perceptions of usefulness are insufficient. You need an appropriate measure of performance and a study design capable of supporting the comparison or inference being made.

This distinction becomes particularly consequential for causal language. Evidence that two variables changed together can establish an association under appropriate conditions, but causal claims require more than simply measuring both variables. The research design must address plausible alternative explanations sufficiently for the intended inference.

More evidence is not necessarily better evidence

Collecting information from several sources or using several methods can be valuable when the sources address meaningful evidentiary needs. It can also create a large dataset containing several weak indicators of the same construct.

Three indirect measures do not automatically become a direct measure simply because they agree. Likewise, agreement across sources can increase confidence in some interpretations while disagreement can reveal meaningful differences in perspective, context, timing, or measurement.

The appropriate question is not “How much evidence can I collect?” but “What evidence is necessary to answer this question well?” The implications of collecting the same or related information from more than one source deserve separate consideration when multiple perspectives or records are relevant.

Changing the question changes the evidence you need

Common mistakes when deciding what counts as evidence

Write an evidence specification before choosing the instrument

Before deciding how to collect data, try completing this sentence: “To answer this question, I need evidence of...” Be specific enough that another researcher could understand what must be observed, measured, documented, or reported.

Then identify the source, representation, and level of directness. This small exercise can reveal surprisingly large problems before they become expensive ones. Methodological problems discovered after data collection have an unfortunate tendency to become limitations sections.

This evidence-first reasoning also helps when practical reality intervenes. Sometimes the strongest evidence for a question cannot realistically be collected. The goal then is not to pretend the compromise disappeared. It is to decide whether the remaining evidence is sufficient for a more carefully bounded question or claim.

Check the evidence before designing data collection

Frequently asked questions about evidence and research questions

Your question determines what must count as evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.