Should You Choose the Design Before Deciding Exactly What Evidence You Need?

Should You Select a Research Design Before Knowing What Evidence You Need?

You have a research question and perhaps already have a design in mind. A cross-sectional survey seems manageable. An experiment sounds rigorous. A qualitative case study might provide the depth you want. But have you established what evidence would actually allow you to answer your question?

This distinction matters because selecting a familiar design can quietly determine what evidence you collect, even when that evidence cannot support the conclusion you originally intended. The resulting study may be competently executed yet unable to answer its central question.

The issue is not whether research questions or designs always come first in a rigid sequence. It is whether the evidentiary demands of the question have been considered before a methodological commitment becomes difficult to reverse.

Identify Your Evidence Requirements Before Committing to a Design

How Evidence Requirements Should Shape Research Design Decisions

A Research Question Implies an Evidentiary Obligation

Every research question creates an obligation to produce evidence appropriate to the answer being sought. Asking how frequently something occurs creates different requirements from asking why it occurs, how participants experience it, or whether an intervention causes an outcome.

Consider four questions about artificial intelligence in university teaching:

• How frequently do instructors use generative AI for assessment preparation?
• How do instructors experience the introduction of generative AI into their assessment practices?
• What factors are associated with instructors' adoption of generative AI?
• Does access to a particular AI-assisted assessment tool improve the quality of instructors' feedback?

All four concern the same broad phenomenon. They do not, however, require interchangeable evidence.

The first calls for appropriately defined and measured frequency data. The second requires accounts or observations capable of illuminating experience. The third requires evidence about variation and relationships between specified factors. The fourth raises a causal question, requiring a credible basis for distinguishing the tool's effect from alternative explanations.

These differences should inform the choice of research design before convenience or disciplinary convention becomes the dominant consideration.

Start With the Claim You Want to Make

A productive way to identify evidence requirements is to imagine the conclusion your study would need to defend. This is not an invitation to decide the result beforehand. Instead, it means specifying the type of claim that would constitute an answer, regardless of whether the findings are positive, negative, mixed, or inconclusive.

Suppose your question is whether an AI-supported feedback system improves students' writing performance.

A conclusion that students generally liked the system would not answer that question. Neither would a finding that students using the system achieved high scores. The intended claim concerns improvement attributable to the system, so the evidence must address performance and provide a credible basis for causal interpretation.

Working backward from the claim your evidence must support helps expose gaps that may otherwise remain hidden until data analysis or peer review.

Distinguish the Evidence Needed From the Data You Can Collect

Researchers sometimes treat available data as though their existence establishes their relevance. Institutional records, survey responses, assessment scores, and interview transcripts may be accessible, but accessibility does not establish evidentiary adequacy.

For example, an institution might possess student grades before and after introducing an educational platform. Those records could describe performance trends. They may not establish whether the platform caused the changes, especially if assessment difficulty, student composition, teaching practices, or other conditions also changed.

The distinction is particularly important when working with secondary data. A dataset may be extensive and reliable for its original purpose while lacking the variables, timing, or contextual information needed for a new research question.

Specify What the Evidence Must Establish

Before selecting a design, identify the essential properties of the evidence you need. These requirements can be expressed without prematurely deciding on a specific method.

These are illustrative relationships, not one-to-one rules. The same design family may serve several purposes, and different designs may address similar questions with different limitations.

The more useful distinction is between what a method can produce and what an inference requires. A statistical model, for example, can estimate an association from cross-sectional data, but the sophistication of that model does not by itself establish temporal ordering or eliminate unmeasured confounding.

Evidence Requirements Include Quality, Timing, and Coverage

Identifying the type of data is only part of the task. You also need to consider whether the evidence will be sufficiently informative and credible.

Measurement quality: Do the instruments, indicators, or observations adequately represent the concepts being studied? A measure of students' satisfaction is not automatically a valid measure of learning.

Temporal relevance: Must the evidence establish what occurred before, during, or after an event? A single measurement may be inadequate when the question concerns individual change or temporal sequence.

Population and setting: Whose experiences, outcomes, or characteristics matter? Evidence from a convenient group may not support claims about a broader population.

Comparison: Does the question require a benchmark, contrast, or counterfactual? If so, how will that comparison be justified?

Context: Are institutional arrangements, cultural practices, implementation conditions, or participant circumstances necessary to interpret the findings?

Uncertainty: What sources of sampling error, measurement error, bias, or interpretive uncertainty could affect the conclusion?

These considerations help distinguish merely collecting relevant-looking data from assembling evidence that can withstand methodological scrutiny.

Why Choosing a Design Too Early Can Distort the Question

Premature design selection can create a subtle form of methodological lock-in. Once a researcher commits to a survey, experiment, interview study, or particular dataset, subsequent decisions may be made to accommodate that commitment.

The research question can then be reinterpreted to fit the available method without explicitly acknowledging the change.

For instance, a researcher may begin by asking whether an intervention improves learning. After deciding that a questionnaire is easiest to administer, the study might measure perceived usefulness and satisfaction instead. Those are legitimate constructs, but they are not substitutes for learning outcomes.

The problem is not using a questionnaire. A questionnaire may be entirely appropriate for measuring certain learning-related constructs. The problem is presenting evidence about one construct as though it answered a question about another.

This is why alignment between the research question and design must be assessed at the level of the intended inference, not simply through similarities in terminology.

Evidence Requirements Do Not Have to Be Completely Fixed

The recommendation to consider evidence first should not be confused with a requirement that every study follow a strictly linear planning process.

In some qualitative traditions, research questions and data collection strategies are refined as understanding develops. Grounded theory studies, for example, may use theoretical sampling to pursue emerging conceptual categories. Ethnographic inquiry may also change emphasis as fieldwork reveals previously unrecognized practices or relationships.

Such flexibility does not eliminate evidentiary responsibility. It changes how evidence requirements are articulated and revised. Researchers still need to justify why particular observations, participants, documents, or analytical decisions are appropriate for the developing inquiry.

Even in more structured quantitative research, feasibility assessments, pilot findings, and measurement limitations can justify revisiting the original question or design. Where confirmatory analyses are involved, changes should be documented transparently, and exploratory departures should not be presented as though they were planned in advance.

The aim is disciplined iteration rather than either rigid sequencing or unrestricted improvisation.

When Should You Finally Commit to a Design?

A design becomes a defensible choice when you can explain how its central features will produce evidence adequate for the intended answer.

At minimum, you should be able to identify the relevant population or cases, the principal phenomena or constructs, the necessary observations or comparisons, the role of timing, and the main threats to the inference.

You should also consider whether the required evidence can be obtained ethically and feasibly. An idealized design that cannot recruit participants, access necessary records, or meet ethical obligations is not an executable research plan.

Where the strongest available design remains insufficient for the original claim, the appropriate response may be to narrow the claim, revise the question, or acknowledge that the proposed study can provide only partial evidence. The distinction between an ideal and feasible research design matters precisely because methodological ambition must remain connected to what can actually be established.

From a Question About AI Feedback to a Defensible Design

The critical decision occurs before data collection: the researcher distinguishes a question about causal effectiveness from one about user perceptions.

Neither question is inherently more worthwhile. They simply require different evidence, and each deserves a design capable of answering it.

Common Mistakes When Selecting Designs Before Defining Evidence

How to Move From Evidence Requirements to a Research Design

Before committing to a design, write a provisional answer to your research question using neutral language that does not assume the result. Identify what you would need to observe, compare, interpret, or estimate to justify such an answer.

Then evaluate candidate designs against those requirements.

Keep a short written justification connecting the question, intended claim, evidence requirements, and selected design. This record can help when preparing a proposal, explaining methodological choices to supervisors, or responding to reviewers.

There may be more than one defensible design for the same question. The aim is not to discover a uniquely correct methodological label but to choose an approach whose evidence and limitations you can justify.

What to Check Before Finalizing Your Research Design

Frequently Asked Questions About Evidence Requirements and Design Selection

Let the Evidence Your Question Requires Guide the Design

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.