How Do You Know Whether Your Research Question Can Actually Be Answered With Data?

Will the Data You Can Collect Actually Answer the Question You Are Asking?

You have a research question. You have access to participants. Perhaps you already have a questionnaire, dataset, interview guide, or institutional records. It may therefore seem obvious that the question can be answered with data.

But consider a study asking, “Why do students drop out of university?” If your only available dataset contains age, sex, degree program, and enrollment status, you certainly have data about students and dropout. You do not necessarily have evidence capable of answering why dropout occurs.

This distinction is fundamental. A dataset can be relevant to a topic without being capable of answering the research question. Before collecting or analyzing anything, you should be able to explain what evidence would constitute an answer and whether your study can actually produce that evidence.

Work Backward From the Answer to the Evidence

Data Answerability Is More Than Data Availability

Researchers sometimes begin with whatever data are easiest to obtain and then ask what research question can be attached to them. Secondary-data research can certainly begin with an existing dataset, and methodological opportunities often influence question development. The danger is assuming that the presence of relevant-looking variables means the intended question has become answerable.

Research-question guidance emphasizes that the question should inform the study design, population, variables or phenomena, and methods needed to answer it. The FINER framework likewise places feasibility at the center of question development, including whether sufficient participants, technical expertise, time, money, and other resources are available.

Data answerability goes one step further: even if data can be obtained, are they the right data for the question?

Start by imagining what an answer would look like

Take the question:

“How frequently do undergraduate students use generative AI for academic writing?”

A satisfactory answer might be a defensible estimate of the frequency or distribution of reported AI use in a defined student population.

Now consider:

“How do undergraduate students decide whether generative AI use is acceptable in academic writing?”

A satisfactory answer would look very different. You might need detailed evidence about students' reasoning, interpretations of rules, experiences, perceived boundaries, and decision-making.

Finally:

“Does access to generative AI cause changes in students' independent writing performance?”

Now the answer requires evidence supporting a causal contrast between alternative conditions of AI access.

The topic is similar. The evidentiary demands are not.

Ask what must be observed, measured, documented, or elicited

Every empirical research question implies some form of evidence.

A prevalence question requires evidence capable of establishing how common something is in the population of interest. A relational question requires measurements of the relevant characteristics. A qualitative question about experience requires evidence through which those experiences can be investigated. A historical question may require documentary or archival evidence. A causal question requires evidence and a design capable of supporting the relevant counterfactual comparison.

This is why researchability depends on having a credible path from question to evidence.

If you cannot describe what evidence would count as an answer, the question may still be conceptually underdeveloped.

The relevant concepts must be represented in the data

Suppose your question asks whether “academic engagement” is associated with achievement. Your dataset contains class attendance.

Is attendance academic engagement?

Perhaps attendance is one indicator of behavioral engagement. Perhaps it is an inadequate representation of the broader construct you intend. The answer depends on your conceptual definition, theoretical framework, measurement strategy, and available evidence.

The methodological problem is construct validity: does the operational measure adequately represent the concept about which you intend to make a claim?

Measurement literature treats validity not as an intrinsic property of a test in every circumstance but as evidence supporting interpretations and uses of scores. The Standards for Educational and Psychological Testing, for example, frames validity in terms of evidence and theory supporting interpretations of test scores for proposed uses.

A convenient variable with a familiar label is therefore not automatically a valid measure of the construct in your research question.

Operational definitions can quietly change the question

Imagine asking:

“Is social media use associated with academic performance?”

You operationalize social media use as the number of minutes students report spending on one platform yesterday. Academic performance is operationalized as one quiz score.

Your empirical study has become much more specific than the wording suggests.

The data may answer a question about yesterday's reported use of one platform and performance on one assessment. Whether that evidence adequately represents the broader constructs “social media use” and “academic performance” requires justification.

This is one reason the research question and operational definitions should be developed together. The question does not necessarily need to name every instrument, but the measures must still correspond to what the question claims to investigate.

The population represented by the data must match the population in the question

Suppose the research question asks:

“What proportion of university students in the Philippines use generative AI for academic writing?”

You survey 180 students from one program at one university using convenience sampling.

The responses are data. The difficulty is whether they can support the population-level estimate requested by the question.

The issue is not simply sample size. Sampling design, coverage, selection, nonresponse, and the relationship between the observed sample and target population all affect what can reasonably be inferred.

A question about “university students in the Philippines” creates an evidentiary obligation quite different from a question about “students enrolled in the participating program.”

More data do not solve a mismatch in the target population

A very large convenience sample can provide precise estimates about the observed sample while still failing to represent the target population appropriately.

Precision and representativeness are different issues.

Ten thousand responses collected through a channel that systematically excludes important groups do not automatically provide a better population estimate than a smaller probability-based sample designed for the intended population.

This matters whenever researchers equate “big data” with “good data.” Volume cannot compensate automatically for selection mechanisms that make the observed data systematically unrepresentative of the question's target.

Ask whether the data capture the relevant time structure

Some questions cannot be answered without information about time.

Suppose you ask:

“Does academic burnout predict subsequent university withdrawal?”

If burnout and withdrawal status are measured simultaneously, the data may not establish the temporal sequence implied by “predict subsequent.”

A question about change requires observations capable of representing change. A question about development may require longitudinal evidence. A question about long-term outcomes requires an appropriate follow-up period.

Timing is therefore not merely a logistical feature of data collection. It can determine whether the data correspond to the question at all.

Cross-sectional data can answer many questions, just not every question

Cross-sectional studies are extremely useful for questions about prevalence, distributions, characteristics, and many observed associations.

Problems arise when researchers ask those data to establish temporal or causal relationships that the design cannot support.

For example:

“What proportion of students report academic burnout?”

can be addressed cross-sectionally.

“Is burnout associated with reported generative AI use?”

may also be addressed as an association.

“Does generative AI use cause burnout?”

requires a different inferential framework.

The issue is not that cross-sectional data are weak in some universal sense. They are appropriate or inappropriate relative to a particular question.

Qualitative data need to provide access to the phenomenon you want to understand

Data answerability is not only a quantitative measurement problem.

Suppose your qualitative question asks:

“How do students negotiate disagreements with instructors about acceptable generative AI use?”

If your interview guide asks only whether students like AI, how frequently they use it, and which tools they prefer, the resulting transcripts may contain little evidence about negotiation, disagreement, or instructor expectations.

You have qualitative data about the topic without qualitative evidence adequate for the question.

Qualitative methodological guidance emphasizes alignment among the research question, design, sampling, data collection, and analysis. The question should guide what kind of data are needed and from whom those data should be generated.

The participants must be able to provide the evidence you need

Suppose you want to know why administrators adopted a particular university policy, but you interview only students.

Students may provide valuable evidence about how they experienced the policy. They may speculate about why administrators adopted it. They are not necessarily appropriate sources for reconstructing the administrators' actual decision-making process.

Source appropriateness matters.

The people easiest to recruit are not always the people capable of answering the question. Likewise, the most accessible documents may not contain the information needed to reconstruct a historical or organizational process.

Self-report data can answer some questions better than others

If your question concerns beliefs, perceptions, attitudes, intentions, or participants' reported experiences, self-report may be entirely appropriate.

If your question concerns actual behavior, however, self-report may provide only one imperfect representation of that behavior.

For example:

“How frequently do students report using generative AI?”

is explicitly a self-report question.

“How frequently do students actually use generative AI?”

makes a broader behavioral claim. Depending on the context, platform logs, observations, digital traces, or other evidence might provide a different basis for answering it.

Neither data source is automatically superior. The question should make clear which phenomenon the evidence is intended to represent.

Existing datasets impose boundaries on the questions you can answer

Secondary-data research often reverses the usual sequence because the data already exist. You may begin with a dataset and ask which questions it can credibly address.

This is entirely legitimate.

But the available variables, population, sampling process, measurement timing, missingness, data provenance, and original purposes of collection constrain what questions can be answered.

If an existing dataset contains “screen time” but does not distinguish social media from educational activity, you cannot simply relabel screen time as social media use because that is the construct your preferred question requires.

A useful secondary-data question therefore emerges from the intersection of substantive importance and what the dataset can actually represent.

Administrative data can be powerful but were often collected for another purpose

Institutional records may provide large samples, longitudinal histories, and outcomes that are difficult to obtain through surveys.

Yet administrative variables are often defined for operational rather than research purposes. Missing data may reflect institutional processes. Coding practices can change over time. Important theoretical constructs may not have been measured at all.

The existence of a column called “withdrawal reason” does not guarantee that it captures the complex reasons students leave university.

Researchers should therefore investigate how the data were generated, what each variable means operationally, and what processes determine whether information is recorded.

Missing data can make an otherwise answerable question difficult

A dataset may nominally contain every variable needed for the question, but extensive or systematically patterned missingness can undermine the analysis.

Suppose socioeconomic information is missing disproportionately among particular groups. A causal or associational analysis requiring that variable may then depend on assumptions about why the information is missing and how the missingness is handled.

Missing-data methods can sometimes address the problem under explicit assumptions. They cannot recreate information without assumptions simply because the research question needs it.

Answerability therefore depends on data quality as well as variable names.

Measurement error can change what the data appear to say

Suppose students substantially underreport prohibited AI use because they fear disciplinary consequences.

The resulting data may underestimate prevalence. If underreporting differs according to another characteristic, associations can also be distorted.

Measurement error is not merely an inconvenience to mention in the limitations section. Severe measurement problems can determine whether the study provides useful evidence about the intended construct.

This is particularly important when researching sensitive, stigmatized, illegal, or institutionally sanctioned behaviors.

A causal question needs more than measurements of X and Y

Suppose your dataset contains AI-use frequency and writing performance. That is enough to calculate an association.

It is not automatically enough to estimate the causal effect of AI use on writing performance.

As discussed in the previous guides, observational causal inference requires an explicitly causal question, a defined causal contrast or estimand, appropriate temporal structure, and assumptions about confounding, selection, measurement, and other aspects of the data-generating process.

The fact that both variables appear in the dataset does not solve the causal identification problem.

This is why a question asking about an “effect” without an experimental design needs more methodological work than an associational question using the same variables.

Prediction requires data capable of evaluating prediction

Prediction also has its own evidentiary requirements.

Suppose you ask:

“Can student engagement predict university dropout?”

Fitting a model to a dataset and obtaining statistically significant coefficients does not necessarily demonstrate useful prediction.

Predictive research needs appropriate separation between model development and evaluation, attention to overfitting, suitable performance measures, and ideally evidence about how the model performs beyond the observations used to construct it.

A dataset adequate for examining an association may therefore be inadequate for developing and evaluating a useful prediction model.

Statistical significance does not prove that the data answered the question

A P value answers a statistical question under a model. It does not independently establish that the correct population was sampled, the constructs were validly measured, the design supported the intended inference, or the analysis corresponded to the research question.

You can obtain a highly statistically significant result from data that answer the wrong question very precisely.

For example, a large dataset may establish an extremely precise association between an imperfect proxy and an outcome. If the research question concerns a different construct or a causal effect, statistical precision does not repair the mismatch.

No statistically significant result is required for a question to be answered

The reverse misconception also occurs.

Suppose the research question asks whether two variables are associated and the estimated relationship is small and uncertain. That does not mean “the research question was not answered” simply because P > .05.

The study may have produced an estimate with uncertainty that is itself informative. Whether the evidence is sufficiently precise to distinguish among substantively important possibilities depends on the design and inferential context.

Research questions should not be written as machines whose only acceptable outputs are “significant” and “not significant.”

Ask whether the analysis answers the question rather than merely uses the data

Suppose your question is:

“How do students' patterns of generative AI use change across four years of university?”

You collect data from first-, second-, third-, and fourth-year students at one point in time and compare the groups.

That design can describe differences among cohorts. It does not directly observe how the same students' use changes across four years.

The cross-sectional comparison may still provide useful evidence, but it answers a somewhat different question from longitudinal within-person change.

Small wording differences such as “differ by year level” versus “change over four years” can therefore imply different data structures.

Ask whether your data permit the intended unit of analysis

A question about schools cannot necessarily be answered by treating thousands of students as thousands of independent schools.

Likewise, a question about countries, classrooms, families, organizations, or repeated observations may involve clustered or hierarchical data structures.

The unit implied by the question should correspond to the unit represented and analyzed in the data. Ignoring dependence among observations can create statistical problems, while making claims at one level from evidence collected at another can create conceptual ones.

Triangulation can strengthen an answer but does not compensate automatically for weak evidence

Researchers may use multiple sources of evidence to examine a question. Interviews, observations, documents, surveys, or administrative records can provide complementary perspectives.

This can strengthen interpretation when the sources illuminate relevant aspects of the phenomenon.

But three weak proxies do not automatically become one strong measure simply because they are triangulated. Multiple sources should have a defensible relationship to the question and to one another.

Sometimes the data can answer only part of the question

Consider:

“How does generative AI affect students' writing performance and development of independent writing skills over time?”

You have one semester of course grades.

Those data may provide some evidence about performance during the semester. They do not necessarily provide evidence about development of independent writing skills over time.

You could:

• collect additional evidence;
• narrow the question;
• separate the question into answerable components; or
• acknowledge that the present study addresses only part of the larger problem.

The worst option is to keep the broad question and quietly answer only the part for which data happened to be available.

A question can be scientifically answerable but not answerable by your study

This distinction is worth preserving.

“Does long-term exposure to air pollution affect cognitive development?” is empirically researchable. Your dataset containing one week of self-reported pollution exposure and one cognitive score may simply be inadequate to answer it.

Likewise, a question requiring national administrative records may be answerable in principle even if you cannot access those records.

The failure is not necessarily in the question. It may be a mismatch between the question and your available study.

When this happens, you need to decide whether to change the evidence or change the question.

Do not let available data silently redefine the research question

Suppose you wanted to study “student learning” but could access only final course grades. It is tempting to proceed and use “learning” and “grades” interchangeably.

That substitution may or may not be defensible. Grades can reflect learning, but they may also reflect attendance, assignment completion, participation, grading practices, prior achievement, and other factors.

If the available measure cannot adequately represent the intended concept, acknowledge the limitation or revise the question to match what the data actually capture.

Data availability should constrain claims openly rather than redefine constructs invisibly.

The strongest test is a question-to-evidence map

Before data collection, write the research question at the top of a page and map each part to the evidence needed.

If one row cannot be completed convincingly, you have found a potential answerability problem before it becomes a disappointing results chapter.

When Available Data Answer a Different Question From the One You Wanted

This is not merely cautious wording. “Does AI improve writing ability?” and “Is reported AI use associated with course grades?” are different scientific questions.

Recognizing that difference before analysis protects the study from producing a precise answer to a question it never actually had the data to answer.

Common Mistakes When Deciding Whether Data Can Answer a Question

Run a Data Answerability Check Before You Collect or Analyze

Before finalizing your protocol, take each research question and work backward from the conclusion you would like to be able to write.

That final situation leads directly to the next problem: what to do when an interesting research question cannot be answered directly. Often the solution is not to abandon the problem, but to identify what part of it can be investigated credibly.

Can Your Data Actually Answer Your Research Question?

Frequently Asked Questions About Research Questions and Data

Having Data Is Not the Same as Having an Answer

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.