What Should You Do When the Best Evidence for Your Question Cannot Realistically Be Collected?

What if you know what evidence you need but cannot actually obtain it?

You have done the methodological reasoning correctly. You know what evidence would answer the research question. Then you discover that you cannot collect it.

The necessary records are inaccessible. The ideal measurement would be prohibitively expensive. Direct observation would violate privacy or be ethically inappropriate. The relevant population is extremely difficult to reach. The event occurred years ago. An experimental manipulation would be impossible or unacceptable. The platform does not record the variable you need.

Real research frequently encounters a gap between ideal evidence and obtainable evidence. The mistake is not necessarily accepting a compromise. The mistake is using a weaker or different form of evidence while continuing to write the research question and conclusions as though nothing changed.

Use the best defensible alternative, then adjust the claim to match it

Methodological compromise is normal; hidden compromise is the problem

First identify why the ideal evidence is unavailable

Different constraints require different responses. “We cannot collect the ideal evidence” is not yet a diagnosis.

The obstacle may be ethical. It may involve access, cost, time, technology, privacy, legal restrictions, rarity of the population or event, researcher expertise, participant burden, or the simple fact that the relevant event has already occurred.

Understanding the constraint matters because some obstacles can be solved while others should not be overcome. Lack of access might be addressed through permissions or collaboration. Excessive participant burden might be reduced through a less intensive procedure. An unethical experimental manipulation, by contrast, should not become feasible through persistence.

Separate “ideal” evidence from “necessary” evidence

Researchers sometimes imagine one perfect design and treat every alternative as methodologically unacceptable. That can be unnecessarily restrictive.

The strongest conceivable evidence is not always necessary to answer a modest research question. A more practical method may still provide adequate evidence for a carefully bounded claim.

Suppose continuous behavioral observation would provide extremely detailed evidence of how students study, but your actual question concerns students' reported study strategies. Appropriate self-report evidence may be entirely sufficient because the question itself concerns reported strategies.

The first task is therefore to return to the evidence the research question genuinely requires and distinguish methodological necessity from methodological wish lists.

Ask what the unavailable evidence would have allowed you to claim

A useful way to evaluate the problem is to identify the conclusion that the ideal evidence would have supported.

Suppose you wanted to examine actual use of an educational platform but cannot obtain system logs. You can still ask participants how often they use the platform. Yet the evidence has changed from recorded behavioral traces to self-reported behavior.

If you retain the alternative, your claim should change accordingly. You may be able to describe how frequently participants report using the platform. You should not silently present those estimates as though they were system-recorded behavior.

This distinction between self-report and independently recorded measures is often central when researchers need a feasible substitute.

Look for another source of the same evidence

Sometimes the evidence is available, but not from the source you originally planned.

If institutional records cannot be accessed, another authorized dataset may contain comparable variables. If direct observation of a process is impossible, archival records, documents, recordings, or other existing sources may provide relevant evidence. If one participant group cannot reliably report a phenomenon, another appropriately positioned source may be able to.

This is where considering whether suitable existing data can replace new primary collection can substantially change the feasibility of a study.

The substitute source still requires evaluation. Different datasets can use different definitions, populations, periods, or measurement procedures. “Contains a variable with the same name” is not sufficient evidence of equivalence.

Consider a proxy when the target cannot be measured directly

Researchers frequently use proxy measures when direct measurement is impossible, impractical, or unavailable. A proxy is an observable measure used to represent another phenomenon of interest.

This can be entirely legitimate. Many theoretical constructs are never directly observable and depend on indicators. Practical research also uses proxies when more direct evidence cannot be obtained.

The key is to evaluate the inferential distance. Ask how strongly the proxy is related to the target, what theoretical or empirical evidence supports that relationship, what alternative processes could produce the proxy, and what aspects of the target it fails to capture.

The distinction between direct and indirect evidence becomes particularly important when compromise pushes the design toward a more distant indicator.

Narrow the research question when the evidence becomes narrower

Sometimes the best solution is not finding another measure. It is asking a question that the available evidence can actually answer.

Suppose you originally ask: “How does generative AI use affect students' learning?” You cannot obtain valid learning-outcome measures, but you can collect detailed evidence about students' perceptions of how AI affects their learning.

Those perceptions may be worth studying, but they answer a different question. A more defensible question might ask how students perceive the influence of generative AI on their learning or how they describe changes in their learning practices.

The revised question is narrower, but it is methodologically coherent. A broad question supported by mismatched evidence is not stronger merely because its wording sounds more consequential.

Sometimes you need to narrow the claim rather than the question

The evidentiary limitation may become clear only after data collection or analysis. Perhaps a planned measure performed poorly, substantial data are missing, or access to one source was unexpectedly lost.

In such cases, changing a preregistered or previously approved research question after seeing the results may create other methodological concerns. The more appropriate response may be to retain the original question but acknowledge that the available evidence permits only a partial answer or weaker inference.

Researchers should document consequential departures from the original plan and distinguish analyses or interpretations that became necessary after the limitation emerged.

Feasibility can justify a different data collection method

Suppose repeated in-person observations would best capture a process, but travel and access make them impossible. Remote observation, participant diaries, interviews, existing recordings, or another procedure may provide some relevant evidence depending on the question.

The alternative should be evaluated for what it loses and what it preserves. An interview about a process may preserve participants' interpretations while losing direct observation of behavior. A diary may provide information closer to the time of the event but depend on participant compliance and self-report.

This is why participant burden and practical constraints legitimately affect method selection, provided the resulting evidentiary consequences are acknowledged.

Do not solve an access problem by making an unjustified causal claim

Some research questions require designs that are difficult or impossible to implement. Causal questions are an important example.

If random assignment is unethical or infeasible, researchers may use appropriate observational, quasi-experimental, natural-experimental, or other designs depending on the question and available conditions. These approaches can sometimes support strong causal inference, but not merely because an experiment was unavailable.

The design must address relevant alternative explanations using assumptions and analytical strategies appropriate to that design. If it cannot, the claim should remain associational rather than causal.

Practical constraints do not change the logical requirements of the inference.

Do not compensate for weaker evidence by collecting more of it

A larger sample does not automatically repair a poor proxy. More interviews do not turn reported behavior into observed behavior. Several indirect measures do not become direct merely through accumulation.

Additional evidence can help when it addresses a specific limitation. Multiple indicators may provide a richer representation of a construct. Different sources may reveal whether an interpretation is robust across contexts. But more sources or methods do not automatically make weak evidence stronger.

The relevant question is whether the additional evidence reduces an important uncertainty rather than merely increasing the volume of data.

Sometimes the defensible decision is not to conduct the study as planned

There are cases in which no feasible alternative can answer the intended question adequately.

If the central outcome cannot be measured, the necessary population cannot be accessed, the required comparison is impossible, or every available proxy is too weak for the intended claim, the study may need substantial redesign. In some cases, the responsible decision is to postpone or abandon that particular question rather than conduct a study that cannot answer it.

This can be frustrating, particularly after substantial planning. Yet methodological feasibility is not established by how much work has already been invested. Sunk costs remain surprisingly unimpressed by research ethics committees.

Limitations should explain consequences, not merely confess imperfections

When compromise is unavoidable, reporting should explain what changed and what that means for interpretation.

“Due to practical limitations, self-reported usage was collected instead of system-recorded usage” is only the beginning. The reader also needs to know that the resulting estimates concern reported behavior and may be affected by recall or response processes.

A useful limitation therefore connects the constraint to the evidence and then to the claim: what could not be collected, what was used instead, how the substitute differs, and what inference should consequently be made more cautiously.

When the ideal measure is unavailable, change the inference

The alternative is not worthless because it is imperfect. It becomes problematic only if the researcher erases the distinction between the evidence that was wanted and the evidence that was actually collected.

Common mistakes when ideal evidence is unavailable

Preserve the question, the evidence, or the claim, but do not pretend all three remained unchanged

When ideal evidence becomes unavailable, map the consequences explicitly. Write down the evidence you wanted, the evidence you can obtain, and the difference between them. Then decide whether that gap is small enough to accept or large enough to require redesign.

This reasoning may also reveal that your original question was broader than necessary. Revisiting the alignment between the question and the collection method can identify a narrower but genuinely answerable study.

A feasible question answered well is generally more informative than an ambitious question answered only rhetorically by data that never had the capacity to resolve it.

When the evidence you wanted cannot be obtained

Frequently asked questions about unavailable research evidence

When the evidence changes, the inference may need to change too

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.