Does Your Question Require Evidence About Harms as Well as Benefits?

What If an Intervention Works but Also Creates Problems?

Suppose a university introduces an AI-supported learning platform and finds that students who receive the intervention achieve higher examination scores than students in a comparison group. The result appears encouraging, particularly if the research design supports a causal interpretation.

But what if the same students spend substantially more time completing assignments, experience greater academic pressure, or become increasingly dependent on automated assistance? What if the platform improves average performance while creating accessibility difficulties for particular groups?

These possibilities do not automatically negate the observed benefits. They do, however, raise questions that an effectiveness estimate focused on examination scores may not answer.

When researchers want to determine whether an intervention is beneficial overall, suitable for adoption, or preferable to an alternative, examining intended benefits alone may provide an incomplete evidentiary basis. Potential harms, burdens, and unintended consequences may also matter.

Harms Evidence Is Needed When Potential Negative Consequences Could Change the Interpretation or Decision

How to Investigate Benefits, Harms, and Unintended Consequences

What Counts as a Benefit or Harm in Research?

A benefit is a favorable consequence of an intervention or activity, assessed in relation to an outcome that matters to the relevant population or decision. A harm is an unfavorable consequence, although its operational definition varies across disciplines.

In clinical research, harms may include adverse events, complications, or other negative health outcomes. In educational research, relevant harms might include increased workload, exclusion from learning opportunities, privacy-related consequences, or deterioration in particular aspects of learning.

Researchers should distinguish outcomes that are directly experienced by participants from consequences affecting other individuals, institutions, or communities.

For example, an educational technology intervention might improve students' writing performance while increasing teachers' administrative workload. The first outcome concerns students, while the second concerns another stakeholder group. Both may be relevant to a decision about institutional adoption.

Not every inconvenience should automatically be classified as a serious harm. Researchers need clear definitions, meaningful outcome measures, and a defensible explanation of why a particular consequence matters.

How Are Harms Different From Research Risks?

Research risks and intervention harms are related but distinct.

A study examining an AI-based assessment system might expose participants to confidentiality risks through its data collection procedures. Separately, the assessment system itself might generate misleading feedback or disadvantage students whose work is poorly represented by its underlying models.

Ethical review commonly addresses risks associated with conducting the study. An evaluation of the intervention's overall value may additionally require evidence about adverse consequences that would remain relevant during routine use.

The distinction matters because ethical approval does not establish that the intervention is harmless or beneficial.

When Must Harms Be Considered Alongside Effectiveness?

Harms become particularly important when the intended research conclusion goes beyond whether an intervention improves one selected outcome.

Consider three questions about an AI-supported tutoring program:

• Does the program improve students' mathematics scores?
• Does the program improve mathematics scores without increasing academic workload or widening performance disparities?
• Should the university adopt the program as part of routine instruction?

The first question primarily concerns effectiveness on a specified outcome. The second explicitly requires evidence about benefits and potential negative consequences. The third involves a broader decision that may depend on effectiveness, harms, costs, feasibility, and institutional priorities.

Even when a study focuses on a single primary outcome, foreseeable serious adverse consequences should not simply be ignored. However, the appropriate level of monitoring and analysis should remain proportionate to the risks and purpose of the study.

Researchers should first establish what effectiveness evidence can demonstrate before deciding which additional outcomes are necessary for a broader benefit-harm assessment.

What Types of Harms Might Researchers Investigate?

Potential harms depend on the intervention and population. A useful classification should emerge from the substantive research problem rather than from a generic inventory.

These examples identify plausible outcomes, not established effects of educational technology. Whether they occur, how frequently they occur, and whether the intervention causes them are empirical questions.

Some consequences may be beneficial for one group and burdensome for another. For example, additional formative assessments could support learning while increasing instructional preparation time.

How Do You Identify Which Potential Harms to Measure?

Researchers should begin with the intervention's intended functions and plausible pathways through which adverse consequences could arise.

Relevant sources include prior studies, theoretical explanations, stakeholder consultation, preliminary observations, and documented experiences with similar interventions.

Participants and other affected groups may identify consequences that researchers would otherwise overlook. In education, this might include accessibility problems, changes in perceived autonomy, or pressures associated with additional monitoring.

The resulting outcomes should be prioritized according to their plausibility, potential severity, relevance to the intended decision, and feasibility of measurement.

A low-probability but potentially serious consequence may deserve attention even when it is difficult to observe in a small study. Conversely, measuring every conceivable minor inconvenience may dilute the evaluation without improving its usefulness.

When relevant consequences are poorly understood, exploratory qualitative research or preliminary monitoring may help identify candidate outcomes for later systematic evaluation.

What Is the Difference Between an Adverse Event and an Intervention-Caused Harm?

An adverse event is an unfavorable occurrence observed during a study or following exposure to an intervention. Its occurrence does not, by itself, establish that the intervention caused it.

For example, a participant may report increased stress during a semester in which an educational platform is introduced. That stress could be related to the platform, examination demands, personal circumstances, or other factors.

Researchers should therefore distinguish the occurrence of adverse outcomes from causal attribution.

Depending on the field and study design, adverse-event reporting may include the nature of the event, timing, severity, seriousness, frequency, and assessment of possible relatedness to the intervention.

Formal regulatory definitions and reporting obligations differ across clinical, educational, and other research contexts. Researchers should follow the requirements applicable to their study rather than importing clinical terminology indiscriminately.

When the research question asks whether an intervention increases the risk of an adverse outcome, a credible comparison is usually necessary.

Why Is the Absence of Reported Harms Not Proof of Safety?

A study may detect no adverse outcomes because the intervention is genuinely low risk. But other explanations are possible.

The sample may be too small to observe uncommon events. Follow-up may be too short to capture delayed consequences. Participants may not recognize or report particular harms, or the study may not have measured the relevant outcomes.

Consider a hypothetical study involving 40 participants in which no adverse events are observed. This does not establish that the true event probability is zero.

For independent observations with a constant event probability, the commonly used rule of three gives an approximate upper 95% confidence bound of 3 divided by the sample size when no events occur. For 40 participants, that is approximately 7.5%.

This approximation depends on its statistical assumptions and is not appropriate for every study structure, such as clustered observations or varying exposure periods. It nevertheless illustrates why zero observed events should not be interpreted as zero possible risk.

Researchers should report what was monitored, how adverse outcomes were ascertained, and the limitations of the available evidence.

How Can Researchers Compare Benefits and Harms?

A benefit-harm assessment requires more than placing favorable and unfavorable outcomes in separate lists.

The researcher must consider the magnitude, frequency, severity, timing, and uncertainty of the outcomes, together with whose interests are affected.

For example, a learning intervention might improve examination scores while increasing average study time. Whether the trade-off is acceptable depends partly on the size and educational importance of the improvement, the additional burden, and the preferences of affected students and institutions.

Different outcomes are not always directly commensurable. An examination-score difference cannot simply be subtracted from a measure of psychological distress to produce a meaningful net-benefit score.

Some decision-analytic approaches use explicit weighting, utility measures, or multicriteria assessment. Such approaches require defensible assumptions about how different consequences are valued.

When those assumptions are unavailable, transparent presentation of separate benefit and harm estimates may be more informative than an artificial composite measure.

How Do Harms Affect Different Groups?

Average outcomes can conceal important variation.

An intervention may improve average performance while offering smaller benefits or greater burdens to participants with particular characteristics. Such differences may be relevant to equity, accessibility, and decisions about implementation.

For example, requiring continuous access to an online platform could impose different participation burdens on students with reliable and unreliable internet connectivity.

However, identifying a difference between groups does not automatically establish that the intervention caused an unequal effect.

Researchers should consider baseline differences, selection processes, measurement validity, and whether subgroup comparisons are sufficiently precise.

Where possible, important subgroup analyses should be justified and specified in advance. Exploratory findings may still be useful, but they should be presented with appropriate caution.

Understanding these differences may require evidence about the circumstances in which participants encounter the intervention.

Should Harms Be Measured During Implementation or Only Afterward?

The timing of measurement should reflect when potential adverse consequences could arise.

Some harms occur immediately, such as technical exclusion during an online assessment. Others may develop gradually, such as sustained workload increases or changes in learning practices.

Monitoring only at the end of an intervention may miss temporary adverse experiences, withdrawals, or problems that led participants to discontinue use.

Repeated assessments, implementation logs, administrative records, and appropriate participant-reporting procedures can help document relevant events.

When an intervention is modified in response to emerging problems, researchers should record those changes. The observed outcomes may then reflect both the original intervention and subsequent adaptations.

This makes implementation evidence useful for interpreting how adverse consequences emerged or were addressed.

How Should Harms Be Reported?

Transparent reporting should distinguish prespecified outcomes from those identified after examining the data.

For each important adverse outcome, researchers should explain how it was defined, measured, and analyzed. Where relevant, they should report the number of participants experiencing the outcome, the denominator, timing, severity, and uncertainty around comparative estimates.

Harms should not be omitted merely because they were statistically nonsignificant or inconsistent with the intervention's intended benefits.

The CONSORT Harms 2022 extension provides guidance for reporting harms in randomized trials. Other research designs may require different reporting approaches, but the underlying principles of transparency and appropriate ascertainment remain useful.

Researchers should also distinguish an outcome that was measured but not observed from one that was never investigated.

Evaluating Learning Benefits and Additional Workload

In this hypothetical study, a five-point improvement does not automatically outweigh two additional hours of weekly work. The two outcomes use different scales and reflect different aspects of students' educational experience.

Nor does the additional workload automatically make the intervention undesirable. Some students or institutions may consider the extra time acceptable if the learning improvement is substantial and durable.

The researcher's responsibility is to provide credible evidence about the relevant consequences and explain the uncertainties. Deciding how much additional workload is acceptable may require stakeholder judgments that cannot be settled by statistical significance alone.

Common Misconceptions About Harms and Benefits

Decide Which Negative Consequences Could Change Your Research Conclusion

Begin by distinguishing the immediate outcome your intervention is intended to improve from the broader judgment the study is expected to inform.

If your purpose is narrowly to estimate an effect on a particular outcome, you may not need a comprehensive benefit-harm evaluation. However, foreseeable consequential harms should still be considered in study planning, ethical oversight, and reporting.

If the research will inform adoption, scaling, or policy decisions, a broader evaluation may be necessary.

Where the study cannot provide reliable evidence about an important adverse outcome, acknowledge that limitation. The absence of evidence should not be converted into a reassuring claim of safety.

It is also useful to distinguish the evidence needed to estimate consequences from the judgments required to value them. Research can clarify the likely magnitude and uncertainty of different outcomes, but choices among competing priorities may remain normative or policy decisions.

Ultimately, the appropriate scope follows the claims and decisions the study is intended to support.

Before Evaluating Harms and Benefits, Check These Questions

Frequently Asked Questions About Harms and Benefits in Research

A Beneficial Outcome Does Not Eliminate the Need to Examine Possible Harms

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.