Does the Research Question Assume a Comparison That Is Not Actually Meaningful?

Just Because Two Groups Can Be Compared, Should They Be?

Researchers frequently build questions around contrasts: users versus non-users, online versus face-to-face students, public versus private institutions, before versus after an intervention, younger versus older participants, high versus low performers.

Statistically, comparing two groups may be straightforward. Scientifically, the comparison can be much harder to justify.

The crucial question is not simply whether the groups differ. It is what their difference would mean. If the comparison groups differ in several important ways, represent fundamentally different populations, experience different contexts, or fail to represent the alternative condition implied by the research question, even an accurately estimated difference may provide a weak answer.

A Useful Comparator Must Help Answer the Actual Question

What Makes One Comparison More Informative Than Another?

Comparator selection is part of the substantive design of a study, not merely an analytical convenience. In comparative-effectiveness research, methodological guidance explicitly links the choice of comparator to the research question and emphasizes that comparison groups should represent meaningful alternatives. The same reasoning is useful well beyond health research.

Before asking whether Group A differs from Group B, ask why Group B is the appropriate reference for understanding Group A.

Begin With the Contrast You Actually Care About

Suppose a researcher asks: “Do students who use generative AI perform better than students who do not?”

What is the substantive contrast? Any AI use versus absolutely no use? Regular use versus no use? AI-supported study versus conventional study strategies? Permitted AI assistance versus an otherwise equivalent task completed without AI?

Those comparisons answer different questions. “Non-users” may seem like the obvious comparator, but it may be a heterogeneous group containing students who avoid AI for very different reasons and use many alternative forms of assistance.

A meaningful comparison begins by defining what alternative condition would make the observed difference informative.

The Comparator Defines the Meaning of the Effect or Difference

Consider a study of a new teaching approach. Comparing it with no instruction asks a different question from comparing it with standard instruction. Comparing it with another active teaching approach asks yet another question.

The intervention itself has not changed, but the comparison has. Consequently, the meaning of “better,” “effective,” or “different” changes as well.

This is particularly important for causal questions. A causal effect is inherently comparative: it concerns outcomes under one condition relative to outcomes under an alternative condition. A poorly specified comparator therefore means a poorly specified causal effect.

Existing Groups May Differ for Reasons Central to the Outcome

Imagine comparing students who voluntarily attend optional tutoring with students who do not. If tutoring attendees obtain higher grades, the difference might reflect tutoring, but it might also reflect motivation, prior difficulties, available time, help-seeking behavior, instructor recommendations, or other factors associated with both attendance and performance.

The comparison is not meaningless simply because confounding exists. Observational research routinely addresses imperfect comparability. The issue is whether the comparison, together with the design and available evidence, can support the intended interpretation.

If the question asks what tutoring causes, this becomes part of the broader need to identify a causal commitment hidden in the research question.

“Users Versus Non-Users” Is Often Less Simple Than It Looks

Technology studies frequently compare users with non-users. Yet adoption is rarely random. People who choose a technology may differ from non-users in digital competence, resources, motivation, prior performance, attitudes, task requirements, institutional support, or opportunity to use the technology.

The meaning of “use” can also vary. A student who asks an AI system to clarify one concept and another who delegates entire assignments to it may both be classified as users.

Before accepting a user/non-user contrast, define both conditions and ask whether the resulting comparison corresponds to the phenomenon you actually want to investigate.

A Before-and-After Comparison Changes More Than Time

Researchers sometimes compare outcomes before and after a policy, technology, curriculum, or institutional intervention and interpret the difference as its impact.

The difficulty is that time itself carries other changes. Participants may mature, external events may occur, institutional practices may shift, measurement may change, and broader trends may affect the outcome. A pre-post contrast can be useful, but “before” does not automatically provide the counterfactual for what would have happened “after” without the intervention.

If the research question is causal, the study needs a defensible strategy for distinguishing the intervention-related change from other temporal changes.

Historical Comparators May Come From a Different World

Comparing a contemporary group with participants or records from several years earlier can sometimes be informative. It can also introduce differences in technology, policy, curriculum, assessment, demographics, economic conditions, professional practice, or data quality.

The question is whether those historical differences interfere with the interpretation of the contrast. A convenient archive is not automatically a suitable comparison condition.

Groups Defined by an Outcome Can Create Circular Comparisons

Suppose researchers divide students into “successful” and “unsuccessful” groups based on grades and then ask whether successful students have higher grades. The answer is built into the grouping rule.

Less obvious versions occur when group definitions contain a variable closely tied to the outcome being compared. Researchers should inspect how categories were created and ensure that the comparison does not merely rediscover the criterion used to define them.

Arbitrary Categories Can Destroy Useful Information

Researchers sometimes convert continuous variables into “high” and “low” groups using a median split or another convenient threshold, then formulate the research question around those categories.

This may be justified when the threshold has substantive meaning, but arbitrary categorization can discard information, reduce statistical power, and create groups that do not correspond to meaningful distinctions in the phenomenon.

Before asking whether “high AI users” differ from “low AI users,” ask what makes those categories substantively different and whether treating use as a continuous or more nuanced exposure would better represent the research problem.

Comparison Requires Comparable Measurement

Even well-chosen groups may be difficult to compare if the outcome does not have sufficiently comparable meaning across them.

For example, comparing course grades across two institutions can be problematic if grading standards, assessments, curricula, or grade distributions differ substantially. The numerical values may share the same scale while representing different academic performances.

This links comparator selection to measurement. Researchers should check whether the variables can be measured in a way that supports the intended comparison.

Different Populations May Answer a Different Question

Suppose one group consists of first-year engineering students from a highly selective university and another consists of graduating business students from an open-admission institution. Comparing their AI use may yield a numerical difference, but attributing that difference to discipline would be difficult because discipline is entangled with year level, institution, admissions context, and potentially many other characteristics.

A comparison does not become scientifically meaningful merely because both groups can be placed in the same table.

Meaningful Does Not Mean Perfectly Matched

Researchers sometimes conclude that groups must be identical on every characteristic except the focal variable. In many real studies, particularly observational research, that is neither possible nor conceptually necessary.

The relevant standard is whether the comparison supports the intended inference under defensible design and analytical assumptions. Some baseline differences can be measured and addressed. Others may be so fundamental that the comparison ceases to illuminate the question.

Why “AI Users Versus Non-Users” May Be the Wrong Comparison

The original comparison could still support a descriptive question about observed differences between self-selected users and non-users. What it cannot do automatically is answer the stronger causal question implied by “improve.”

Common Mistakes When Choosing What to Compare

Choose the Comparator by Working Backward From the Difference You Want to Interpret

Write down the contrast implied by your question in plain language: “I want to know what differs when ___ is present rather than ___.” Then examine whether the proposed groups genuinely represent those two conditions.

Next, list other ways the groups differ that could affect the outcome or alter the meaning of the comparison. The purpose is not to demand impossible perfection. It is to determine whether the comparison can answer the question you intend to ask.

Check Whether Your Proposed Comparison Makes Scientific Sense

Questions About Comparison Groups in Research

The Right Comparator Is the One That Makes the Difference Interpretable

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.