Does Your Question Require Evidence About Change Over Time?

Do You Need to Know What Is Happening Now or How Something Has Changed?

Suppose you want to investigate whether students become more confident using artificial intelligence tools as they progress through a university course. You could survey students at the end of the semester and ask about their current confidence. You might even ask them whether they believe their confidence has improved.

But would either approach establish how their confidence actually changed?

Current confidence and change in confidence are different research targets. A student who reports high confidence at the end of a course may have entered with equally high confidence. Another who reports moderate confidence may have improved substantially from a low starting point.

Questions about change require evidence that represents the relevant temporal process. The challenge is determining whether you need to observe the same individuals repeatedly, compare populations across periods, reconstruct earlier events, or establish the order in which particular developments occurred.

When Does a Research Question Require Evidence Collected Across Time?

How to Identify the Temporal Evidence Your Research Question Requires

First, Determine What Kind of Change You Want to Establish

The phrase change over time can refer to several distinct phenomena. Researchers may want to establish whether individuals develop, whether the composition of a population changes, whether events become more frequent, or how a process unfolds.

Consider these questions:

• Do individual students become more confident using AI tools during a semester?
• Has the proportion of university students using AI tools increased over three academic years?
• How does teachers' acceptance of automated assessment develop during implementation?
• Does student engagement decline more rapidly during particular stages of an online course?

Each question requires temporal evidence, but the units being followed and the intended conclusions differ.

The first concerns individual development. The second concerns population-level change. The third examines a developmental or implementation process. The fourth concerns the shape and timing of a trajectory.

Before selecting a design, specify what is expected to change, the unit in which change occurs, and the period over which it should be observed.

Why a Single Measurement Usually Cannot Establish Actual Change

A measurement at one point provides information about the observed state at that point. Without an appropriate earlier measurement or credible historical evidence, it generally cannot establish how that state differs from a previous one.

For example, a student who scores 85 on a digital literacy assessment at the end of a course may have improved, declined, or remained stable. The final score alone cannot distinguish these possibilities.

Retrospective self-reports can provide information about perceived change. Asking students whether their confidence improved may be useful when the question concerns their own assessment of development.

However, perceived change is not identical to change demonstrated through repeated measurements. Retrospective judgments can be affected by recall error, changing standards of self-evaluation, and knowledge acquired after the earlier period.

A single interview may also reconstruct a sequence of experiences through narrative evidence. Such an approach can be appropriate for understanding remembered trajectories, provided the study does not treat retrospective accounts as equivalent to contemporaneous observation.

Individual Change and Population Change Are Different

One of the most consequential distinctions concerns whether the same units are observed repeatedly.

Suppose a university surveys 500 students in one year and another 500 students the following year. The proportion reporting weekly AI use increases from 40% to 60%.

If both samples adequately represent comparable target populations, this may support an inference that population prevalence increased. It does not establish that particular students increased their own usage.

Some students in the second sample may not have been enrolled during the first survey. Others may have graduated or left the university.

To estimate within-student change directly, the researcher generally needs linked observations of the same students across the relevant periods.

The distinction is central when choosing between repeated cross-sectional and longitudinal designs.

How Many Measurement Points Are Necessary?

The required number of observations depends on the question and the complexity of the temporal pattern.

Two appropriately comparable measurements can establish an observed difference between two occasions. For example, baseline and end-of-semester scores can be used to calculate individual score changes.

However, two measurements provide limited information about what happened between them.

Students may have improved steadily, improved early and subsequently declined, or experienced fluctuations that happened to produce the same final difference.

Three or more appropriately spaced observations can provide more information about the shape of change, although they do not automatically establish a particular growth model or causal process.

For complex trajectories, seasonal variation, or rapidly changing outcomes, more frequent measurement may be necessary.

Researchers should therefore select the number of measurement occasions according to the temporal pattern they need to detect, rather than assuming that a baseline and final measurement will answer every change-related question.

Why the Timing Between Measurements Matters

Measurement intervals determine which changes a study can observe.

If an intervention is expected to affect immediate task performance, measuring outcomes only six months later may miss short-lived changes. Conversely, measuring an outcome immediately after an intervention may not establish whether its effects persist.

Timing also matters when outcomes develop gradually. Changes in professional practice, institutional culture, or learning strategies may require extended observation.

Researchers should consider the expected speed of change, the possibility of delayed effects, relevant academic or organizational cycles, and the duration of follow-up.

For example, student engagement may fluctuate around examinations and assignment deadlines. A study measuring engagement only during examination weeks might mistake temporary academic pressure for a general semester-long pattern.

Decisions about when data are collected therefore influence not only convenience but also the meaning of the findings.

Measurement Must Remain Comparable Across Time

Evidence of change requires confidence that observed differences reflect the phenomenon rather than changes in how it was measured.

Suppose a researcher assesses students' digital literacy using one questionnaire at baseline and a substantially different questionnaire at follow-up.

A difference between the scores may reflect actual development, but it may also result from differences in item difficulty, content coverage, or scoring procedures.

Using identical instruments can improve comparability, although it may introduce practice effects or become inappropriate if the construct or population changes.

Where different instruments are necessary, researchers should establish an appropriate basis for linking or comparing scores.

For latent constructs such as motivation, engagement, or self-efficacy, longitudinal measurement invariance may be relevant. Without sufficient evidence that the construct is measured comparably, apparent mean differences can be difficult to interpret.

In qualitative longitudinal research, comparability does not necessarily mean asking identical questions at every encounter. Researchers may adapt interviews to emerging experiences while maintaining a coherent analytical focus on development and change.

Distinguish Change Scores From Change Trajectories

A change score represents the difference between two measurements. A trajectory describes how an outcome develops across multiple occasions.

For a simple two-wave quantitative study, an individual change score can be calculated as follows:

The result establishes a 13-point observed difference between the two occasions. It does not reveal the trajectory between them, establish that the change exceeds measurement error, or identify what caused the improvement.

When repeated measurements are available, researchers may use approaches such as longitudinal mixed-effects models, growth models, or other methods appropriate to the outcome, sampling structure, and assumptions.

These methods can examine variation in trajectories, but greater analytical sophistication does not compensate for poorly timed or incomparable measurements.

What Does Temporal Ordering Contribute to Explanation?

Temporal ordering is particularly important when a research question concerns causal influence.

If a proposed cause occurs after the outcome, it generally cannot explain that earlier outcome through the proposed forward causal pathway.

Establishing that an exposure precedes an outcome can therefore help address one requirement of causal reasoning.

However, temporal precedence is not sufficient for causation.

Suppose students' AI usage increases before their examination scores improve. The sequence is compatible with an effect of AI use, but other explanations remain possible. Students may simultaneously receive additional tutoring, spend more time studying, or encounter easier assessments.

A causal claim requires a credible comparison between the outcomes expected under different exposure conditions, together with defensible assumptions.

Longitudinal evidence can strengthen causal analysis, but a longitudinal design is not automatically a causal design.

Why Changes in Group Composition Can Mislead

Observed changes may arise because different individuals contribute data at different times.

For example, suppose a university's average student satisfaction score rises between two academic years. The increase might reflect improved services, but it might also reflect changes in the student population, response patterns, or the proportion of first-year and graduating students.

Repeated cross-sectional studies should therefore examine whether the target population, sampling frame, and measurement procedures remain comparable.

Panel studies face a related problem when participants leave the study.

If students with low engagement are more likely to stop responding, the average engagement score among remaining participants may increase even without a corresponding improvement among the original cohort.

Researchers should assess missingness and attrition, consider plausible missing-data mechanisms, and use appropriate analytical or sensitivity procedures when possible.

There is no universal percentage of attrition that automatically makes longitudinal findings valid or invalid. Its implications depend on who is missing, why observations are missing, and the intended analysis.

Can Existing Records Provide Evidence of Change?

Yes. Evidence about change does not always require collecting new data prospectively.

Administrative records, historical documents, archived assessments, institutional reports, and existing longitudinal datasets may provide observations across relevant periods.

For example, a researcher investigating changes in student course completion might analyze several years of institutional records.

However, existing records must be evaluated for changes in definitions, data systems, reporting practices, population coverage, and missing information.

A change in how a university defines course completion could create an apparent trend unrelated to students' actual behavior.

Retrospective evidence can be valuable, but its temporal structure and measurement limitations must support the intended inference.

When Is Qualitative Longitudinal Evidence Appropriate?

Some research questions concern how experiences, meanings, identities, or practices develop over time rather than how numerical scores change.

A researcher might investigate how novice teachers negotiate professional identity during their first year of AI-supported instruction.

Repeated interviews, observations, reflective diaries, or documentary evidence could illuminate transitions, turning points, and changing interpretations.

Qualitative longitudinal inquiry can examine how participants understand earlier experiences differently as new events occur. This is not necessarily a measurement problem; changing interpretation may itself be part of the phenomenon.

Nevertheless, researchers should distinguish changes in participants' experiences from changes in how those experiences are remembered or narrated.

Such evidence is particularly useful when the question concerns development, adaptation, or the unfolding of a process within a specific context.

Does Evidence of Change Automatically Mean Evidence of Improvement?

No. Change is a descriptive concept, whereas improvement involves an evaluative judgment.

An increase in a score is not necessarily beneficial. For example, increased time spent on a learning platform might indicate productive engagement, difficulty completing tasks, or inefficient navigation.

Even when the direction of change is desirable, its practical importance may remain uncertain.

Researchers should establish what constitutes a meaningful outcome, whether the observed change is large enough to matter, and whether the measurement captures the intended benefit.

Questions about intervention effectiveness require additional consideration of causal attribution, not merely evidence that an outcome changed.

Did Students' Confidence in Using AI Change During a Semester?

If the researcher had instead surveyed different representative samples at each occasion, the study might have supported conclusions about changes in the population's average confidence. It would not directly establish individual trajectories.

The distinction matters because the required temporal evidence follows from the precise question, not simply from the number of surveys administered.

Common Mistakes When Interpreting Change Over Time

How to Decide What Temporal Evidence Your Study Needs

Begin by identifying the unit of change. Are you studying the same individuals, a changing population, an institution, a behavior, or a sequence of events?

Next, determine whether the question concerns a difference between two occasions, a longer trajectory, or the development of a process.

Only then should you decide how many observations are needed and how far apart they should occur.

Also examine whether the available evidence is prospective, retrospective, or a combination. Existing records may provide valuable temporal information, but their definitions and coverage must remain appropriate.

If your proposed design cannot establish the kind of change named in your question, reconsider what evidence would actually justify the intended answer.

A defensible study may need a narrower question, such as perceived improvement rather than measured improvement, or population-level change rather than individual development.

Before Studying Change Over Time, Verify Your Evidence Plan

Frequently Asked Questions About Evidence of Change Over Time

Evidence of Change Must Represent the Relevant Temporal Process

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.