01 · The Question
Should You Ask People What They Did or Observe What They Actually Did?
Suppose you want to know how often students use an online learning platform. You could ask them to estimate their usage, observe their behavior, or retrieve system logs. At first glance, the recorded data may seem like the obvious choice: why ask people when the system can tell you?
Now change the question. Suppose you want to know whether students found the platform frustrating, useful, or mentally demanding. A system log cannot simply reveal those experiences.
Self-report, observed, and recorded measures provide different kinds of evidence. Choosing among them is not merely a matter of finding the most accurate data source. You first need to decide what phenomenon you actually want to know about.
03 · What You Need to Know
What Changes When the Source of Measurement Changes?
Self-Report Means the Participant Provides the Information
A self-report measure asks participants to provide information about themselves. This can include attitudes, beliefs, perceptions, feelings, experiences, intentions, behaviors, demographic characteristics, or past events.
Self-report is therefore broader than “subjective opinion.” Asking someone how satisfied they are is both self-report and subjective. Asking someone for their age is also self-report, but age is not inherently a subjective construct. The distinction between objective and subjective measurement is related to, but not identical with, the distinction between self-reported and externally obtained information.
Self-report may be the most appropriate source when the construct concerns something only the participant can readily report, such as perceived usefulness, confidence, attitudes, beliefs, intentions, or experienced discomfort.
Observed Measures Record Behavior or Events Through Observation
Observed measures obtain evidence by watching, detecting, or systematically recording behavior or events rather than relying on the participant to report them afterward.
Observation can be conducted by trained human observers or, depending on the phenomenon, by technological systems. Researchers might record classroom participation, task behavior, interactions, adherence to a procedure, or the occurrence and duration of predefined events.
Observation does not remove measurement decisions. Researchers must define what counts as the target behavior, determine when and where observation occurs, decide how events are coded, and, when human raters are involved, establish sufficiently consistent application of the coding system.
Recorded Measures Use Data Already Captured by a System or Process
Recorded measures draw on information captured through systems, devices, institutional processes, administrative databases, or other records. Examples can include attendance records, examination results, transaction records, learning-management-system logs, sensor readings, or routinely collected administrative data.
These records may reduce dependence on participants' memory, but they are not automatically complete or accurate. A record reflects what the system was designed and able to capture.
An LMS can record that an account accessed a page. That does not necessarily establish who was looking at the screen, whether the material was read carefully, what the student was thinking, or whether meaningful learning occurred.
| Approach |
Where the information comes from |
Particularly useful when |
Potential concern |
| Self-report |
The participant |
Perceptions, experiences, beliefs, intentions, or information participants can report |
Recall, interpretation, response tendencies, social desirability, or reporting ability |
| Observed |
Systematic observation of behavior or events |
The relevant behavior can be meaningfully observed and coded |
Observer effects, sampling of behavior, coding decisions, or rater inconsistency |
| Recorded |
Systems, devices, administrative processes, or existing records |
Relevant events, behaviors, or quantities are reliably captured |
Missing records, system definitions, measurement limitations, or incomplete construct coverage |
Self-Report Can Measure Things Observation Cannot
A common criticism of self-report is that people may not accurately report their own behavior or internal states. That concern is legitimate in many situations, but it does not follow that externally recorded data can replace self-report for every construct.
Consider perceived difficulty. An observer can record how long someone takes to complete a task and how many errors they make. Neither observation tells you directly how difficult the participant experienced the task to be.
Likewise, a student can spend two hours on an online module and still perceive it as easy. Another can spend 30 minutes and experience substantial cognitive effort. Behavior and experience may be related, but they are not interchangeable.
If your construct is explicitly subjective, replacing self-report with an external behavioral proxy may actually move you farther from the thing you intended to measure.
Observed and Recorded Data Can Avoid Some Self-Report Problems
When researchers need information about behavior that can be reliably observed or recorded, external sources can avoid some problems associated with participants' recall and estimation.
If you need the number of assignments submitted through a platform, a reliable submission record may provide more appropriate evidence than asking students to remember how many they submitted. Similarly, if precise timing matters and a system accurately timestamps the relevant event, the recorded time may be preferable to retrospective estimation.
Yet this advantage applies only if the records actually correspond to the target variable. Using an easily available digital trace as an indicator of a broader construct introduces the same problem encountered with indirect measures and proxies: the researcher must justify the inference from what was recorded to what the study claims it represents.
Observation Has Its Own Measurement Problems
Observational data can look straightforward because researchers are recording behavior rather than asking participants about it. In practice, observation involves its own inferential and procedural choices.
What qualifies as “participation”? Does raising a hand count? Answering a question? Contributing to group work? How long must a behavior last before it is recorded? What happens when two observers classify the same event differently?
Researchers using human observers may need clear coding rules, observer training, and evidence concerning inter-rater agreement or reliability where appropriate. They must also consider whether being observed changes participants' behavior and whether the observation period adequately represents the behavior of interest.
Recorded Data Are Not Automatically Ground Truth
The phrase “system-generated data” can create an impression of neutrality. In reality, recorded data are produced by technical and administrative systems with definitions, thresholds, failures, missingness, and design choices.
A fitness device may fail to classify some movement accurately. A learning platform may record a session as active even when the user has walked away. Administrative records may contain delayed entries or coding errors. A digital trace may capture only activity occurring within the particular platform.
Watch Out
Do not treat externally recorded data as “ground truth” merely because participants did not report them. Ask what generated the record, what the system actually detects, what it misses, and whether the resulting variable represents the construct you intend to study.
Disagreement Between Sources Can Be Informative
Suppose students report studying for eight hours per week, while a platform records only four hours of activity. It may be tempting to conclude that the students overreported their study time.
Perhaps they did. But they may also have studied from downloaded materials, printed readings, handwritten notes, books, or other resources outside the platform. The two measures may cover different behavioral domains.
Disagreement can therefore arise from reporting error, recording error, different observation periods, different definitions, or differences in what each source actually captures.
Before deciding that one measure is correct and another is wrong, compare their conceptual and operational definitions.
Using Multiple Sources Can Answer Questions That One Source Cannot
Sometimes the relationship between sources is itself substantively interesting. A researcher might ask whether perceived participation corresponds with observed participation, whether reported study time aligns with digital records, or whether perceived performance matches assessed performance.
Using multiple sources can also reduce dependence on one method, but collecting more data is not automatically stronger measurement. The sources should have a clear role in the research design.
If self-report and recorded behavior represent different constructs, combining them into one score merely because both concern the same broad topic could obscure rather than improve measurement.
