03 · What You Need to Know
What Qualitative Research Can and Cannot Establish About Magnitude
What Does a How Much Question Actually Ask?
The expression "how much" can refer to several different kinds of inquiry.
Sometimes it asks for a numerical quantity, such as the average number of hours instructors spend preparing assessments or the proportion of students who use generative AI.
In other situations, it concerns the extent to which a phenomenon influences people's experiences, practices, or decisions.
Consider these questions:
- How many instructors use generative AI for assessment?
- How much additional preparation time does AI-supported assessment require?
- To what extent does uncertainty about AI policies shape instructors' assessment decisions?
- How deeply does academic pressure affect doctoral students' experiences of research?
The first two questions seek quantities that ordinarily require numerical measurement. The latter questions may concern experienced significance, variation, and processes that qualitative inquiry can investigate.
The wording alone does not determine the method. The intended form of the answer does.
How Is Numerical Magnitude Different From Experiential Extent?
Numerical magnitude refers to quantities expressed through defined units or numerical measures. Experiential extent concerns how substantially or pervasively a phenomenon is understood to affect participants' lives, decisions, or practices.
Numerical magnitude
How many, how often, how long, or how large, expressed through a defined numerical measure and appropriate observations.
Experiential extent
How deeply, widely within an experience, or consequentially a phenomenon shapes participants' interpretations, activities, or circumstances.
For example, a qualitative interview may reveal that uncertainty about AI policies influences nearly every stage of an instructor's assessment planning. This can support an interpretation that the uncertainty is pervasive within that instructor's professional experience.
It does not establish how many hours the instructor spends responding to the uncertainty or what proportion of instructors experience the same problem.
The distinction helps researchers decide whether numerical or experiential evidence is required for the intended conclusion.
Can Qualitative Researchers Count Participants, Codes, or Themes?
Yes. Qualitative research does not prohibit counting.
Researchers may report the number of participants who discussed a particular experience, the distribution of coded observations across cases, or the occurrence of defined events within a dataset.
For example, a researcher might report that 12 of 20 interviewed instructors discussed uncertainty about institutional AI policies.
That statement can be an accurate description of the analyzed interview sample, assuming the coding and reporting procedures are appropriate.
However, it does not automatically mean that 60% of all university instructors experience such uncertainty.
The distinction arises because qualitative participants are often selected purposively according to their relevance to the research question rather than through a probability sampling process designed for population estimation.
Counts should therefore be interpreted within the boundaries of the cases and evidence collected.
Why Is Theme Frequency Not the Same as Theme Importance?
A theme that appears frequently is not necessarily more analytically important than one discussed by fewer participants.
Some experiences may be common but relatively straightforward. Others may be uncommon yet reveal an important institutional contradiction, ethical concern, or previously unrecognized process.
For example, most instructors might discuss the convenience of automated feedback, while only a few describe serious accessibility difficulties experienced by particular students.
The accessibility issue may be highly consequential even if it appears in relatively few interviews.
In reflexive thematic analysis, themes are developed as patterns of shared meaning relevant to the research question, not merely as categories ranked by frequency.
Other qualitative approaches may legitimately use systematic frequency information, particularly when their coding and analytical procedures are designed for that purpose.
The researcher should explain the analytical role of counting rather than assume that more mentions automatically indicate greater importance.
When Can Qualitative Evidence Support Comparisons of Extent?
Qualitative evidence can support carefully bounded comparisons across participants, cases, settings, or periods.
For example, researchers might compare how extensively AI-related policy uncertainty influences assessment practices in two academic departments.
Interviews and observations could reveal that one department has established routines for resolving uncertainty, while instructors in another repeatedly negotiate decisions without formal guidance.
Such evidence may support a contextualized comparison of how uncertainty is experienced and managed.
However, claiming that one department has a statistically higher prevalence of uncertainty would require appropriate quantitative measurement and sampling.
Qualitative comparisons should make clear whether the conclusion concerns differences in meaning, practice, process, or observed occurrence within the selected cases.
Can Qualitative Research Establish How Often Something Happens?
Sometimes, within a clearly bounded observational context.
Suppose a researcher observes ten classroom sessions and systematically records instances in which students challenge AI-generated feedback.
The researcher could report the number of observed challenges across those sessions if the event is defined consistently and the observations are sufficiently complete.
This is a numerical description derived from qualitative fieldwork or structured observation.
However, the count does not automatically establish how often the behavior occurs across all classrooms or throughout an academic year.
Such broader conclusions require attention to the observation schedule, selection of settings, completeness of recording, and population or time period to which the estimate is intended to apply.
Where frequency estimation becomes the principal research objective, a quantitative observational design may be more appropriate.
Can Qualitative Research Estimate Prevalence?
Conventional purposively sampled qualitative interview studies are generally not designed to estimate population prevalence.
Prevalence estimation requires a defined population, a clear case definition, and an appropriate strategy for measuring and representing that population.
For example, interviewing students who have experienced difficulties with AI-supported learning can provide detailed evidence about those difficulties. It cannot establish how common the difficulties are among all students if participants were selected precisely because they experienced them.
There are research designs that combine qualitative classification with quantitative population estimation. For instance, researchers may qualitatively code responses collected from a probability sample and subsequently estimate the proportion of respondents exhibiting particular characteristics.
In that situation, the prevalence estimate depends on the quantitative sampling and measurement framework, even though qualitative interpretation contributes to classification.
Researchers should distinguish qualitative evidence about a phenomenon from the evidence required to estimate its prevalence or frequency.
What About Words Such as Most, Many, Few, and Rarely?
Qualitative researchers sometimes use frequency-related language to describe patterns in their data.
Words such as "most," "many," and "few" may be appropriate when their referent is clear and the underlying evidence supports the characterization.
For example, "Most interviewed participants described uncertainty about the policy" refers to the participants in the study.
By contrast, "Most university instructors are uncertain about the policy" implies a broader population claim that the same interview sample may not support.
Researchers should avoid shifting silently from statements about the analyzed sample to statements about an entire population.
Terms such as "rarely" and "frequently" also require care when the study did not systematically observe event frequency. A participant's statement that an event occurs frequently is evidence about the participant's account, not necessarily an independently verified frequency estimate.
Can Qualitative Research Examine Intensity?
Yes. Qualitative inquiry can investigate how strongly participants experience a phenomenon and what makes that experience consequential.
For example, doctoral students may describe academic uncertainty as a minor inconvenience, a recurring source of frustration, or a defining feature of their research experience.
Qualitative analysis can examine the meanings, circumstances, and consequences associated with these differences.
However, qualitative descriptions of intensity should not automatically be treated as standardized numerical measurements.
If the research question requires comparing average stress scores or estimating the magnitude of a change, an appropriately validated quantitative instrument may be necessary.
Qualitative evidence can help researchers understand what intensity means to participants, while quantitative measures may estimate its distribution under specified operational definitions.
Can Qualitative Evidence Establish the Magnitude of an Intervention's Effect?
Qualitative research can investigate perceived consequences, experiences of change, and processes through which an intervention may influence participants.
However, estimating an average numerical causal effect ordinarily requires quantitative outcome measurements and a credible causal comparison.
For example, interviews may reveal that students believe AI-supported feedback helped them revise their arguments more effectively.
Those accounts do not establish how many points the intervention increased writing scores or how much of an observed improvement was caused by the intervention.
Similarly, a participant's report of substantial improvement may reflect an important experience without providing a standardized estimate of effect magnitude.
When the central question concerns whether an intervention produces a measurable effect, the research design must support the relevant causal contrast.
How Can Qualitative Evidence Inform Later Quantitative Measurement?
Qualitative inquiry may help identify relevant dimensions of a phenomenon before researchers attempt to quantify it.
Suppose little is known about how students experience dependency on AI-supported writing tools.
Interviews and observations might identify distinct experiences, such as difficulty initiating writing without assistance, reliance on automated suggestions, or reduced confidence in independent revision.
These findings could inform the conceptual development of a subsequent measurement instrument.
However, converting qualitative categories into survey items requires additional work. Researchers should examine content relevance, response processes, measurement structure, and other validity considerations appropriate to the intended use.
Qualitative findings can therefore support the development of quantitative evidence without themselves establishing population magnitude.
When Is Mixed Methods Appropriate for a How Much Question?
Mixed methods may be justified when the research question requires both numerical estimation and contextualized understanding.
For example, a university might want to estimate the proportion of students who experience difficulties evaluating AI-generated feedback and understand how those difficulties arise during academic writing.
A quantitative component could estimate the prevalence of defined difficulties, while a qualitative component investigates students' reasoning and practices.
The two components should be integrated according to an explicit research purpose.
However, mixed methods is not automatically necessary when the question requires only a population proportion or only an interpretive account.
Researchers should consider whether integrating quantitative and qualitative evidence is genuinely necessary rather than combining methods by default.
How Should Researchers Report Qualitative Claims About Extent?
Reporting should identify the scope of the evidence and the meaning of any frequency-related language.
If counts are used, researchers should explain what was counted, how categories were defined, whether participants could contribute multiple observations, and how the counts relate to the analytical purpose.
For example, counting the number of times a theme appears in a transcript is not equivalent to counting the number of participants who expressed the theme.
Participants who speak at greater length may contribute more coded segments. Interview prompts may also influence how often particular topics arise.
Researchers should therefore avoid interpreting raw coding frequencies as straightforward measures of prevalence or importance.
When the analysis is primarily interpretive, detailed evidence and transparent reasoning may be more informative than numerical theme rankings.
Watch Out
Do not convert a proportion observed in a purposively selected qualitative sample into a population prevalence estimate. The arithmetic may be correct while the population inference remains unjustified.
