03 · What You Need to Know
How Underdeveloped Components Affect Mixed-Methods Quality
What Is an Underdeveloped Mixed-Methods Component?
An underdeveloped component is a quantitative or qualitative part of a mixed-methods study that lacks sufficient methodological development for the contribution it is expected to make.
The problem may involve an unclear research purpose, unsuitable sampling, inadequate measurements, superficial analysis, or unsupported conclusions.
For example, a qualitative component intended to explain why an intervention worked may rely only on a few general satisfaction comments. Those comments might provide useful descriptive feedback, but they may be insufficient to establish a causal process.
Likewise, a quantitative component intended to estimate population prevalence may use a small convenience sample without considering selection or measurement limitations.
Underdevelopment should be assessed relative to the component's intended inference, not merely its sample size, length, or relative emphasis.
Does Mixed Methods Require Equally Large or Equally Detailed Components?
No. Mixed-methods designs may intentionally assign different levels of priority to quantitative and qualitative components.
A randomized intervention evaluation may contain a smaller embedded qualitative process evaluation. A qualitative instrument-development study may include a focused quantitative pilot assessment.
These arrangements can be legitimate when the components are adequate for their respective purposes.
Equal sample sizes are neither necessary nor generally meaningful because quantitative and qualitative sample adequacy follows different methodological considerations.
The central issue is whether each component can support the specific contribution assigned to it.
A small qualitative component may be appropriate for a narrowly defined interpretive task, while a much larger component may remain inadequate if its sampling or analysis does not address the question.
How Can a Weak Qualitative Component Undermine a Mixed-Methods Study?
Qualitative weaknesses can arise when researchers collect limited or poorly targeted evidence and then use it to support broad explanations.
For example, interviews may be conducted only with participants who completed an intervention successfully. Their accounts may provide useful insight into successful participation but limited evidence about barriers, disengagement, or implementation failure.
Leading interview questions may encourage participants to confirm the researcher's assumptions.
Superficial coding may produce broad categories without examining meaning, context, contradictions, or relationships among observations.
Another problem occurs when researchers treat the number of times a theme is mentioned as proof of its explanatory importance.
These weaknesses become especially consequential when the qualitative component is expected to explain a quantitative result.
The researcher may have a credible statistical finding but an inadequately supported account of why it occurred.
How Can a Weak Quantitative Component Undermine a Mixed-Methods Study?
A quantitative component may be underdeveloped when its measurements, sampling, statistical procedures, or causal assumptions do not support the intended inference.
Suppose a qualitative study develops a conceptual framework describing how instructors negotiate trust in AI-supported assessment.
The researcher then distributes a brief questionnaire to 25 conveniently available colleagues and claims that the resulting percentages validate the framework across higher education.
The survey may provide preliminary feedback, but it would not automatically support broad population inference or establish the validity of the conceptual framework.
Similarly, a quantitative component may use a newly created scale without examining whether its items adequately represent the intended construct.
Statistical significance does not compensate for invalid measurement, biased sampling, or an inadequate design.
The quantitative component must therefore be evaluated according to the claim it is expected to support.
What Are the Most Common Forms of Underdevelopment?
Underdevelopment can occur within either component or in the relationship between them.
| Problem |
Illustrative example |
Potential consequence |
| Unclear purpose |
Interviews are added without specifying what they should establish. |
The qualitative component contributes little to the central question. |
| Inadequate qualitative sampling |
Only enthusiastic participants are interviewed about implementation barriers. |
Important experiences and alternative explanations may be overlooked. |
| Inadequate quantitative sampling |
A convenience survey is used to estimate population prevalence. |
The intended population inference may be unsupported. |
| Weak measurement |
A single satisfaction item is treated as a measure of learning effectiveness. |
The outcome does not represent the intended construct. |
| Superficial qualitative analysis |
Interview quotations are grouped without a coherent analytical procedure. |
Interpretive claims may lack sufficient evidentiary support. |
| Inadequate statistical analysis |
Clustered classroom observations are analyzed as independent individuals. |
Uncertainty estimates and statistical conclusions may be misleading. |
| Weak integration |
Quantitative and qualitative findings are reported separately without relating them. |
The claimed mixed-methods contribution remains unclear. |
| Inferential overreach |
Participant accounts are treated as proof of a causal mechanism. |
The integrated conclusion exceeds the evidence. |
These problems differ in severity. Some can be addressed through clearer reporting or more appropriate analysis. Others may require additional evidence or substantial redesign.
Why Does Weak Integration Matter Even When Both Components Are Strong?
A study may contain a rigorous quantitative analysis and a rigorous qualitative analysis yet still lack a coherent mixed-methods contribution.
For example, quantitative data might estimate students' AI literacy, while qualitative interviews investigate instructors' general attitudes toward educational technology.
Both components may be valuable independently. However, their relationship may be too indirect to support the proposed overarching explanation.
Integration requires more than presenting results under adjacent headings.
Researchers should explain how the findings relate, which constructs or cases connect them, and what the combined interpretation establishes.
Fetters, Curry, and Creswell emphasize integration as a defining feature of mixed-methods design, including integration through connecting, building, merging, and embedding.
If integration is absent, the project may be better described as multiple complementary studies rather than one integrated mixed-methods investigation.
Can the Second Component Create an Unsupported Causal Explanation?
Yes. This is a particularly important risk in intervention research.
Suppose a randomized trial estimates that an AI-supported feedback system improves writing performance.
Researchers then interview a small group of successful participants who report that automated feedback helped them recognize weaknesses in their arguments.
These accounts may be consistent with a proposed mechanism.
However, they do not automatically establish that recognizing argumentative weaknesses caused the improvement.
Participants who improved may differ from those who did not, and retrospective accounts may not capture every relevant influence.
The causal-effect estimate and mechanism interpretation therefore require separate evaluation.
When the study seeks evidence about mechanisms, the qualitative component must be capable of examining the proposed process and relevant alternative explanations.
Does an Underdeveloped Component Invalidate the Stronger Component?
Not necessarily.
Suppose the quantitative component is a well-conducted randomized trial, while the qualitative follow-up provides limited explanatory evidence.
The trial may still support a credible estimate of the intervention's effect, subject to its own design and analysis.
What becomes questionable is the additional claim that the qualitative evidence establishes why the effect occurred.
Similarly, a rigorous qualitative investigation may retain its interpretive contribution even when a supplementary survey cannot support the intended population estimate.
Researchers should therefore evaluate component-specific conclusions separately from integrated conclusions.
The problem is not that the entire project automatically becomes invalid. It is that the weakest evidentiary connection may constrain what the integrated study can legitimately claim.
How Does an Underdeveloped Component Affect the Study's Research Questions?
Mixed-methods studies often include an overarching question and component-specific questions.
For example, an intervention study might ask whether AI-supported feedback improves writing performance and how students experience the feedback process.
If the qualitative component contains insufficient evidence about students' experiences, the second question may remain inadequately answered even if the effectiveness question is addressed convincingly.
Researchers should avoid treating favorable findings from one component as compensation for the other component's limitations.
The study's objectives, methods, results, and conclusions should remain aligned.
When a component cannot answer its assigned question, the researcher may need to narrow that question or reconsider its inclusion.
Can a Small Qualitative Sample Still Be Adequate?
Yes. Small samples are not inherently underdeveloped.
Qualitative sample adequacy depends on the study's scope, participant relevance, methodological orientation, and depth of evidence.
Malterud and colleagues describe information power as one way to consider how sample characteristics and study aims influence the amount of data needed.
For example, intensive interviews with a small number of participants may be appropriate for a focused investigation of a specific experience.
However, the same sample may be insufficient for explaining variation across numerous institutions, participant groups, and implementation conditions.
The researcher should justify adequacy in relation to the intended qualitative contribution rather than a universal numerical threshold.
Similarly, a small quantitative pilot may be suitable for assessing feasibility while being insufficient for estimating a precise intervention effect.
Can a Pilot Component Be Included Without Weakening the Study?
Yes, provided its role is described accurately.
A pilot quantitative component may examine feasibility, item functioning, recruitment procedures, or preliminary variability estimates.
A preliminary qualitative component may help refine interview questions, identify relevant concepts, or inform subsequent sampling.
These contributions can be legitimate without supporting confirmatory claims.
The problem arises when preliminary evidence is presented as though it establishes population prevalence, validates a complete theory, or proves a causal mechanism.
Researchers should distinguish feasibility evidence, exploratory findings, and confirmatory conclusions.
A component is not weak simply because it is preliminary. It becomes methodologically problematic when the claims exceed its intended and actual evidentiary capacity.
How Can Researchers Strengthen an Underdeveloped Qualitative Component?
The appropriate improvement depends on the limitation.
If participant selection is inadequate, researchers may need to include cases representing relevant variation or contradictory experiences.
If interviews are superficial, more focused questions, task-based observations, or additional contextual evidence may be necessary.
If analysis is underdeveloped, researchers should clarify the analytical approach and examine how interpretations are supported by the data.
Reflexivity, documentation of analytical decisions, and consideration of alternative interpretations may improve transparency and credibility.
However, these practices should be applied consistently with the selected qualitative tradition.
For example, inter-rater agreement is not a universal requirement across all forms of thematic analysis.
Adding more quotations without strengthening the analytical reasoning is unlikely to resolve a substantive interpretive weakness.
How Can Researchers Strengthen an Underdeveloped Quantitative Component?
Quantitative improvements may involve clearer operational definitions, better instruments, appropriate sampling, or revised analytical procedures.
If the study aims to estimate prevalence, researchers should examine population coverage, selection, nonresponse, and measurement.
If it aims to estimate an intervention effect, the design must support a credible causal comparison.
If it aims to test a newly developed scale, researchers need evidence supporting the intended score interpretation and measurement structure.
Additional observations may improve precision, but they cannot automatically correct systematic bias or invalid measurement.
Researchers should identify which evidence is insufficient for the intended claim before deciding how to strengthen the component.
What If the Study Cannot Support Two Rigorous Components?
Time, funding, access, and methodological expertise may constrain the scope of a mixed-methods project.
When these constraints prevent adequate development of both components, the researcher should reconsider the study's priorities.
One option is to retain the component most essential to the central question and remove the supplementary component.
Another is to narrow the second component's objective so that the available evidence can support a modest but useful contribution.
A third is to conduct the components sequentially across separate studies, allowing each to receive appropriate methodological attention.
The choice should follow the research purpose rather than a commitment to retaining the mixed-methods label.
How Should Researchers Decide Whether to Remove a Component?
Removing a component may be appropriate when it contributes little to the central inference or cannot be strengthened adequately.
Consider what would happen to the research question if the component were removed.
If the main question remains fully answerable, the component may be optional.
If removing it leaves an essential part of the question unresolved, the researcher may need to strengthen the component or revise the question.
Researchers should also examine whether the component introduces unsupported conclusions that could mislead readers.
A narrower study with coherent evidence may be more defensible than a broader study whose integration depends on weak assumptions.
What Should Be Reported When One Component Has Important Limitations?
Limitations should be reported at the component level and at the level of integration.
For example, researchers might explain that the qualitative interviews provide exploratory insight into participants' experiences but cannot establish the prevalence of those experiences or identify the causal mechanism behind the quantitative effect.
Similarly, a supplementary survey may describe responses among the sampled participants without supporting population generalization.
Integrated conclusions should make clear which findings are supported, which remain provisional, and which questions were not adequately answered.
Transparent reporting does not eliminate methodological weaknesses, but it helps prevent those weaknesses from being concealed within an overly broad conclusion.
How Do Reporting Standards Help Identify Weak Components?
Mixed-methods reporting guidance encourages researchers to explain why the approach was selected, how the components were designed, and how their findings were integrated.
The GRAMMS framework, for example, includes attention to methodological justification, component-specific procedures, integration, and limitations.
These reporting considerations can help reveal whether a component has a clear role and whether the integrated inference is supported.
However, compliance with a reporting checklist does not itself establish methodological quality.
Researchers must still evaluate the validity of the measurements, adequacy of sampling, analytical reasoning, and assumptions underlying their conclusions.
Watch Out
Do not assume that adding a second method automatically compensates for limitations in the first. Weak qualitative evidence cannot simply repair an inadequate causal comparison, and weak quantitative evidence cannot establish population generalizability for an interpretive account. Integration does not remove the inferential requirements of either component.
