How Do You Synthesize Studies That Use Different Methods?

What Do You Do When the Studies Are Studying the Same Problem Differently?

Your literature may contain experiments, surveys, interviews, observations, case studies, longitudinal designs, and mixed methods research. They concern the same broad problem, but they do not produce the same kind of evidence.

This creates a genuine synthesis problem. If you separate every method completely, you may miss relationships across the literature. If you combine everything as though the findings were directly equivalent, you may make comparisons the research designs cannot support.

The task is therefore not to make methodological differences disappear. It is to determine what each study contributes to the review question, which findings can reasonably be compared, and how different forms of evidence may converge, diverge, or complement one another.

Integrate the Evidence Without Pretending the Methods Are Equivalent

Different Methods Can Contribute to the Same Question in Different Ways

First ask what each method allows you to know

An experiment estimating an intervention effect and an interview study exploring participants' experiences are not two interchangeable attempts to produce the same answer. Their research questions, forms of data, analytical procedures, and inferential possibilities differ.

The experiment may estimate whether an intervention changes a specified outcome under particular conditions. Interviews may illuminate how participants experience the intervention, why they use it in particular ways, or which aspects they perceive as consequential.

Both can matter to a review of the intervention, but they contribute differently.

Separate method from relevance

Methodological difference does not automatically make studies impossible to synthesize. The more important question is whether their findings bear meaningfully on the same review question or on related dimensions of it.

A survey examining the association between feedback frequency and engagement, an experiment testing a feedback intervention, and interviews exploring how students interpret feedback may all contribute to understanding feedback and engagement. Their findings cannot simply be pooled, but they may form a coherent explanatory account.

Determine what is genuinely comparable

Before combining findings, examine the studies' populations, interventions or exposures, comparators, outcomes, constructs, settings, timing, and design characteristics. Cochrane guidance emphasizes that examining these characteristics is a necessary precursor to synthesis because it helps determine which studies can legitimately be grouped.

Two quantitative studies may be less comparable than one quantitative and one qualitative study if the quantitative studies operationalize the central phenomenon in fundamentally different ways. Method labels alone therefore do not determine comparability.

Different methods may converge on a broader interpretation

Suppose a controlled study reports improved course completion after a mentoring intervention. Survey research finds that students who report stronger mentoring relationships also report greater academic belonging. Interview studies describe mentors as helping students interpret institutional expectations and recover from setbacks.

These findings do not constitute three replications of the same result. Yet they may support a broader interpretation in which mentoring is associated with persistence and may operate partly through relational and navigational support.

The wording matters. The synthesis should preserve the difference between an estimated intervention effect, an observed association, and participants' accounts of their experiences.

Different methods may explain apparent disagreement

Methodological variation can sometimes explain why findings appear inconsistent. A questionnaire may detect little change in a broad construct, while interviews reveal substantial change in one specific dimension of participants' experience. An experiment may find a small average effect while qualitative evidence suggests substantial variation in how participants engage with the intervention.

Rather than deciding that one method is correct and the other is wrong, ask whether they are observing different dimensions, time scales, populations, or mechanisms.

Complementarity is not the same as confirmation

Researchers sometimes describe qualitative findings as “confirming” quantitative results when the two merely point in compatible directions. That language can flatten important differences.

If a quantitative study reports improved retention and interviews suggest that participants felt more supported, the interviews do not necessarily confirm the measured retention effect. They may offer complementary evidence about participant experience or a possible explanation that would require further investigation.

Sometimes studies should be synthesized separately first

When evidence types are substantially different, one defensible strategy is to synthesize similar forms of evidence separately and then integrate the resulting syntheses.

This principle is explicit in formal mixed methods evidence synthesis. JBI distinguishes approaches in which quantitative and qualitative evidence may undergo separate synthesis before the resulting evidence is integrated. In its convergent segregated approach, for example, quantitative and qualitative evidence are synthesized separately and then brought together at the integration stage.

An ordinary literature review does not need to imitate a formal JBI review, but the underlying logic is useful: preserve methodological integrity before making cross-method interpretations.

In other cases, evidence can be transformed or integrated more directly

Formal mixed methods synthesis includes more than one integration strategy. JBI's guidance distinguishes, among other approaches, integration following separate syntheses from approaches involving transformation of data so that evidence can be combined in a common form. The appropriate approach depends on the review question and the nature of the evidence.

These are methodological procedures, not shortcuts for ordinary literature-review writing. If you are conducting a systematic or mixed methods review, the synthesis method should be specified prospectively where possible and reported transparently.

Methodological heterogeneity can limit synthesis

Sometimes the responsible conclusion is that certain studies should not be combined. Cochrane notes that substantial diversity in populations, interventions, outcomes, study designs, or other characteristics can make a pooled summary misleading. It also cautions that concerns about diversity do not disappear merely because a researcher switches from meta-analysis to another synthesis method.

For example, studies with very different non-randomized design features may be subject to systematically different sources of bias, and Cochrane recommends separate analysis where such design features differ substantially.

Avoid the hierarchy trap

Different methods have different strengths and limitations relative to particular questions. It is usually unhelpful to treat one method as universally superior and then use all other evidence merely as decoration.

An experiment may be better suited to estimating a causal intervention effect under specified assumptions. It may tell you little about why implementation failed in one context. An ethnographic study may provide rich evidence about practice and meaning but is not designed to estimate an average treatment effect.

Methodological appraisal should therefore ask whether the design is appropriate for the claim being made and how well the study was conducted, rather than ranking all evidence on one universal ladder.

Synthesizing an Experiment, a Survey, and Interviews

The inappropriate synthesis

“All three studies show that automated feedback improves programming learning.”

This statement collapses different outcomes and designs. Only the hypothetical experiment directly estimates an effect on the specified test outcome. The survey concerns associations with self-reported confidence and frustration. The interviews concern students' experiences and learning behaviors.

A method-sensitive synthesis

“The hypothetical evidence suggests that automated feedback can support programming learning, although its contribution appears more complex than improved performance alone. Experimental evidence indicates a modest improvement in test scores, while survey and interview findings suggest that students' experiences depend partly on the clarity of feedback and how they respond to suggested corrections. The qualitative evidence also raises a possible limitation: immediate correction may facilitate task completion without necessarily ensuring conceptual understanding.”

The synthesis gains depth precisely because the methodological differences remain visible.

Common Mistakes When Synthesizing Different Methods

Compare Contributions Before You Compare Conclusions

When studies use different methods, begin by asking what each study contributes to the question. Record the design, population, context, construct, outcome or phenomenon, and type of inference each study supports.

Only then decide whether findings should be directly compared, discussed as complementary, synthesized separately before integration, or kept distinct because the comparison would be misleading.

For formal reviews, the methodological choice should be explicit. JBI's manual provides separate guidance for different forms of evidence synthesis, including qualitative, effectiveness, textual, mixed methods, umbrella, and scoping reviews, reflecting the fact that different evidence questions require different synthesis procedures.

Before Combining Studies That Use Different Methods, Check:

Frequently Asked Questions About Synthesizing Different Methods

Preserve Methodological Differences While Connecting the Evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.