03 · What You Need to Know
The Methods Tell You What Kind of Evidence You Are Actually Looking At
Methods connect the research question to the results
A results section tells you what was observed or estimated. It does not, by itself, tell you how those observations came into existence.
The methods provide that connection. Depending on the study, they describe the design, setting, participants or other data sources, sampling and eligibility criteria, variables or constructs, measurement instruments, intervention or procedures, data collection, analytical strategy, and ethical procedures. Methodological reporting is intended both to help readers understand and interpret the results and, where applicable, to provide enough detail for others to reproduce or replicate the work.
This is why methods matter for more than procedural curiosity. They help determine what the evidence can reasonably support.
A cross-sectional survey, randomized experiment, qualitative interview study, longitudinal cohort, laboratory experiment, secondary-data analysis, and systematic review can all investigate related questions. They do not necessarily produce the same kind of evidence or justify the same kinds of inference.
You do not need the same methodological depth for every paper
During an initial search, reading every methodological detail of every potentially relevant paper would be inefficient. You may only need enough information to establish what kind of study you are looking at and whether it fits your purpose.
As the paper becomes more consequential, the threshold changes.
| What you are doing |
Methods attention |
What you may need to establish |
| Screening for relevance |
Quick check |
Study design, population or data source, key variables or intervention, and broad analytical approach |
| Learning the general literature |
Focused reading |
How the study was conducted and major methodological limitations affecting interpretation |
| Citing an important finding |
Closer reading |
Whether the design, measurement, sample, and analysis support the particular claim you intend to make |
| Comparing studies |
Closer reading |
Whether apparently similar findings were generated using sufficiently comparable designs, populations, measures, and analyses |
| Adapting a method or instrument |
Detailed reading |
Procedures, implementation details, measurement, assumptions, analytical decisions, and relevant supplementary material |
| Replicating or critically reviewing the study |
Comprehensive scrutiny |
Whether the methods are sufficiently appropriate and transparent to evaluate or reproduce the work |
This proportional approach follows the broader principle that the amount of a paper you need to understand depends on how you intend to use it.
Read closely when you need to know what the study design allows you to conclude
The study design is one of the first methodological features worth identifying because it shapes the kinds of conclusions the evidence can support.
Suppose a paper reports that students who use a particular learning technology achieve higher grades. That finding could come from a randomized experiment, an observational comparison, a cross-sectional survey, or an analysis of existing institutional data. Those designs may all produce useful evidence, but they raise different questions about selection, confounding, temporality, and causal interpretation.
If your own sentence says that the technology caused students to perform better, knowing the design is no longer optional.
Watch Out
Do not infer the strength or type of evidence from the confidence of the paper's wording alone. Identify the study design and consider what that design can reasonably establish before converting a reported association, difference, or pattern into a stronger claim.
Read closely when the sample affects whether the finding matters to you
Who or what contributed the data?
That question can substantially change how you interpret a result. Methods sections commonly report the study setting, recruitment or sampling procedures, eligibility criteria, sample characteristics expected by the design, and other information needed to understand where the data came from.
If a study of an educational intervention involved volunteers from one specialized program, for example, you should not automatically assume that the same findings apply to every university student. If a survey used a convenience sample, that feature may matter when considering how well the respondents represent a broader population.
The relevant questions depend on the design. You might ask:
- How were participants, cases, documents, observations, or other units selected?
- Who was eligible and who was excluded?
- Where and when was the study conducted?
- Does the sample correspond to the population about which I want to make a claim?
- Could the selection process have produced systematic differences relevant to the result?
These questions do not have one universally correct answer. A narrowly defined sample may be entirely appropriate for a narrowly defined research question. The problem arises when conclusions travel farther than the evidence warrants.
Read closely when a variable may not mean what you think it means
Two papers can use the same label while measuring substantially different things.
Consider "student engagement." One study might operationalize it as attendance, another as self-reported behavioral engagement, another as interaction logs from a learning management system, and another as a multidimensional scale incorporating behavioral, emotional, and cognitive dimensions.
The methods tell you what the construct actually became in the study.
This matters whenever your interpretation depends on measurement. Ask what instrument, procedure, coding scheme, operational definition, or data source was used. Where relevant, investigate whether the measure has evidence supporting its reliability or validity for the context in which it was used.
A similar issue arises with outcomes. A study described as examining "learning" may measure test performance immediately after an intervention, course grades, self-perceived learning, retention weeks later, or something else entirely. Those outcomes should not be treated as interchangeable merely because they sit beneath the same broad concept.
Read closely when you want to understand what participants actually experienced
Intervention studies often sound simpler in the abstract than they are in practice.
A paper might state that one group received "AI-assisted feedback," "blended learning," "gamification," "simulation-based instruction," or another intervention. Those labels do not tell you exactly what happened.
If the intervention matters to your interpretation or your own research, inspect the procedural details. What did participants actually receive? For how long? How frequently? Under what conditions? What did the comparison group receive? Were instructors or facilitators involved? Was implementation standardized or allowed to vary?
This becomes particularly important when you are trying to explain why studies that appear to investigate the same intervention reach different conclusions. Their labels may match while their actual implementations differ considerably.
Read closely when you are comparing findings across papers
Two studies can report apparently conflicting findings without truly testing the same thing.
One might examine first-year undergraduates while another studies postgraduate students. One might use a validated performance assessment while another uses self-reported perceptions. One intervention might last an entire semester while another lasts one session. One analysis may adjust for important baseline differences while another does not.
Without examining those methodological differences, you may conclude that the literature is inconsistent when the studies are answering somewhat different questions.
Same topic
Two papers use similar terminology or investigate a broadly related phenomenon.
Comparable evidence
The relevant designs, populations, operational definitions, procedures, outcomes, and analyses are similar enough for the comparison you want to make.
Methodological comparison is therefore part of interpreting disagreement in the literature, not merely a technical exercise performed after the substantive comparison.
Read closely when the analysis determines the meaning of the result
Methods sections commonly describe how the collected data were analyzed. In quantitative research, this may include statistical tests, models, covariates, treatment of missing data, sample-size or power considerations, significance thresholds, software, and other analytical decisions. Qualitative and mixed-methods studies require their own forms of analytical description.
You do not necessarily need to know how to perform every analysis yourself. You do need enough understanding to interpret the result if that result matters to your conclusion.
Ask what question the analysis was intended to answer. What variables entered the analysis? What comparisons were made? Were adjustments made, and if so, for what? Does the analysis correspond to the research question? What does the reported output actually represent?
If the statistical component is the barrier, the goal is not to acquire an entire statistics degree before continuing. Focus first on understanding the statistical results that matter to the claim and then investigate methodological details that materially affect their interpretation.
Read closely when a result surprises you
A surprising finding is a good reason to return to the methods.
Perhaps an intervention produced an unexpectedly large effect. Maybe a relationship contradicts most of the literature you have encountered. Perhaps two apparently similar studies reach opposite conclusions.
Before constructing an elaborate theoretical explanation, examine whether methodological differences offer a simpler account.
Check the population, measurement, comparison condition, study duration, analytical strategy, missing-data treatment, and other design features relevant to the finding. The difference may be substantive, methodological, or some combination of both.
This does not mean explaining away every inconvenient result through methodology. It means understanding what produced the evidence before deciding what the discrepancy means.
Read closely when you intend to adapt or replicate the research
If you want to reproduce a procedure, use an instrument, implement an intervention, or adapt an analytical approach, the methods section becomes one of the most important parts of the paper.
Published methodological guidance emphasizes that methods should contain sufficient detail to allow appropriately trained researchers to understand how the study was conducted and, where applicable, reproduce or replicate the work. In practice, however, some information may be condensed, located in supplementary files, or supplied through citations to earlier protocols.
You may therefore need to go beyond the main methods section. Look for supplementary materials, appendices, study protocols, preregistrations where available, instrument documentation, cited methodological papers, or other sources necessary to reconstruct what was done.
If essential procedural information is absent, do not quietly invent the missing step.
Read closely when you are formally evaluating the study
Critical appraisal, peer review, dissertation examination, evidence synthesis, and other formal evaluations require more than identifying the paper's headline finding.
The methods are central because validity cannot be evaluated independently of how the research was designed and conducted. Critical-appraisal literature consequently treats methodological information as essential for judging whether the design, sample, procedures, and analyses are appropriate for the research question.
The exact questions should be matched to the study design. Criteria appropriate for a randomized controlled trial are not identical to those used for a qualitative study, diagnostic accuracy study, cohort study, or systematic review.
Where formal appraisal is required, use an appropriate reporting or critical-appraisal framework rather than an improvised universal checklist. Reporting guidelines such as CONSORT, STROBE, PRISMA, and related EQUATOR Network resources are tailored to particular study types and can also help readers identify the methodological information that should be reported.
Do not mistake reporting quality for methodological quality
A detailed methods section makes evaluation easier, but detail alone does not establish that the method was appropriate. Conversely, incomplete reporting can prevent you from determining whether a potentially sound method was actually implemented well.
Methodological quality
Whether the design and procedures were appropriate for producing credible evidence for the research question.
Reporting quality
Whether the article gives you enough information to understand and evaluate what was done.
These problems can overlap, but they should not be collapsed into one judgment. When an important methodological detail is missing, the defensible conclusion may simply be that you cannot determine what happened from the report.
Missing information is itself something to notice
Readers sometimes respond to an incomplete methods section by supplying the missing information mentally: surely the researchers randomized participants properly; presumably the instrument was validated; they probably controlled for that variable.
Do not do this.
Critical-appraisal guidance specifically warns against assuming that an unreported procedure occurred. If an important aspect of the methodology is not described in the paper or accompanying materials, distinguish between what you know and what you cannot determine.
This principle is especially important when the missing information affects the credibility of a finding you intend to use.
You may need to move repeatedly between methods and results
Close methodological reading is rarely isolated from the rest of the paper.
You may encounter a result and return to the methods to determine how the outcome was measured. You may read that a particular model was used and return to the results to see which estimate it produced. A subgroup analysis may send you back to check whether the subgroup was prespecified or how it was defined.
Result A statistically significant difference is reported between two groups.
Methods How were those groups created, and what outcome was measured?
Result How large was the difference, and how uncertain is the estimate?
Methods What analysis generated that estimate, and what factors were included?
Interpretation Given the design and analysis, what conclusion is actually justified?
This kind of movement is one reason there is no universally correct section order for reading a paper. Methods become important at the point where your questions require them.
