03 · What You Need to Know
How Study Design and Statistical Analysis Should Influence Each Other
Start With What You Want to Know
Before thinking about a statistical test, identify the substantive question. Are you trying to describe a population, compare groups, estimate change, examine an association, predict an outcome, or estimate an intervention effect?
These goals place different demands on the evidence. A question about prevalence requires a different design from a question about change over time. A causal question generally requires stronger design considerations than a descriptive question. A question about individual trajectories requires repeated observations that a one-time cross-sectional measurement cannot provide.
The first decision is therefore not “Which statistical test should I use?” It is “What evidence would actually answer this question?”
The Design Determines How the Data Come Into Existence
Study design concerns much more than the label attached to a methods section. It determines who or what is observed, how units are selected, whether an intervention is assigned, whether a comparison group exists, when measurements occur, whether observations are repeated or clustered, and what alternative explanations the study can address.
Those features constrain what can legitimately be inferred from the resulting data.
A regression model fitted to cross-sectional observational data does not retroactively create randomization. A paired test cannot create a missing baseline measurement. A multilevel model can account for certain forms of clustering, but it cannot repair every weakness in how clusters were sampled, assigned, or measured.
This is why the analysis should match the research question and design rather than being treated as an independent technical choice.
A Statistical Test Is Downstream of the Analytical Question
A test usually evaluates a specified statistical quantity under a set of assumptions. Before choosing it, you need to know what quantity matters.
For example, “Do the groups differ?” is still incomplete. Do you mean a difference in post-intervention means, change from baseline, proportions reaching a threshold, event rates, trajectories over time, or something else? Are the groups independent, matched, clustered, or repeatedly observed?
Only after those features are clear does a particular statistical procedure become meaningful.
Scientific question
What do you want to learn about the phenomenon?
Study design
How will the study generate evidence capable of addressing that question?
Statistical analysis
How will the resulting evidence be summarized or modeled to estimate, compare, predict, or test what the question requires?
Analysis Planning Can and Should Send You Back to the Design
Saying that design comes before the statistical test does not mean statistics should enter only after the design is finalized.
Imagine planning an intervention study in which students are assigned to treatment by classroom. Thinking through the analysis reveals that outcomes from students in the same classroom may be correlated. That realization affects not only the eventual statistical model but potentially the number of classrooms required, allocation procedures, sample-size calculations, and what information must be collected about clusters.
Likewise, planning an analysis of change may reveal the need for a baseline measurement. A longitudinal model may require more measurement occasions than originally proposed. A planned subgroup analysis may expose inadequate representation of the subgroup.
Statistical planning therefore acts as a design diagnostic. This is one of the main reasons you should think about data analysis before collecting data.
Sample Size Is an Obvious Point of Interaction
Sample-size planning often depends on both the study design and intended analysis. Relevant inputs may include the primary outcome, target effect or precision, variability, allocation ratio, clustering, repeated measurements, expected attrition, significance level or interval-based criterion, and statistical model.
This creates a feedback loop. You cannot sensibly determine sample size without some idea of the primary analysis, but you should not choose the scientific design merely because a familiar test produces a convenient sample-size calculation.
If the planned study is infeasible, the appropriate response may involve reconsidering the research question, design, outcome, precision target, recruitment strategy, or analytical approach. The answer is not necessarily to substitute whichever test requires fewer participants.
Randomization and Blocking Illustrate Why Design Can Improve Analysis
In experiments, design features such as randomization, blocking, stratification, matching, or repeated measurement can affect statistical efficiency and the credibility of the resulting comparisons.
NIST describes statistically sound experimental design as part of effective data collection, including decisions about repetitions, sequencing, and randomization. ICH statistical guidance likewise treats statistical considerations as integral to clinical trial design and analysis rather than as an afterthought.
The broader principle applies well beyond those settings: good design can simplify analysis and strengthen what the resulting estimates mean. Statistical complexity is not necessarily a badge of methodological sophistication. Sometimes the best analysis is straightforward precisely because the design did much of the hard work.
Do Not Choose a Design Merely Because You Know the Test
Familiarity is a practical consideration, but it should not determine the scientific architecture of the study.
If the question calls for repeated measurements, converting the design into two unrelated groups simply because you know an independent-samples t-test sacrifices information needed for the question. If data are naturally clustered, pretending they are independent so that a simpler procedure can be used does not make the design simpler; it makes the analysis misaligned.
When the appropriate analysis exceeds your current expertise, the solution may be methodological consultation rather than redesigning the question around the techniques you already know.
Do Not Choose the Most Sophisticated Test Either
The reverse problem also occurs. Researchers sometimes choose a complex technique because it appears more advanced and then search for a research design that justifies using it.
Methodological complexity has no intrinsic scientific value. A more elaborate model can introduce additional assumptions, require larger samples, make interpretation harder, and answer a question the study never needed to ask.
The appropriate method is the one that addresses the analytical target while respecting the design and data structure. If a simpler method does that adequately, complexity does not improve the study merely by being complexity.
Statistical Assumptions Can Reveal Design Requirements
Analytical methods rely on assumptions concerning matters such as independence, functional form, distributions, variance structures, censoring, missingness, or model specification. Some assumptions can be assessed or addressed during analysis, but others are closely tied to the design.
Independence is a good example. If students are sampled within classrooms, the dependence is not an unfortunate feature discovered by a normality test. It is part of how the data were generated. The analysis needs to respect it, and the design and sample-size planning should ideally anticipate it.
Thinking about assumptions early helps distinguish problems that can be addressed analytically from problems that require changes to data collection.
The Exact Test May Not Need to Be Fixed at the Earliest Design Stage
There is a difference between planning the analytical strategy and prematurely committing to every technical detail.
Early in design, you may know that the study requires a comparison of repeated outcomes while still considering several defensible modeling approaches. As the protocol becomes more precise, the primary method can often be specified in greater detail.
The question of whether you should choose the statistical test before collecting data therefore depends partly on the purpose of the study and how consequential the choice is. Confirmatory research generally benefits from greater advance specification than open-ended exploratory work.
Think of Design and Analysis as an Iterative Planning Loop
A better workflow moves back and forth before data collection.
1. Research question Define what you want to know.
2. Intended inference Clarify the comparison, association, effect, prediction, description, or other quantity that would answer the question.
3. Study design Determine how observations must be sampled, assigned, measured, and timed to generate that evidence.
4. Analytical strategy Identify a method appropriate to the question, design, variables, and data structure.
5. Design check Ask whether the analysis exposes missing measurements, inadequate sample size, dependencies, or other design problems.
6. Refine before collection Revise the design or analysis until the chain is coherent and feasible.
This preserves the proper hierarchy without pretending the process is strictly linear. The scientific question leads, but good statistical thinking helps determine whether the proposed design can answer it.
