03 · What You Need to Know
Why Nonresponse Is More Complicated Than the Response Rate
What Is Nonresponse?
Nonresponse occurs when information is not obtained from a sampled unit or for a requested item.
The U.S. Census Bureau distinguishes forms such as unit nonresponse, where a sampled person, household, establishment, or other unit does not provide the required survey information, and item nonresponse, where a participating respondent does not answer a particular question or provide a particular requested value.
Unit nonresponse
A selected sampling unit does not provide the required survey response.
Item nonresponse
A respondent participates but does not provide information for a particular question or variable.
The two can have different causes and may require different remedies. A person who never answers the survey invitation presents a different methodological problem from someone who completes the survey but skips a sensitive income question.
What Makes Nonresponse Become Nonresponse Bias?
The Census Bureau defines nonresponse bias as deviation of the expected value of an estimate from the population parameter because respondents and nonrespondents differ.
Two ingredients therefore matter: the extent of nonresponse and the difference between respondents and nonrespondents on the quantity relevant to the estimate.
Imagine a survey estimating generative AI use among faculty members. If AI users and non-users respond at similar rates, substantial nonresponse could still produce relatively little bias in that particular prevalence estimate. If frequent AI users are much more likely to respond because the topic interests them, the resulting estimate could overstate adoption.
The bias is therefore connected to both participation and the outcome.
Why Does a Low Response Rate Not Automatically Mean High Bias?
Response rate measures the proportion of eligible sampled units that respond under a specified calculation. It does not directly measure how different respondents are from nonrespondents.
Pew Research Center's methodological research has demonstrated this distinction clearly. Low response rates can increase the potential for nonresponse bias, but the response rate alone does not establish whether meaningful bias is present.
This can seem counterintuitive. If 80% of a sample does not respond, surely the study must be badly biased? Perhaps, but not necessarily. If the 20% who respond are similar to the full selected sample on the characteristics and outcomes relevant to the estimates after appropriate adjustment, bias may be limited.
The reverse is also possible. A relatively high response rate can still produce consequential bias if the smaller group of nonrespondents differs sharply on the outcome of interest.
Watch Out
Do not use response rate as though it were a percentage measure of validity. An 80% response rate does not mean the estimates are “80% accurate,” and a 40% response rate does not mean they are “60% biased.”
Response Rate and Nonresponse Bias Are Related, but Not Interchangeable
A lower response rate means more of the originally selected sample is unobserved. That creates greater opportunity for respondents and nonrespondents to differ and can make the results more dependent on weighting or modeling assumptions.
For this reason, response rates remain important quality indicators. The Census Bureau's statistical quality standards require response-rate calculation and, for its programs, require nonresponse-bias analyses when specified unit, item, or total-quantity response thresholds are not met.
Those thresholds are agency quality-control requirements, not universal cutoffs proving that every external study above a particular response rate is unbiased or every study below it is invalid.
Nonresponse Bias Is Specific to the Estimate
Respondents and nonrespondents can differ on one characteristic while being similar on another.
Suppose younger instructors are less likely to respond than older instructors. That age imbalance may strongly affect estimates of outcomes closely associated with age, while having much less effect on an outcome that does not vary meaningfully by age.
Nonresponse bias should therefore be considered for important estimates rather than treated solely as one global property of the dataset.
The Census Bureau's definition reflects this point by describing nonresponse bias relative to the population parameter being estimated.
Why Do People Not Respond?
Nonresponse can arise for many reasons.
Some sampled units cannot be contacted because addresses, telephone numbers, or email details are incorrect. Others never see the invitation. Some are temporarily unavailable. Some refuse. Language, accessibility, technology, survey burden, privacy concerns, topic sensitivity, distrust, lack of interest, and competing demands can all affect participation.
The reason matters because it may reveal how nonresponse relates to the research outcome.
For example, people with very heavy workloads may be less likely to complete a workplace-stress survey precisely because they are too busy. Their absence could lead the responding sample to understate workload or stress.
Survey Design Can Influence Nonresponse
Researchers are not passive observers of response rates. The design of the data collection can make participation easier or harder.
Survey length, invitation wording, reminders, contact mode, accessibility, timing, incentives, perceived legitimacy, question burden, and respondent experience can all influence participation.
Item nonresponse can also vary according to question design. Pew Research Center has found, for example, that open-ended questions tend to generate more item nonresponse than closed-ended questions, with burden and requested response length among relevant considerations.
Reducing unnecessary respondent burden is therefore not merely a matter of politeness. It can be part of data-quality design.
What Is Nonresponse Bias Analysis?
A nonresponse bias analysis evaluates whether and how respondents differ from nonrespondents in ways that may affect survey estimates.
The ideal comparison would involve the study outcomes for both respondents and nonrespondents, but if those outcomes were known for nonrespondents, the nonresponse problem would be much easier. Researchers therefore often rely on auxiliary information.
Possible approaches include comparing respondents and nonrespondents on variables available from the sampling frame, comparing the achieved sample with reliable population benchmarks, examining response rates across meaningful subgroups, studying early and late respondents when methodologically defensible, conducting follow-up studies of nonrespondents, or evaluating estimates before and after weighting adjustments.
No single diagnostic proves the absence of bias. Each approach provides evidence about particular dimensions of the nonresponse process.
Comparing Respondents With Population Benchmarks Can Reveal Imbalance
Suppose an official employee database shows that 45% of the target population consists of early-career employees, but only 20% of survey respondents belong to that group.
That discrepancy is evidence of differential representation. It does not by itself tell you exactly how much the outcome estimates are biased, because that depends on whether early-career employees differ on those outcomes.
Still, such comparisons can identify where adjustment or further investigation is needed.
Can Weighting Reduce Nonresponse Bias?
Often it can reduce some of it.
Survey weights may be adjusted so that respondents better reflect known population distributions on variables such as age, education, geography, sex, occupation, or other relevant characteristics. Pew Research Center, for example, uses weighting to align survey samples with population benchmarks on multiple dimensions.
The Census Bureau similarly identifies post-sampling adjustments such as raking and post-stratification as methods that can improve the accuracy of estimates and reduce effects of nonresponse and coverage error.
But weighting depends on the information available. If respondents and nonrespondents differ on an important unmeasured characteristic not adequately captured by the adjustment variables, residual bias may remain.
Weighting is therefore a mitigation strategy, not a guarantee.
Follow-Up Can Be More Valuable Than Simply Sending the Same Reminder Repeatedly
Repeated contact can increase participation, particularly when initial nonresponse reflects missed invitations or temporary unavailability.
More targeted follow-up may also help reach groups that are otherwise underrepresented. Researchers can consider different contact times, modes, languages, accessible formats, or appropriately designed incentives where ethically and practically suitable.
Some survey programs conduct specific nonresponse follow-up or subsampling procedures to obtain information from units that did not initially respond.
The objective is not merely to maximize the response-rate statistic. It is to improve coverage of the selected sample and reduce consequential differences between respondents and nonrespondents.
Panel and Longitudinal Studies Face Attrition Too
Nonresponse does not occur only at initial recruitment.
In longitudinal research, participants may stop responding over successive waves. This attrition can become problematic when dropout is systematically associated with study variables.
A study may begin with a strong sample and gradually become more selective as particular types of participants leave.
The Census Bureau's quality standards therefore require cumulative response-rate measures for panel and longitudinal surveys in its statistical programs, reflecting the fact that nonresponse accumulates across waves.
Item Nonresponse Can Bias Particular Analyses
A participant may complete most of a questionnaire but omit income, mental-health questions, academic grades, or another sensitive or difficult variable.
If missingness is systematically related to the value that would have been reported or to variables associated with it, analyses based only on complete responses can be distorted.
The appropriate treatment depends on the missing-data mechanism, study design, amount of missingness, available auxiliary information, and planned analysis. Simply deleting every record with any missing item may be inefficient or biased under some conditions.
Nonresponse therefore extends beyond the question of how many people completed the survey.
Do Not Confuse Nonresponse Bias With Every Other Sampling Problem
Nonresponse occurs after a unit has some route into the study but fails to provide the requested data. Undercoverage occurs when relevant units are absent from the frame or recruitment mechanism. Convenience and self-selection can affect who receives or encounters an invitation in the first place.
These processes can coexist, but distinguishing them helps identify appropriate remedies.
A study can have excellent response among people on an incomplete sampling frame and still poorly cover the target population. Conversely, a nearly complete frame can produce an achieved sample distorted by differential nonresponse.
Both are part of the broader problem of how the sampling process can distort findings, but they enter at different stages.
