Decreasing Survey Response Rates in the Time of COVID-19: Implications for Analyses of Population Health and Health Inequities

Abstract

Objectives. To examine whether, and if so how, US national and state survey response rates changed after the onset of the COVID-19 pandemic.

Methods. We compared the change in response rates between 2020 and 2019 of 6 (3 social and economic, 3 health focused) major US national surveys (2 with state response rates).

Results. All the ongoing surveys except 1 reported relative decreases (∼29%) in response rates. For example, the household response rate to the US Census American Community Survey decreased from 86.0% in 2019 to 71.2% in 2020, and the response rate of the US National Health Interview Survey decreased from 60.0% to 42.7% from the first to the second quarter of 2020. For all surveys, the greatest decreases in response rates occurred among persons with lower income and lower education.

Conclusions. Socially patterned decreases in response rates pose serious challenges and must be addressed explicitly in all studies relying on data obtained since the onset of the pandemic.

Public Health Implications. Artifactual reduction of estimates of the magnitude of health inequities attributable to differential response rates could adversely affect efforts to reduce these inequities. (Am J Public Health. 2023;113(6):667–670. https://doi.org/10.2105/AJPH.2023.307267)

Reckoning with the toll of the COVID-19 pandemic on population health requires addressing not only the direct harms caused—on both health and the societal determinants of health—but also the production of scientific knowledge about population health and health inequities.¹ In the United States, 1 issue concerns how survey response rates for surveys designed to be representative at the state or national level, as well as response rates for specific health investigations (e.g., on COVID-19 or other outcomes), have been affected by pandemic disruptions.^1–3 Also at issue are concurrent societal and institutional reckonings with structural racism precipitated by the police murder of George Floyd on May 25, 2020, combined with growing political and economic polarization, together affecting attitudes toward public health and other government agencies.^4–6

A plausible scenario is that nonnegligible differential response rates^1,2 would be induced by the racialized and economic inequities in COVID-19’s toll and the concomitant societal polarization.^1,7 Although at first not well documented, evidence now makes clear that, especially before vaccines became available and efforts to make them equitably accessible were funded and implemented, COVID-19 disproportionately infected people in low-wage jobs that required them to be physically present at work (typically with no sick leave) and, by extension, their family members.^2,7 Extant racialized economic occupational and residential segregation in turn led to disproportionately elevated COVID-19 rates in lower-income neighborhoods, especially those with higher concentrations of Black, Latinx, and American Indian and Alaska Native residents, and among persons in institutional settings, including underresourced nursing homes and prisons.^2,7

We accordingly examined whether, and if so how, response rates of major US national and state surveys changed after the onset of the COVID-19 pandemic. Three considerations motivated our inquiry: (1) the implications of any such impacts on response rates and the production of scientific knowledge about population health,^1,2 (2) the lack of any readily identifiable published articles documenting changes in response rates across numerous national surveys,³ and (3) our awareness of the challenges our team faced in implementing a community-based study designed before COVID-19 for which recruitment began in March 2020, when the pandemic was declared a national emergency.⁸

METHODS

We focused on major US population data resources designed to be representative that are widely used in US population health and other population-based research. We included 6 national surveys conducted by the US Census, the US Department of Health and Human Services and agencies under its aegis (e.g., US National Center for Health Statistics, US Centers for Disease Control and Prevention), and the US Department of Labor, of which 2 provided data on response rates by states. The 6 surveys included are listed in Table 1 and their source information is provided in Table A (available as a supplement to the online version of this article at http://www.ajph.org).

TABLE 1—

Response Rates Immediately Before and During the COVID-19 Pandemic for US National and State Social and Health Surveys Designed to Be Representative of the Population: 2019–2022

	Response Rates, %			Absolute Difference in Response Rates, Percentage Points		Relative Difference in Response Rates vs 2019
	Immediately Prepandemic	Pandemic Time 1 (2020)	Pandemic Time 2 (2021)	Time 1 vs Prepandemic	Time 2 vs Prepandemic	Time 1 vs Prepandemic	Time 2 vs Prepandemic
US Census ACS (http://bit.ly/3ZDfyMM ) a
US housing units	86.0	71.2	85.3	−14.8	−0.7	0.83	0.91
US group quartersb	90.9	47.2	74.8	−43.7	−16.1	0.52	0.82
State housing unitsc
Median	87.0	73.0	87.1	−14.0	0.10	0.83	1.00
Minimum	75.4	63.1	75.9	−12.3	0.10	0.84	1.00
Maximum	92.0	85.2	92.7	−6.8	0.70	0.93	1.00
State group quartersb,c
Median	93.0	50.4	80.2	−36.6	−12.8	0.54	0.86
Minimum	65.3	28.9	35.5	−36.4	−29.8	0.49	0.54
Maximum	98.2	70.9	95.7	−27.3	−2.5	0.72	0.97
US Current Population Survey (http://bit.ly/3FcDXAJ ) a
United States	82.3	76.2	72.0	−6.1	−10.3	0.93	0.87
US Bureau of Labor Statistics: Consumer Expenditure Surveys (http://bit.ly/3YBOPz1 ) a
Diary	49.5	33.9	NA	−15.6	NA	0.68	NA
Interview	52.2	45.8	NA	−6.4	NA	0.88	NA
US National Health Interview Survey (https://bit.ly/3J7gstZ ) a
Household	60.0	42.7	NA	−17.3	NA	0.71	NA
Adult sample	57.9	41.1	NA	−16.8	NA	0.71	NA
Child sample	57.6	40.1	NA	−17.5	NA	0.70	NA
US National Health and Nutrition Examination Survey (http://bit.ly/3Yz6LtY ) a
Screener response	85.3	NA	NA	NA	NA	NA	NA
Interview	49.6	NA	NA	NA	NA	NA	NA
Examination	44.0	NA	NA	NA	NA	NA	NA
BRFSS Survey (2019: https://bit.ly/40ijd2v; 2020: https://bit.ly/3LAxUK8; 2021: https://bit.ly/3Z2Ncur)
United States
Mean	50.0	47.8	44.6	−2.2	−5.4	0.96	0.89
Median	49.4	47.9	44.0	−1.5	−5.4	0.97	0.89
Minimum	37.3	34.5	23.5	−2.8	−13.8	0.92	0.63
Maximum	73.1	67.2	60.5	−5.9	−12.6	0.92	0.83

Note. ACS = American Community Survey; BRFSS = Behavioral Risk Factor Surveillance System; NA = not available. Relative times (immediately prepandemic, pandemic time 1, and pandemic time 2) vary by survey. See Table A (available as a supplement to the online version of this article at https://www.ajph.org) for full dates.

^a Concerns stated in the survey documentation regarding social differentials in response rates, including inability to correct for these differentials using conventional weighting approaches (see Table A for descriptions). Each of these surveys additionally reported changes in survey design in response to the COVID-19 pandemic (see Table A for descriptions).

^b Defined by the ACS report as “places where people live or stay in a group living arrangement that is owned or managed by an organization providing housing and/or services for the residents . . . such as college residence halls, residential treatment centers, skilled nursing facilities, group homes, military barracks, prisons, and worker dormitories” (http://bit.ly/3ZDfyMM).

^c For the state-level ACS response rates, states include 50 US states plus the District of Colombia and Puerto Rico.

We tallied the absolute and relative change in survey response rates, comparing data reported in the most recent prepandemic period (2019 to February 2020) to the data reported since the onset of the pandemic emergency period (after March 2020 to 2021, with 1 survey providing data for 2022). We also recorded the information provided, if any, about (1) changes in study design because of the pandemic, and (2) differential changes in response rates by social groups and discussion regarding weighting methodologies.

RESULTS

Among the 6 national surveys, 5 continued with reduced operation after March 2020, 1 halted operation entirely, and all but 1 reported notable reductions in response rates (Table 1). Comparing the 2020 with the 2019 data, the median absolute difference for the 9 national data points available was −15.6 percentage points (mean = −15.5%; range = −43.7 to −1.5), and the median relative change was 0.71 (mean = 0.77; range = 0.55–0.97); that is, a 29% lower response rate than in 2019. The survey least affected was the one that relied solely on remote interview methods before the pandemic (Behavioral Risk Factor Surveillance System; median absolute difference = −1.5%). The one with the greatest absolute decrease was the group quarters survey for the US Census American Community Survey (absolute difference = −47.2%). For the 4 surveys reporting 2021 response rates, the median relative difference was 0.91 and was smaller compared with 2019. For the 1 survey reporting data for 2022, the relative difference in response rates compared with 2019 was 0.87 (Table 1). Survey documentation consistently reported greater reductions in response rates among persons with lower income and lower educational attainment, as well as reduced coverage among Black and Hispanic populations. Three surveys (American Community Survey, Current Population Survey, and National Health Interview Survey) conducted analyses showing that standard weighting methods could not correct these problems (Table 1 and Table A).

DISCUSSION

In a context of societal disruptions owing to the COVID-19 pandemic, it is unsurprising that US national and state surveys have experienced substantial decreases in response rates, with 4 of the 5 major US surveys reporting relative reductions on the order of 29% (Table 1).^1–3 It is likewise not surprising that these reductions were socially patterned, with the greatest decreases among persons with lower income and lower education, and reduced coverage especially affecting the US Black and Hispanic populations (i.e., the social groups hardest hit by the onset of COVID-19; Table 1).^1–7

Plausible hypotheses to explain these trends include (1) inabilities of survey staff to connect remotely with and enroll participants from whom data were previously obtained by in-person interviews, including persons in households without telephones and persons residing in group quarters (Table 1); and (2) increased respondent burden and distrust linked to the heightened weariness and wariness among persons whose lives and livelihoods were adversely affected by the pandemic^1,2,4,7,9 combined with heightened polarization about government agencies and their work.^5,6 Preliminary data suggest similar problems have affected project-specific health investigations that enrolled participants during the pandemic.^2,9

Socially patterned differential decreases in survey response rates threaten capacity for accurate investigation of trends in and analysis of population health, health inequities, and societal determinants of health.^1–3,9–11 Of concern is the potential impact on the range of values observed, the associations between the variables that predict selection and other variables of interest, and the population data need for reweighting,¹¹ with 3 surveys documenting the inability of conventional weighting methods to correct these problems (Table 1 and Table A).

One additional concern warranting investigation is how differentially decreasing response rates could lead to artifactual reduction of estimates of the magnitude of health inequities (e.g., if persons most harmed by societal injustice disproportionately are absent from the data or, if enrolled, have missing data not at random).^1,9,10 Such scenarios are plausible, given concerns about data quality documented in Table 1 and could undercut efforts to reduce these inequities. Rectifying the impacts of structural racism and other types of injustice on population health monitoring and the scientific production of public health and biomedical knowledge entails addressing these challenges.^1,4,7,10,12

In summary, the COVID-19 pandemic has magnified existing challenges of recruitment, enrollment, and data analysis and has presented new challenges for public health agencies and researchers.^2,3,9–11 It is incumbent on those who undertake population surveys and those who undertake health research to explicitly situate the societal context, including the pandemic and sociopolitical context, in which participants provided data, the impacts on response rates and missing data, and implications for analyzing population health data and health inequities.