Impacts of introducing and lifting nonpharmaceutical interventions on COVID-19 daily growth rate and compliance in the United States

Significance

The coronavirus pandemic has become the most recent and urgent public health issue to threaten health systems and test government responses globally. The United States is leading with the highest number of infections despite the implementation of nonpharmaceutical interventions (NPIs). These measures may vary in their effectiveness across populations as the ability to physically distance is a privilege and is highly intersected with ethnicity and socioeconomic status. Furthermore, the United States has now lifted many NPIs; however, there is no evidence on whether and to what degree lifting NPIs can diminish the health gains of previously implemented policies. This study provides timely evidence for policymakers to inform next steps to mitigate the pandemic.

Abstract

We evaluate the impacts of implementing and lifting nonpharmaceutical interventions (NPIs) in US counties on the daily growth rate of COVID-19 cases and compliance, measured through the percentage of devices staying home, and evaluate whether introducing and lifting NPIs protecting selective populations is an effective strategy. We use difference-in-differences methods, leveraging on daily county-level data and exploit the staggered introduction and lifting of policies across counties over time. We also assess heterogenous impacts due to counties’ population characteristics, namely ethnicity and household income. Results show that introducing NPIs led to a reduction in cases through the percentage of devices staying home. When counties lifted NPIs, they benefited from reduced mobility outside of the home during the lockdown, but only for a short period. In the long term, counties experienced diminished health and mobility gains accrued from previously implemented policies. Notably, we find heterogenous impacts due to population characteristics implying that measures can mitigate the disproportionate burden of COVID-19 on marginalized populations and find that selectively targeting populations may not be effective.

The coronavirus (COVID-19) pandemic has become the most recent and urgent public health issue to threaten health systems and test government responses globally. As of October 12, 2020, more than 37 million have been infected worldwide. The United States is leading with the highest number of infections at over 7 million despite the implementation of nonpharmaceutical interventions (NPIs) to reduce transmission, encourage social distancing, and alleviate the burden of the pandemic on healthcare systems (1).

In this study, we evaluate the impacts of implementing and lifting NPIs in US counties on the daily growth rate of COVID-19 cases and on compliance with imposed social distancing, measured through mobility from January 1 to June 3, 2020. We use daily county-level data and exploit the staggered introduction of NPIs across counties and over time. Importantly, we assess the heterogeneity of these effects across counties’ sociodemographic and economic characteristics.

The immediate health benefits of NPIs in mitigating transmission are hardly disputed (2, 3). However, their effectiveness may vary depending on population characteristics such as ethnicity and socioeconomic status. The pandemic is exacerbating existing health disparities with socially marginalized populations reported to have higher infection rates and poorer outcomes (4–6). These populations may also be more vulnerable because maintaining physical distancing is a privilege (5). While the effectiveness of NPIs is conditional on voluntary compliance, not all individuals can work remotely or can afford to be furloughed (5, 7). Studies show that the ability to work remotely varies by ethnicity (7, 8), occupation, and employer (8). Therefore, the ability to comply with NPIs is highly intersected with ethnicity and socioeconomic status. These factors must be considered when assessing the health gains associated with NPIs. However, there is no evidence evaluating heterogenous impacts with regards to population characteristics.

While countries are reintroducing lockdown measures in response to a steady rise in infections, NPIs were lifted after the first wave due to their high social and economic costs. The United States lifted NPIs from end of April 2020 after their initial implementation in early March 2020 but reintroduced lockdown measures again from November 2020 (9). During the initial introduction of lockdown measures in March 2020, the Trump administration recommended that local governments implement measures selectively targeting vulnerable populations including the elderly, while allowing businesses to reopen and contribute to economic recovery (10). While lifting NPIs may be associated with economic recovery, there is no evidence on whether and to what degree lifting NPIs can diminish health gains of previously implemented policies. This type of strategy relies on continued voluntary adherence to behavioral changes and protective behavior legislated during lockdown. It is likely that compliance to protective behaviors varies over time due to the perceived risk of contracting the disease, leading to either waning or increasing health-protective behaviors. These behavioral risk factors will be shaped by political discourse and whether NPIs have been lifted. Therefore, the extent to which lifting NPIs will impact health and compliance outcomes remains unclear.

We address these questions by assessing the impacts of introducing and lifting NPIs in the United States on the daily growth rate of COVID-19 cases and mobility, including differential impacts across counties’ population characteristics.

We contribute to a growing literature analyzing NPIs. Three studies have found that social distancing policies decrease COVID-19 growth rates (11–17). Only Courtemanche et al. (12) assess the impact of the different NPIs finding that shelter-in-place orders and closures of restaurant and entertainment-related businesses impacted the growth of cases, while bans on large social gatherings and school closures had no effect. No studies have assessed the impact of lifting NPIs.

We extend the literature in important ways. We provide a comprehensive analysis of the role of NPIs on COVID-19 cases and compliance as the mechanism. By evaluating both the introduction and lifting of NPIs, we are able to compare the health and mobility gains/losses between the two strategies. Importantly, unlike previous studies that capture any potential mechanisms through a reduced form approach, we explicitly show that part of the variation in health outcomes is driven by variation in compliance, i.e., reduction in mobility due to NPIs. This study also assesses the heterogenous impacts of NPIs across counties’ sociodemographic and economic characteristics, which is an important contribution to mitigate the disparities exacerbated by the pandemic. Lastly, we assess impacts of NPIs on selectively targeted population groups by allocating measures into four categories as suggested by the Trump administration—NPIs targeting vulnerable populations, businesses, the general population through “weaker” measures, and those targeting the general population via “stronger” measures.

Results

NPI Implementation and Lifting Impacts.

Figs. 1–10 present results and SI Appendix, Tables S8–S11 include results tables. We have converted the impact of NPIs on the daily COVID-19 growth rate from percentage points (ppt) to cases per 100,000 using the formula presented in SI Appendix, Table S1.

Fig. 1.

Averaged and grouped impacts of implementing NPIs on COVID-19 daily growth rate. Note: Graph plots the point estimates as the central point and vertical bands represent the 95% confidence interval.

Fig. 10.

Weekly impacts of lifting NPIs on daily COVID-19 growth rate and percentage of devices staying home. Note: Graph plots the point estimates as the central point, and vertical bands represent the 95% confidence interval.

Implementing one NPI leads to a reduction of the daily COVID-19 growth rate by 2 ppt or 176 cases per 100,000 (Fig. 1 and SI Appendix, Table S8, M1). However, lifting one NPI leads to a significant increase in the daily growth rate of 1 ppt or 354 cases per 100,000 (Fig. 6 and SI Appendix, Table S9, M11). Introducing NPIs leads to a significant impact on the percentage of devices staying home of 0.4% (Fig. 3 and SI Appendix, Table S8, M6), while lifting one NPI leads to a reduction of 0.3% in percentage of devices staying home (Fig. 8 and SI Appendix, Table S11, M16).

Fig. 6.

Averaged and grouped impacts of lifting NPIs on COVID-19 daily growth rate. Note: Graph plots the point estimates as the central point, and vertical bands represent the 95% confidence interval.

Fig. 3.

Average and grouped impacts of implementing NPIs on percentage of devices staying home. Note: Graph plots the point estimates as the central point, and vertical bands represent the 95% confidence interval.

Fig. 8.

Averaged and grouped impacts of lifting NPIs on percentage of devices staying home. Note: Graph plots the point estimates as the central point, and vertical bands represent the 95% confidence interval.

With regards to weekly impacts, introducing NPIs leads to a significant reduction in cases from week 1 up to week 12. The percentage of devices staying home increases over time and is significant from week 1 up to week 19 (Fig. 5 and SI Appendix, Table S9, M9). As NPIs are lifted, the impact on cases is delayed with a significant impact at week 4 until week 10 (Fig. 10 and SI Appendix, Table S10, M14). The impact on the percentage of devices staying home is significant at week 5 up to week 9 (Fig. 10 and SI Appendix, Table S11, M19).

Fig. 5.

Weekly impacts of implementing NPIs on daily COVID-19 growth rate and percentage of devices staying home. Note: Graph plots the point estimates as the central point, and vertical bands represent the 95% confidence interval.

Population Characteristics.

Impacts are stronger for counties with non-White populations above the country-wide median. One additional NPI implemented leads to a decrease of 2 ppt in cases or 210 cases per 100,000 (Fig. 2 and SI Appendix, Table S8, M2b) and 0.5% in percentage of devices staying home for counties with non-White populations above the median (Fig. 4 and SI Appendix, Table S9, M7b). When NPIs are lifted, those counties experience an increase of 1 ppt on cases or 486 cases per 100,000 (SI Appendix, Table S10, M12b) and 0.4% reduction in percentage of devices staying home (SI Appendix, Table S11, M17b).

Fig. 2.

Heterogenous impacts of implementing NPIs by non-White populations and household income on COVID-19 daily growth rate. Note: Graph plots the point estimates as the central point, and vertical bands represent the 95% confidence interval.

Fig. 4.

Heterogenous impacts of implementing NPIs by non-White population and household income on percentage of devices staying home. Note: Graph plots the point estimates as the central point, and vertical bands represent the 95% confidence interval.

Counties with median household income above the country-wide median experience stronger impacts at −2 ppt or −191 cases per 100,000 (Fig. 2 and SI Appendix, Table S8, M3b) and 0.6% in percentage of devices staying home (Fig. 4 and SI Appendix, Table S9, M8b) when NPIs are implemented. One NPI lifted leads to an increase of 1 ppt in cases or 377 cases per 100,000 (Fig. 7 and SI Appendix, Table S10, M13b) and a 0.4% decrease in percentage of devices staying home (Fig. 9 and SI Appendix, Table S11, M18b) in counties with median household income above the country-wide median.

Fig. 7.

Heterogenous impacts of lifting NPIs by non-White populations and household income on COVID-19 daily growth rate. Note: Graph plots the point estimates as the central point, and vertical bands represent the 95% confidence interval.

Fig. 9.

Heterogenous impacts of lifting NPIs by non-White populations and household income on percentage of devices staying home. Note: Graph plots the point estimates as the central point, and vertical bands represent the 95% confidence interval.

NPIs by Target Populations.

Implementing NPIs has an average significant impact of −2 ppt or −163 cases per 100,000 (Fig. 1 and SI Appendix, Table S8, M1). However, only “stronger” measures targeting the general population and those targeting businesses have a statistically significant impact. Stronger measures have the highest impact of −3 ppt or −357 cases per 100,000, followed by NPIs targeting businesses with an impact of −1.3 ppt or −41 cases per 100,000 (Fig. 1 and SI Appendix, Table S8, M5). Implementing NPIs targeting vulnerable populations and the general population through “weaker” measures are weakly significant (Fig. 1 and SI Appendix, Table S8, M5).

Lifting one NPI in the vulnerable population has the highest impact of 1.6 ppt or 557 cases per 100,000 while lifting stronger NPIs targeting the general population has an impact of 1 ppt or 465 cases per 100,000 (Fig. 6 and SI Appendix, Table S10, M15). NPIs targeting businesses have the smallest impact at 0.8 ppt or 475 cases per 100,000 and NPIs targeting the general population through weaker measures have no significant impact.

NPIs implemented targeting the general population through stronger measures have the largest impact on percentage of devices staying home at 0.8%, while the remaining groups have no significant impacts (Fig. 3 and SI Appendix, Table S9, M10). Only lifting “strong” NPIs targeting the general population has an impact on percentage of devices staying home of −0.6% (Fig. 8 and SI Appendix, Table S11, M20). Fig. 11 summarizes impacts in a heat map showing the possible combinations of groups existing in counties during our study period.

Fig. 11.

Heat map of NPIs selectively targeting groups (A) implemented (B) lifted, on COVID-19 transmission and compliance. Notes: The heat map shows the relative strengths of groups of NPIs where lighter colors refer to smaller impacts and darker colors refer to larger impacts. Estimates for bundles of NPIs, combining two groups, are estimated by adding the point estimates of each respective NPI.

Robustness Checks.

SI Appendix presents results on a falsification test to assess parallel trends where the policy is switched on before its actual introduction. Results show that the parallel trends assumption is fulfilled and the coefficients on the pre-policy variables are not statistically different from zero (with the exception of 2 wk before NPI introduction for the COVID-19 daily growth rate in cases; SI Appendix, Fig. S3).

Results of alternate specifications of a 7-d lagged effect for COVID-19 daily growth rate (SI Appendix, Figs. S5–S10) are largely similar to the main results with slightly lower (higher) magnitude of the coefficients for implementing (lifting) NPIs. While in the main results, the impact of implementing NPIs is significant from week 1 onward, in the lagged model it is significant from week 3 onward. This is in line with Goodman-Bacon and Marcus (18) who suggest a delay in the impact of the policy as a check on the research design. Results using generalized propensity score reweighting (SI Appendix, Figs. S11–S20) are also qualitatively and quantitatively similar to the main results with the exception of the implementation of “weak” NPIs targeting the population which are now strongly significant and show a greater impact than NPIs targeting businesses. However, strong population measures still have the highest impact on cases. Finally, the event study design (SI Appendix, Figs. S21–S30) also shows similar impacts of NPIs as the main findings. Notably, coefficients on the pre-policy variables in the event study design are not statistically different from zero, thus providing additional confidence on the identified causal impact of the policy.

Discussion

Introducing NPIs can lead to health gains through reduced COVID-19 cases via compliance measured through percentage of devices staying home. However, while some behavioral changes in mobility seem to hold after lifting NPIs, these are short-lived. Therefore, lifting NPIs diminishes the gains accrued by previously implemented policies. We find that implementation of NPIs can help mitigate the disproportionate burden of COVID-19 on marginalized populations. NPIs are more effective in counties with non-White populations above the country-wide median, possibly due to a wider spread shutdown of businesses. Counties with higher median household income also experience a larger reduction in cases and have stronger compliance, potentially due to individuals in these counties having more flexibility to work remotely (5, 7).

We hypothesize that compliance may be a possible mechanism through which NPIs affect cases as the effect of NPIs on cases and percentage of devices staying home closely mirror one another. NPIs are effective in reducing cases but only up to 12 wk, suggesting that there may not be any additional health gains to be accrued afterward. NPIs are also effective in increasing the percentage of devices staying home throughout our study period indicating no evidence of behavioral fatigue. This may explain why there are no effects when measures are lifted until week 4 on cases and on percentage of devices staying home until after the same week. The delay in compliance, measured through percentage of devices staying home, suggests that individuals may be displaying self-protective behavior in the early stages when NPIs are lifted but only up to a certain point. Therefore, additional interventions need to be implemented simultaneously including case isolation, contact tracing, and mass testing.

Last, the strategy laid out by the Trump administration may not be effective given our findings. Targeting vulnerable populations through day care, school closures, and banning visitors to nursing homes leads to weakly significant impacts. NPIs targeting the general population through stronger measures lead to the largest reduction in cases followed by NPIs targeting businesses. However, impacts of weaker measures for the general population are weakly significant. Previous studies have not found any impacts of banning large events (12). However, we find that adjusting the threshold from 500 to 100 individuals leads to a significant decrease in cases. When measures are lifted, NPIs targeting vulnerable populations experience the largest increase in cases followed by stronger NPIs targeting the general population and businesses. This suggests that while implementing NPIs may be ineffective at protecting vulnerable populations, measures that allow for mobility in the general population can affect other population groups. Therefore, it is important to consider the target population size when enacting NPIs rather than simply focusing on selective vulnerable populations.

Our findings should be considered in the light of the limitations of the analyses. Given the bundled nature of how NPIs were implemented, our data does not allow assessing the impact of each NPI separately. Instead, we show cumulative impacts and group NPIs dependent on their target populations. By doing so, our study aligns with the current narrative. Counties often implemented one or more NPIs and are likely to do so in the event of further pandemic waves, similar to the first. Due to data limitations, we control for testing at the state level rather than at the county level. Last, counties are only treated if there was an explicit mandate. However, some businesses may have shut or reopened without a county-wide mandate leading to an underestimation of estimated effects.

Despite these limitations, our study provides timely evidence into the specific strategies that local governments can implement to protect their populations at a time when there is worrying sentiment around the US government’s ability to effectively control COVID-19.

Materials and Methods

Data.

We evaluate eight NPIs implemented from January 1 to June 3, 2020: 1) shelter-in-place, 2) restaurants closures for dine-in, 3) closures of nonessential businesses, 4) gathering size limitations including religious gatherings, 5) day care closures, 6) banning visitors to nursing homes, 7) K–12 school and university closures, and 8) a physical distancing measure to keep 6 ft apart. Introduction and end dates of NPIs were validated using several sources (19–23). Movie S1 shows the number of policies implemented and lifted in counties.

We measure the impact of implementing and lifting NPIs on the daily growth rate of COVID-19 cases at county level. Cases are from John Hopkins University and USAFacts (1, 24). We control for the daily growth rate in testing at state level from the COVID-19 Tracking Project (25) to control for bias introduced by differences in testing rates across states and over time. Additionally, we use county-level characteristics to assess differential impacts (26, 27).

To evaluate the mechanism through which NPIs have an impact through compliance, we assess mobility measured as the percentage of mobile devices staying home out of total mobile devices aggregated from 40 million devices on the county level (28). This measure is as a proxy measure for the percentage of individuals staying home, therefore, allowing us to observe the mobility of individuals through their devices’ locations coming from GPS pings from both cellphones and smartphones. SI Appendix includes additional information on NPIs, data, and sources.

Methods.

We exploit the spatial and temporal variation in the introduction and lifting of NPIs across counties using a staggered difference-in-differences (DID) approach. For implementation of NPIs, we compare counties with NPIs in place (treated) with counties that do not have NPIs in place (control) before and after implementation. We set our panel at county and day level including counties in the sample from their fifth case. Therefore, we impose an exclusion criterion by observing counties from their fifth case, and exclude 179 counties that at the time of data collection had no or very few cases. These counties may be systematically different from counties that have cases due to the widespread transmission of COVID-19. Using days from the fifth case rather than calendar time, allows the time unit of the panel to standardize counties with respect to the epidemiological curve or the stage that counties are in with regards to the progression of the outbreak. It also allows controlling for panel time fixed effects that are a flexible way of capturing the natural shape of the spread of infection. This is a methodological difference with regards to previous studies allowing to better identify the impact of NPIs (12, 13, 15, 17). Our sample size is 192,379 observations including 2,963 counties over 19 wk.

Our two outcomes of interest are the daily growth rate of COVID-19 cases and the percentage of devices staying home. Our variable of interest, the DID variable, is defined as the cumulative sum of NPIs introduced in county i in state j at time t, and captures the intensity of exposure to the treatment. We control for time fixed effects (counting from the fifth case) and county fixed effects to control for any unobserved time-invariant and county-specific confounders. We additionally include calendar week fixed effects to control for seasonality for our compliance outcome. SEs are clustered at state level (29).

To assess impacts of lifting NPIs, we set our panel by introducing counties in the sample from the first day when they introduced any of the eight NPIs and from their fifth case. We thus compare counties that have lifted NPIs (treated) with counties that still have NPIs in place (control). The sample has 191,120 observations, 2,963 counties over 13 wk. The variable of interest, the DID variable, is defined as the cumulative sum of NPIs lifted in county i in state j at time t.

We estimate 10 models for NPIs’ introduction, M1–M5, on the COVID-19 daily growth rate and M6–M10 on percentage of devices staying home. For NPIs’ lifting, we estimate M11–M15 on the COVID-19 daily growth rate and M16–M20 on percentage of devices staying home. M1 and M6 assess the additive impact of one additional NPI implemented and M11 and M16 estimate the impact of one additional NPI lifted. We evaluate differential impacts due to population characteristics by stratifying the analyses giving rise to M2–M3 for NPI implementation on COVID-19 growth rate and M7–M8 for percentage of devices staying home, while for NPIs lifting, we have M12–M13 for the COVID-19 growth rate and M17–M18 for percentage of devices staying home.

M2 and M7 evaluate the impact of introducing NPIs considering the percentage of non-White populations in counties as below the country-wide median for M2a and M7a, and above the median for M2b and M7b. Similarly, M12 and M17 estimate the impact of lifting NPIs considering the percentage of non-White populations in counties. M3 and M8 stratify counties by household income for introducing NPIs: below the country-wide median for M3a and M8a and above the median for M4b and M10b. Similarly, M12 and M18 estimate the impact of lifting NPIs by median household income.

M4 and M9 are analogous to M1 and M6; however, the DID impact is split in weeks allowing to assess the dynamics of the effects for introducing NPIs, whereas M14 and M19 assess weekly impacts for lifting NPIs.

M5 and M10 assess introducing NPIs and M15 and M20 estimate lifting NPIs by the population groups they target: 1) vulnerable populations including the elderly, children, and their families (banning nursing homes, day care and school closures); 2) businesses (closures of nonessential businesses and restaurants); 3) a weak general population category (physical distancing measure to keep 6 ft apart, gathering size limitations allowing gatherings up to 500 and allowing religious gatherings); and 4) a strong general population category (shelter-in-place orders, gathering size limitations banning gatherings above 100 individuals, and cancellations of religious gatherings). We include all four DID variables as counties typically implemented more than one NPI in each period. SI Appendix presents additional information on methods and descriptive statistics.

Robustness checks.

We conduct several robustness checks to assess sensitivity of the results with methods presented in SI Appendix. First, following Angrist and Pischke (30), we run a falsification test by introducing the policy before its actual onset to test for parallel trends, the main assumption behind DID. Second, to account for the delay between infection, symptom onset, and a reported positive test, we run alternate specifications by lagging our outcome variable, COVID-19 daily growth rate by 7 d to assess the lagged impact of the policies on COVID-19 daily growth rate. A 7-d lag is reasonable considering an average incubation period of 5 d and a lag of at least 1 d between testing and results. Third, we conduct generalized propensity score reweighting (31) as an additional robustness check to further ensure balance between control and treatment groups. We match on characteristics that can affect transmission as suggested by Goodman-Bacon and Marcus (18). Last, we use an event study design similar to that in Courtemanche et al. (12), where we interact the treatment with multiple indicators of time before and after treatment. This strategy allows us to assess pretreatment trends, which if nonsignificant, increases our confidence on the identified causal impact of the policy (32, 33). All analyses were done in Stata 16.0.

Data Availability

Some study data are available. Data on COVID-19 cases are publicly available through John Hopkins University (https://coronavirus.jhu.edu/us-map) and USAFacts (https://usafacts.org/visualizations/coronavirus-covid-19-spread-map/). Data on COVID-19 tests are publicly available through the COVID-19 Tracking Project (https://covidtracking.com/data). However, data on mobility was acquired through SafeGraph (https://docs.safegraph.com/docs/social-distancing-metrics) and therefore is not publicly available.