The effectiveness of COVID deaths to COVID policies: A robust conditional approach

Abstract

This paper examines the effectiveness of four major COVID-19 social distancing policies, (i) shelter-in-place orders (SIPO), (ii) non-essential business closures, (iii) mandatory quarantine for travelers, and (iv) bans on large gatherings, on both COVID cases and COVID deaths. Results indicate that states are highly ineffective in producing the fraction of the population that does not have COVID-19 or the fraction of the population that does not die from COVID-19. We find that having any form of social distancing policies increases the fraction of the population not considered a positive COVID-19 case by 23.5 percentage points. Results also show that having any of the four major social distancing policies reduces the fraction of the population who has died of COVID-19 by 1.3 percentage points between March 1, 2020 and September 1, 2020; during the first 100 days, effectiveness would improve by 2.1 percentage points. Evidence suggests that there is no effective uniform national COVID-19 social distancing policy. Furthermore, conditional efficiency regressions after 100 days suggest that behavioral noncompliance and premature expiration of social distancing policies both negatively impact effectiveness. Partial regression plots suggest that bans on large gatherings and the closure of non-essential businesses were the two most impactful COVID-19 social distancing policies.

1. Introduction

Late in 2019, a new virus emerged in Wuhan, China; it would take several months before the virus would spread rapidly throughout the United States (U.S.) and the rest of the world. On January 29, 2020, the White House assembled a Coronavirus task force, whose objective was to “coordinate and oversee the administration’s efforts to monitor, prevent, contain, and mitigate the spread” of the virus (COVID, 2020). It became clear in March 2020 that COVID-19 was present in the U.S. and spreading rapidly; various social distancing policies were suggested and states began to implement policies that were found to be most suitable for their respective states.

Efforts to limit the reproduction rate of a virus are based on prosocial activities by individuals, social distancing policies implemented by governments to cover shortfalls in prosocial behaviors, and economic policies to offset the harms from these policies (Regmi and Lwin, 2021, Sachs et al., 2022, Walsh et al., 2021). This paper is motivated to analyze the efficiency of major state-level policies implemented by U.S. state governments in response to the COVID-19 pandemic: (i) shelter-in-place orders (SIPO), (ii) non-essential business closures, (iii) mandatory quarantine for travelers, and (iv) bans on large gatherings. Exiting studies have found that COVID-19 social distancing policies have been effective in reducing COVID-19 cases and lowering the death rate, also known as “flattening the curve” (Abouk and Heydari, 2020, Courtemanche et al., 2020, Dave et al., 2020, Friedson et al., 2020, Gearhart et al., 2022, Siedner et al., 2020).

As Stock (2020) notes, the most effective policy to achieve a given COVID-19 transmission rate needs evaluation in terms of economic costs and lives saved. There are, broadly, several major policy sets that policymakers could have adopted. The “suppression” strategy is to use social distancing policies to reduce the reproduction rate (R0) of the virus to below 1, to quickly curb the spread of the virus; this would be the “zero COVID” policies followed by countries around the world (Regmi and Lwin, 2021, Walsh et al., 2021). To ensure that suppression is effective, it is important to implement these strategies earlier, rather than later (Mendez-Brito et al., 2021). The “flatten the curve” strategy reduces R0 to between 2 and 4, but greater than 1; the goal is to provide time for health systems to build up and enable public health and social measures. Though cumulative cases under “flatten the curve” will be similar to those under no policies, they will be spread over a longer period of time (Anderson et al., 2020, Muehlschlegel et al., 2021, Thunstrom et al., 2020). The last major policy set is no control and letting the virus go through the population in the hope to achieve herd immunity (Sachs et al., 2022). Kompas et al. (2021) note that early mandated suppression is much more effective, in terms of economic costs, than unmitigated approaches to a pandemic, and that a “go early, go hard” approach to tackling COVID-19 was likely optimal.

The economic and financial impacts of COVID-19 policies are well-documented, and include GDP loss, income flow loss, employment losses, increased economic uncertainty, and reduced hours of work (Baker et al., 2020, Baldwin, 2020, Binder, 2020, Carlsson-Szlezak et al., 2020a, Carlsson-Szlezak et al., 2020b, Coibion et al., 2020a, Coibion et al., 2020b, Demirguc-Kunt et al., 2020, Gupta et al., 2020, Lewis et al., 2020, Mulligan, 2020, Rojas et al., 2020). There is a need to measure the “efficiency” of COVID-19 social distancing policies, especially on COVID-19 case and death rates.2 This is especially important in the U.S. for several reasons. First, the undercounting of reported versus actual deaths is near the lowest in the world, meaning that the concerns put forth in Millimet and Parmeter (2022) are not as substantial in our analysis (Sachs et al., 2022). Second, the Americas had relatively high COVID-19 mortality rates, related to a number of factors: (i) population demographics, including a relatively large fraction of elderly persons (Hoffmann and Wolf, 2021, Saxena and Hashmi, 2021); (ii) a high prevalence of comorbidities (Dorjee et al., 2020, Thakur et al., 2021); (iii) inequities in healthcare and health access (Dalsania et al., 2022, Paremoer et al., 2021); and (iv) anti-vaccine propaganda (Crawshaw et al., 2022, Regmi and Lwin, 2021). Therefore, the effectiveness of a “flatten the curve”, rather than “suppression” policy, pursued by the United States, needs to be empirically verified. Gearhart et al. (2022), for instance, note that social distancing policies were highly effective in flattening the curve and reducing COVID-19 cases, especially bans on large gatherings. They also note that the premature revocation of social distancing policies was counterproductive, leading to COVID-19 case spikes.

Our analysis will provide empirical backing to some of the optimal policy design models (Acemoglu et al., 2020, Avery et al., 2020, Brotherhood et al., 2020) so policymakers can easily apply effective methods to combat COVID-19, as the duration and consequences of inaction or poor action grow larger. Therefore, we utilize nonparametric order-m efficiency estimators, then condition the efficiency scores on the state-level policy variables, utilizing the methods by Cazals et al. (2002), Daraio and Simar, 2005, Daraio and Simar, 2007a, Daraio and Simar, 2007b, and De Witte and Kortelainen (2013). This enables us to measure the state-level effectiveness of how COVID-19 related outcomes, namely the number of individuals in a state who have not died from COVID-19 per 100,000 population, are impacted based on three factors that affect COVID-19 mortality rates: (i) the number of individuals in a state who are not considered a positive COVID-19 case per 100,000 population; (ii) the state obesity rate; and (iii) the state-level average number of days, out of 30, citizens spend in poor physical health.

We focus on death as an output measure because these are, ultimately, the final output for public policy decision makers. Similarly, Gearhart et al. (2022) have used a similar methodology to ascertain the impact of social distancing policies on COVID-19 case rates, but not COVID-19 death rates. Though COVID-19 death reports sometimes face a backlog during the COVID-19 “waves”, the duration of our analysis time span resolves these issues. Similarly, we undertake our analysis on longer time spans than other authors (notably Dave et al., 2020, Friedson et al., 2020) since, as Dr. Fauci has noted in several interviews, deaths can lag cases by potentially more than 3 weeks (Newsweek, 2020). Therefore, deaths on September 1, 2020 could have been from cases in late July or early August, 2020.3 Lastly, focusing on a single country, rather than multiple countries limits some of the concerns noted in Millimet and Parmeter (2022), who found that reported deaths often fall far short of actual deaths.4

Our results confirm and extend the literature and provide insight as to which policies may be more effective in combatting future pandemics. After establishing that having social distancing policies improves the effectiveness of COVID-19 cases, in line with Gearhart et al. (2022), we find that during COVID-19, without taking into consideration social distancing policies, the average state is 21.4 percent inefficient, where our output is the fraction of the population that did not die from COVID-19. If pre-pandemic ineffectiveness and pandemic unreadiness were wholly cured, this would have been 37,000 fewer cumulative COVID-19 deaths by September 1, 2020, holding the number of COVID-19 cases constant. Second, we find that there is tremendous heterogeneity across states, and there is limited geographical explanations for these findings. For instance, North Dakota was considered fully effective in combatting COVID-19, while West Virginia was 42.6 percent ineffective.

Third, we find that having at least one social distancing policy would improve average effectiveness by 1.3 percentage points, which would have been 2000 cumulative lives not lost to COVID-19 by September 1, 2020. Depending on the statistical value of a life, this could be $20 billion in “lost life” savings, without speaking to the value of reduced infection cases. If we focused on the first 100 days of the pandemic, this value would increase by nearly 50-percent; conditional effectiveness improved by 2.1 percentage points for any social distancing policies in the first 100 days after March 1, equating to nearly 3700 lives saved. Though this is a small fraction of the total deaths given the demographic, comorbidity, and healthcare access issues found in the U.S., any improvement would have been meaningful (Dalsania et al., 2022, Dorjee et al., 2020, Hoffmann and Wolf, 2021, Paremoer et al., 2021, Saxena and Hashmi, 2021, Thakur et al., 2021). Fourth, we find that having all four policies increased average effectiveness by only 0.5 percentage points (or 600 fewer cumulative deaths) between March 1, 2020 and September 1, 2020. The likeliest explanations were that multiple policies led to: (i) confusion on the parts of business and individuals as to what was allowed; and/or (ii) behavioral noncompliance, where additional policies led to increased resistance by business owners and consumers.

Fifth, we find that the earliest adopters of SIPOs saw the biggest effectiveness gains. California, which implemented its SIPO on March 19, 2020, saw an average effectiveness improvement of over 2 percentage points; nearly 270 fewer deaths by September 1, 2020. Sixth, we see that the policies were more effective for the first 100 days of the sample, rather than the entire sample. In fact, average conditional effectiveness decreased after 100 days of social distancing policies, possibly through three potential mechanisms: (i) many states allowed social distancing policies to expire, suggesting that much of the initial benefits of the social distancing policies dissipated over time without them in place, though it does indicate that the benefits of these policies followed well after their expiration; (ii) behavioral noncompliance was a significant factor in states with long-lasting COVID-19 social distancing policies, where individuals chose to not follow relevant state law after long periods of time under social distancing policies; or (iii) individuals believed that the economic costs began to outweigh the health costs of COVID-19, further fueling the winter surge in COVID-19. We believe that our results are suggestive of both behavioral noncompliance in states with longer-lasting social distancing policies as well as the premature expiration of a variety of effective social distancing policies, many in late April to mid-May.

Lastly, partial regression plots suggest that the two most effective social distancing policies were closures of non-essential businesses and bans on large gatherings. The reminder of the paper is organized as follows. Section 2 compiles a list of previous literature, while Section 3 presents the data and methods used in the paper. Section 4 presents the empirical results while Section 5 concludes the study.

2. Literature

There are three major types of policies used to combat epidemics like COVID-19: (i) no control, allowing unchecked spread in the hope of achieving herd immunity; (ii) “flatten the curve”, to increase the time between the spread of cases; and (iii) “suppression”, also known as zero COVID, to eliminate the pandemic quickly (Anderson et al., 2020, Mendez-Brito et al., 2021, Regmi and Lwin, 2021, Sachs et al., 2022, Thunstrom et al., 2020, Walsh et al., 2021). The goal of “flatten the curves” social distancing policies followed by the U.S. and around the world have slowed the spread of COVID-19, reducing cases in the hope that the healthcare system is able to cope with the increased demands for medical care (Australian Government Department of Health, 2020, Public Health England, 2020, Public Health Agency of Canada, 2020, White House, 2020). As mentioned in Dave et al. (2020), in the U.S., the authority to issue public health policies rests with state and local governments, with attendant penalties enforced by local governments (Dave et al., 2020, Francassa, 2020, Caswell, 2020). Sachs et al. (2022) note that subnational COVID-19 policy responses were a major weakness of COVID-19 policies.

We discuss the consequences of a pandemic before moving on to research that examines the specific impacts of COVID-19 policies on these outcomes. The importance of combatting COVID-19, with a combination of appeals to behavioral compliance, social distancing policies to reduce the spread of the disease, and economic policies to offset the harms imposed by individual and government policies, are numerous. First, even in highly vaccinated populations, economic disruptions occur from endemic diseases, including a shortage of healthcare workers if they are exposed, school closures, disruptions to businesses, and increases in economic uncertainty (IMF, 2022). Second, high levels of COVID-19 transmitted through the population (cases and deaths) led to worsened mental health, with significant rises in anxiety and depression (Aknin et al., 2022, Santomauro et al., 2021). Unfortunately, marginalized communities fared worse from COVID-19, with younger people, women, and people with children under the age of 5 showing the largest increases in psychological distress (Muehlschlegel et al., 2021, Pierce et al., 2020, Sun et al., 2023).

Third, COVID-19 disrupted the economic advancement of women. It forced many families back to traditional gender stereotypes, where females disproportionately dropped out of the labor force; especially in families with children (Garijo, 2020, Power, 2020, Su et al., 2022). In the United States, working mothers decreased work time by 21.1-percent, compared to a reduction in work time for fathers by 14.7-percent (U.S. Census Bureau, 2021). The impacts of COVID-19 were not confined to parents only, as children faced health, safety, and economic consequences related to COVID-19, in large part due to school closures (Barron et al., 2022). In the U.S. and Mexico, there was almost 13 months (continuous) of no schooling for 62 million children, with dramatic racial disparities in access to live remote teaching (Levinson et al., 2021). In fact, the World Bank (2022) has estimated that this generation of children could lose up to $10 trillion globally in lifetime earnings because of the disruption of COVID-19 on their lives. Overall, even though COVID-19 policies were detrimental to economic growth, the considerable negative impacts from death and illness from COVID-19 led to a total economic toll of $14 trillion by 2023 (Walmsley et al., 2023). We have seen that, even with unprecedented policies to mitigate the impacts of COVID-19 (health, social, and economic), considerable external impacts exist.

We now focus on the impact of specific policies on certain types of behavior, first, discussing the literature that examines optimal policies meant to address a pandemic. This literature follows the Susceptible Infected Recovered (SIR) epidemiological model of contagion. The models forecast disease scenarios based on population composition and infection rates, conditioned on policies that could be put into place. Acemoglu et al. (2020), Berger et al. (2020) and Eichenbaum et al. (2020) found that more targeted COVID-19 policies, namely the quarantine of at-risk individuals, would have lessened the economic impact of COVID-19. Acemoglu et al. (2020) found that a uniform lockdown, which would have been analogous to zero COVID suppression, of over a year would have led to economic costs of over 24 percent of GDP; a targeted lockdown would reduce both fatalities, last for less than a year, and cost only 10 percent of annual GDP. Bethune and Korinek (2020) noted that a decentralized approach, allowing subnational units to conduct their own COVID-19 policies, would not be an efficient outcome, as individuals would choose to not engage in sufficient social distancing. Beyond “optimal” policies, most research has suggested that earlier approaches to combatting COVID-19 lessened the economic costs and improved public health outcomes (Grafton et al., 2021, Kompas et al., 2021).

Economic studies that have not employed efficiency analysis have focused on the causal impact of SIPOs on social distancing (Abouk and Heydari, 2020, Courtemanche et al., 2020, Dave et al., 2020, Friedson et al., 2020, Siedner et al., 2020), with many finding that SIPOs had modest impacts on reducing social mobility. Friedson et al. (2020) found that the SIPO in California reduced both COVID-19 cases, and COVID-19 related deaths. Dave et al. (2020) found that SIPOs not only were effective (nationally) in residents staying at home, but also reducing cumulative COVID-19 cases and deaths.

A growing body of literature is investigating the effectiveness of COVID-19 responses using efficiency estimators. Millimet and Parmeter (2022) found that there was substantial underreporting of COVID-19 deaths in nearly all countries, and that the difference between reported and actual deaths needs to be addressed when assessing the impact of COVID-19 social distancing policies on outcomes. Similar to the results in Sachs et al. (2022), Breitenbach et al., 2020a, Breitenbach et al., 2020b and Ibrahim et al. (2020) found that most developed countries were unprepared for a pandemic, and among the least effective in combatting COVID-19, though there will still considerable issues worldwide in addressing the pandemic. Unfortunately, each of these studies is beset by issues. DEA estimators suffer from the well-known curse of dimensionality. Similarly, DEA (by construction) is a full frontier estimator, meaning that outliers will alter the underlying production process. Given the inclusion of a number of developing countries with developed countries, it is likely that the developing countries, which were largely considered fully (or near fully) efficient were inappropriate comparison groups for the developed countries.5 Another issue in all papers is that it is likely that some of the inputs would be better considered as environmental variables, rather than direct inputs in the production process. In failing to address the impact of these secondary environmental variables in some two-stage framework, the authors likely have efficiency estimates that are spurious, rather than causal. Gearhart et al. (2022), utilizing the same methodology proposed in this paper, found that social distancing policies, specifically bans on large gatherings, were highly effective in reducing COVID-19 cases. They also found that the premature lifting of COVID-19 social distancing policies were detrimental in combatting the epidemic.

3. Data and theory

The data for this study are obtained from multiple sources. State-level annual data on health outcomes are obtained from the UWIPHI County Health Rankings and Roadmaps database; these are aggregated from a variety of health survey sources.

Data on COVID-19, including cumulative (or daily) cases, deaths, and hospitalizations at the state level are collected by the Center for Systems Science and Engineering at John Hopkins University, as a free repository.6 Lastly, we utilize the Kaiser Family Foundation database on state social distancing action effective and rollback dates. The four main social distancing policies that we use are: (i) SIPOs; (ii) non-essential business closures; (iii) mandatory quarantine for travelers; and (iv) bans on large gatherings.7 Data are collected from March 1, 2020 to September 1, 2020. Fig. 1 provides a visualization of county-level death rates per million population; there is no discernible geographical pattern that would suggest that certain states were uniquely better positioned to tackle the pandemic, which is supported by Sachs et al. (2022), who noted that most areas were ill-prepared for a pandemic.

Fig. 1.

Deaths per million population by county.

Unconditional efficiency estimation utilizes two types of variables: (i) inputs; and (ii) outputs. Newer estimation techniques acknowledge that there are secondary variables that are not inputs, but play a role in the shaping of the production frontier. These are called environmental variables, and allow us to estimate conditional efficiency estimates. We start by replicating the analysis from Gearhart et al. (2022), that COVID-19 social distancing policies influence the effectiveness of COVID-19 cases. Therefore, our first output measure is the fraction of the population that is not considered a COVID-19 case. The inputs utilized in this process are measures of social mobility from SafeGraph, Inc. SafeGraph has anonymized population movement data from cellphones of nearly 45 million devices. We use two state-by-day measures of mobility: (i) the percent of the state-day population that remains at home for the entire day; and (ii) the average distance, in meters, the average person moves outside the home in a state-day, standardized so that bigger values are associated with traveling less outside of the home.8

For our environmental variables, we utilize a variety of state-level policies aimed at reducing the spread of COVID-19, largely through decreasing social contact (White House, 2020). These policies were meant to “flatten the curve” and allow the healthcare sector and public health officials time to combat the epidemic. A majority of studies have focused on the impact of SIPO policies, and the differential impacts of these policies in early policy adopters and high population density states (Dave et al., 2020); the overall effectiveness of these policies on COVID-19 case rates (Abouk and Heydari, 2020, Friedson et al., 2020, Sears et al., 2023); and the impact on health outcomes (Dave et al., 2020, Friedson et al., 2020). One benefit of our study is that we use a variety of different social distancing policies, as there is considerable state-level heterogeneity in which policies were adopted. This is especially important because many states had multiple policies occurring at the same time. For instance, California had the nation’s earliest SIPO on March 19th, but also instituted essential business closures on March 19th and placed a ban on large gatherings on March 16th. We therefore estimate efficiency results under the following scenarios: (i) any of the 4 policies put in place (Any 4); or (ii) a SIPO policy only (SIPO only).

If we are able establish a relationship between the effectiveness of COVID-19 cases and social distancing policies, we will investigate the impact of social distancing policies on deaths from COVID-19. Even though policymakers attempt to control the COVID-19 case rate through various legislative measures, the end goal is to minimize hospitalizations and/or deaths from COVID-19.9 Therefore, our output measure will be the fraction of the population that is not considered a COVID-19 death. Like for COVID-19 cases, this is a state-day tally. Our inputs are: (i) the fraction of the population that is not considered a positive COVID-19 case; (ii) the average number of days (in the past 30) individuals are considered in good health; and (iii) the average fraction of the population that is not considered obese. The first is a daily measure, captured at the state level. The latter two are an annual state-level measures. The latter two are also meant to address the relatively high COVID-19 mortality rate in America, due to demographics and comorbidities (Banerjee et al., 2020, Dorjee et al., 2020, Hoffmann and Wolf, 2021, Saxena and Hashmi, 2021, Tartof et al., 2020, Thakur et al., 2021).

For our environmental variables, we utilize a broad range of social distancing policies: (i) any of the 4 policies put in place (Any 4); (ii) all 4 policies in place (All 4); (iii) a SIPO policy, non-essential business closure policy, and ban on large gatherings (S/B/L); or (iv) a SIPO policy (SIPO only). Similarly, though a variety of local, state, and federal policies have been addressed at reducing the spread of COVID-19 cases, this is an input to the ultimate goal of the healthcare system: minimizing the stress on the system, in terms of hospitalizations and deaths, by “flattening the curve”. This suggests that daily COVID-19 cases is an input, while daily COVID-19 deaths is an output.

Estimators:

Non-parametric estimators are often used by researchers because they do not require an a priori specification of the functional relationship that is being estimated, unlike parametric efficiency estimators. Due to the lack of distributional assumptions, incorporating multiple inputs and/or outputs is seamless. However, certain non-parametric estimators, such as the data envelopment analysis (DEA) estimator, suffer from well-known shortcomings. The shortcomings include having less than root-n convergence due to the curse of dimensionality, where the number of observations required to obtain meaningful estimates increases with the number of production inputs and outputs used in the estimation, and the estimator being sensitive to outliers (Kneip et al., 1998). This is because these types of efficiency estimators are “full frontier” estimators and incorporate all observations into the production frontier.

Newer non-parametric estimators have been developed in recent years: these include the order-$\alpha$ and order-m estimators. Both estimators eliminate many of the issues found in other non-parametric estimators, like the DEA estimator, and are not sensitive to outliers and have the classical, parametric, root-n rate of convergence; they are “partial frontier” estimators, meaning that some observations are “super-efficient” and are left outside of the production frontier (Cazals et al., 2002, Simar and Wilson, 2008, Wheelock and Wilson, 2009). These partial frontier estimators provide the distributional flexibility of non-parametric estimators while simultaneously providing traditional statistical features found in parametric estimators. In this analysis, we utilize the order-m estimator. Next, we briefly discuss the statistical features of the order-m estimator, noting that we have a set, $\chi$, of n state-day observations, characterized by p inputs, $x\left(x_1\dots x_p\right)$ and q outputs, $y\left(y_1\dots y_q\right)$.

Unconditional Order-m Estimator:

The order-m estimator was developed by Cazals et al. (2002), and denotes the best production set as the free disposal hull (FDH) of undominated input–output combinations

$${\Psi }_{FDH}=\left\{\left(x,y\right)\in {\mathfrak{R}}_{+}^{p+q}|x\le X_i,y\le Y_i\in \chi \right\}$$ (1)

We choose an output-orientation for the order-m estimator, where given a fixed level of inputs, what is the output shortfall for a county if it would produce as efficient as the observations on the best practice frontier.10 As in Fried et al. (2008), the output-oriented efficiency estimates measure the distance to the best practice frontier

$$\lambda \left(x_0,y_0\right)=sup\left\{\lambda |x_o,\lambda y_o\right\}\in {\Psi }_{FDH}$$ (2)

In the output-orientation, an inefficient observation has an efficiency score, $\lambda$, larger than 1. The value $\left(\lambda -1\right)$ indicates the potential percentage increase in output if the observation would produce as efficient as its reference partner. Unfortunately, the model in (2) is deterministic, and may be influenced by outliers. Cazals et al. (2002) mitigate these outlier issues by drawing from the sample population, with replacement, subsamples of size $m<n$ among those $Y_i$ such that $x_0\ge X_i$ (observations with fewer inputs than the evaluated observation).

Details of this method are shown in Cazals et al. (2002), who note that the order-m estimator achieves the parametric root-n rate of convergence. This partial sample size, m, is determined as the value for which the number of super-efficient observations is constant. The sampling and efficiency estimations are done B times (where B is sufficiently large), and the order-m efficiency estimates, ${\lambda }^m\left(x_0,y_0\right)$ are obtained as the arithmetic average of the B inefficiencies or, conversely, an integral formulation of this bootstrap.

Conditional Order-m Efficiency Measures:

Order-m efficiency estimates can be adapted to take into account exogenous variation, denoted by a vector of environmental variables z, between states, that do not require the separability assumption that is found in Simar and Wilson, 2007, Simar and Wilson, 2011. These methods were developed by Daraio and Simar (2005), who proposed drawing subsamples of size m by a given probability, which is determined by a Kernel function around the variables Z, drawn B times with replacement. The B efficiency estimates are averaged to obtain the conditional order-m estimates ${\lambda }^m\left(x_0,y_0|z_0\right)$, where interpretations are similar to the unconditional order-m estimator. However, the estimation allowed only continuous environmental variables to be included in the estimation. As shown in Daraio and Simar (2005), the conditional order-m estimator may not have root-n convergence, as the convergence rate depends on the dimension of Z, necessitating a parsimonious specification of the vector of exogenous variables that influence efficiency scores.

De Witte and Kortelainen (2013) extend the conditional order-m estimator to include discrete environmental variables, which does not influence the convergence rate of the conditional order-m estimator any further, as econometric theory states that the convergence rate of non-parametric estimators for conditional density and distribution functions involving mixed variables only depend on the number of continuous variables. De Witte and Kortelainen (2013) construct a ratio of conditional to unconditional estimates, $\frac{{\lambda }^m\left(x_0,y_0|z_0\right)}{{\lambda }^m\left(x_0,y_0\right)}$, which are regressed on the Z environmental variables.

Since the regression of environmental variables Z is on a ratio, the marginal coefficient is less meaningful than a standard regression. If Z is multivariate, one can utilize partial regression plots for the visualization of the effect to determine how a single environmental variable in Z affects the production process, holding the other variables in Z constant. In the output-orientation, a horizontal line implies no effect and an increasing (decreasing) smoothed regression curve shows that Z improves (decreases) efficiency in the production process. De Witte and Kortelainen (2013) obtain p-values of the significance of the influence of Z on the efficiency scores, based on the work of Li and Racine (2007), by utilizing a local linear regression estimation and a non-parametric naive bootstrap procedure. Unlike the second-stage efficiency regression proposed by Simar and Wilson (2007), this model does not need a separability assumption for proper inference to be determined.

4. Results

Table 1 provides our estimates where we utilize SafeGraph, Inc.’s social mobility measures as inputs, and the fraction of the population that is not considered (at any point) a COVID-19 case as output in the first 100 days.

Table 1.

Efficiency estimates, unconditional and conditional, population without cases as output variable, first 100 days.

State	Unconditional	Conditional (Any 4)	Conditional (SIPO only)	State	Unconditional	Conditional (Any 4)	Conditional (SIPO only)
Alabama	0.2533	0.4477	0.341193	Montana	0.2234	0.7206	0.352871
Alaska	0.2604	0.6043	0.344204	Nebraska	0.2645	0.7178	0.419481
Arizona	0.2485	0.3696	0.27889	Nevada	0.2411	0.4620	0.257208
Arkansas	0.2363	0.5119	0.414822	New Hampshire	0.2571	0.4365	0.274209
California	0.2424	0.3951	0.169192	New Jersey	0.2828	0.5536	0.242398
Colorado	0.2621	0.5028	0.332942	New Mexico	0.2518	0.3920	0.280787
Connecticut	0.2756	0.5102	0.305339	New York	0.2926	0.5391	0.246928
Delaware	0.2712	0.3853	0.279664	North Carolina	0.2459	0.4072	0.255534
DC	0.3052	0.6001	0.318939	North Dakota	0.2548	0.7617	0.428852
Florida	0.2493	0.5113	0.327735	Ohio	0.2373	0.3784	0.237795
Georgia	0.2562	0.4442	0.340264	Oklahoma	0.2253	0.4879	0.288499
Hawaii	0.7727	0.7675	0.640538	Oregon	0.2272	0.3656	0.173362
Idaho	0.2306	0.5402	0.322617	Pennsylvania	0.2633	0.5771	0.329734
Illinois	0.2775	0.5348	0.266731	Rhode Island	0.2577	0.5551	0.443799
Indiana	0.2635	0.5044	0.312149	South Carolina	0.2383	0.4762	0.344535
Iowa	0.2640	0.4264	0.422937	South Dakota	0.2658	0.4457	0.431784
Kansas	0.2420	0.4820	0.315762	Tennessee	0.2471	0.4323	0.34703
Kentucky	0.2322	0.3691	0.41617	Texas	0.2338	0.3625	0.320744
Louisiana	0.2742	0.5060	0.293413	Utah	0.2425	0.6061	0.314499
Maine	0.2338	0.5225	0.212855	Vermont	0.2506	0.6476	0.327564
Maryland	0.2779	0.3554	0.313276	Virginia	0.2644	0.4686	0.250814
Massachusetts	0.2912	0.3257	0.434786	Washington	0.2380	0.4150	0.209593
Michigan	0.2603	0.4793	0.239212	West Virginia	0.2281	0.4916	0.285922
Minnesota	0.2572	0.6184	0.280493	Wisconsin	0.2440	0.4364	0.262541
Mississippi	0.2652	0.4465	0.360886	Wyoming	0.2333	0.6106	0.414167
Missouri	0.2286	0.4960	0.319624

Summary statistics
Variable	Unconditional	Conditional (Any 4)	Conditional (SIPO only)
Mean	0.2636	0.4981	0.3205

Note: To interpret the efficiency estimates, a value of less than 1 for the unconditional and conditional order-m estimators correspond to being “inefficient”, while a value of greater than 1 corresponds to “super-efficient”. Any 4 refers to any of the 4 policies in place on day j. SIPO refers to shelter-in-place orders.

The average state is 73.4-percent inefficient, based on social mobility data, in leading to fewer cases with COVID-19. There are two potential explanations for this finding: (i) behavioral noncompliance; or (ii) the importance of super-spreader events in the early days of COVID-19 during the pandemic when a majority of individuals were compliant. Our unconditional results suggest that super-spreader events were the likely cause, given the high degree of people staying-at-home. However, once we condition on either having any of the four social distancing policies (Any 4) or having a SIPO only (SIPO only), conditional effectiveness improves by 23.5 and 5.7 percentage points, respectively. This suggests the highly effective nature, in the first 100 days, of COVID-19 social distancing policies. Our estimates imply that social distancing policies led to 140.7 fewer cases per 100,000 population in the first 100 days, while having a SIPO only led to 37.7 fewer cases per 100,000 population in the first 100 days. These estimates are lower in magnitude to those provided in regression estimates by Dave et al. (2020) and Friedson et al. (2020), but qualitatively similar. Now that we have highlighted the relationship between COVID-19 social distancing policies and COVID-19 cases, we investigate the relationship between these same policies and COVID-19 deaths.

Table 2 provides estimates for the unconditional and conditional order-m estimates, where the fraction of the population that has not died from COVID-19 is the output variable. Unconditional effectiveness refers to how states fared in combatting COVID-19 deaths without social distancing policies; conditional effectiveness refers to how states addressed their shortcomings using social distancing policies.

Table 2.

Efficiency estimates, unconditional and conditional, March 1 to September 1.

State	Unconditional	Conditional (Exogenous variables labeled)
	Eff.	Any 4	All 4	S/B/L	SIPO
Alabama	0.6231	0.6264	–	0.6252	0.6252
Alaska	0.7341	0.7436	0.7432	–	0.7367
Arizona	0.6481	0.6582	0.6567	–	0.6519
Arkansas	0.6000	0.6059	–	–	–
California	0.7937	0.8067	–	0.7988	0.8103
Colorado	0.8650	0.8679	–	0.8685	0.8685
Connecticut	0.9775	0.9870	0.9775	–	0.9847
Delaware	0.8697	0.8906	0.8851	–	0.8772
DC	0.9776	0.9977	–	0.9847	0.9846
Florida	0.6852	0.6990	0.6949	–	0.6879
Georgia	0.6983	0.7025	–	0.7002	0.7008
Hawaii	0.9176	0.9430	0.9368	–	0.9287
Idaho	0.7596	0.7677	0.7653	–	0.7632
Illinois	0.7575	0.7629	–	0.7641	0.7642
Indiana	0.7648	0.7686	–	0.7687	0.7687
Iowa	0.9827	0.9900	–	–	–
Kansas	0.9244	0.9422	0.9391	–	0.9285
Kentucky	0.6000	0.6127	–	–	–
Louisiana	0.6877	0.6984	–	0.6923	0.6923
Maine	0.8186	0.8350	0.8291	–	0.8342
Maryland	0.9055	0.9288	0.9244	–	0.9109
Massachusetts	0.8468	0.8812	–	–	–
Michigan	0.7132	0.7173	–	0.7160	0.7194
Minnesota	0.9737	0.9883	–	0.9737	0.9812
Mississippi	0.6795	0.6842	–	0.6816	0.6816
Missouri	0.6337	0.6368	–	0.6352	0.6360
Montana	0.8225	0.8383	0.8331	–	0.8254
Nebraska	0.9736	0.9824	–	–	0.9732
Nevada	0.7194	0.7245	–	0.7229	0.7247
New Hampshire	0.8534	0.8671	–	0.8581	0.8617
New Jersey	0.8278	0.8370	0.8278	–	0.8351
New Mexico	0.6582	0.6732	0.6727	–	0.6631
New York	0.7819	0.7874	0.7819	–	0.7914
North Carolina	0.7469	0.7584	–	0.7506	0.7519
North Dakota	1.0000	1.0000	–	–	–
Ohio	0.7616	0.7749	0.7714	–	0.7672
Oklahoma	0.6481	0.6561	0.6552	–	0.6516
Oregon	0.7650	0.7692	–	0.7702	0.7806
Pennsylvania	0.8111	0.8154	–	0.8149	0.8149
Rhode Island	0.8302	0.8484	0.8453	–	0.8347
South Carolina	0.6757	0.6893	0.6802	–	0.6781
South Dakota	0.8898	0.8930	–	–	–
Tennessee	0.6067	0.6103	–	0.6090	0.6090
Texas	0.7707	0.7829	0.7805	–	0.7735
Utah	0.8316	0.8369	–	–	0.8354
Vermont	0.8442	0.8594	0.8524	–	0.8497
Virginia	0.8097	0.8155	–	0.8145	0.8166
Washington	0.7427	0.7548	–	0.7475	0.7492
West Virginia	0.5738	0.5831	0.5824	–	0.5767
Wisconsin	0.8412	0.8459	–	0.8462	0.8464
Wyoming	0.8505	0.8567	–	–	–
Mean efficiency (% Point)	0.7857	1.297	0.496	0.393	0.609
Maximum change (% Point)	–	5.15 (MA)	4.09 (NM)	1.61 (IL)	1.89 (CA)
Minimum change (% Point)	–	0.00 (ND)	0.00 (NJ)	0.00 (MN)	0.00 (NE)
Census division	Eff.	Percentage point improvement
New England	0.8618	2.075	0.973	0.555	0.951
Middle Atlantic	0.8069	0.777	0.000	0.474	0.850
East North Central	0.7676	0.811	1.282	0.599	0.721
West North Central	0.9111	0.839	1.592	0.094	0.386
South Atlantic	0.7714	1.693	1.536	0.539	0.608
East South Central	0.6273	0.971	–	0.340	0.340
West South Central	0.6766	1.325	1.202	0.674	0.518
Mountain	0.7693	1.139	1.371	0.440	0.527
Pacific	0.7906	1.544	1.712	0.649	1.133

Note: To interpret the efficiency estimates, a value of less than 1 for the unconditional and conditional order-m estimators correspond to being “inefficient”, while a value of greater than 1 corresponds to “super-efficient”. SIPO refers to shelter-in-place orders. Business refers to non-essential business closures. Large refers to large gathering bans. Travel refers to mandatory quarantine for travelers. Any 4 refers to any of the 4 policies in place on day j. All 4 refers to all 4 of the policies in place on day j. S/B/L refers to a SIPO, business, and large policy in place on day j.

In general, the mean effectiveness is 0.7857, or the average state is 21.4-percent ineffective (or could produce 21.4-percent more output for its input level to be considered fully effective) in the findings for the unconditional estimates. This would correspond to about 37,000 fewer COVID-19 deaths on September 1, 2020 (cumulatively) than what was experienced. The mean unconditional effectiveness also hides heterogeneity across the states. For instance, New York State had an unconditional effectiveness of 0.7818 meaning that output was 21.8-percent too low, given input values. Had New York State been fully effective in combatting COVID-19, there would have been nearly 7200 fewer deaths cumulatively on September 1, 2020. The minimum unconditional effectiveness is given by West Virginia, which is 42.6 percent ineffective. The maximum unconditional effectiveness was North Dakota, which was considered fully effective. The bottom of Table 2 shows that there was tremendous regional heterogeneity in COVID-19. The West North Central Census region of the country was the most effective, being only 8.9-percent ineffective while the worst was the East South Central Census region of U.S. considered 37.3-percent ineffective. These suggest, in line with Sachs et al. (2022), that the U.S. was unprepared for a pandemic, and that few subnational units could adopt a “herd immunity” strategy.

The last 4 columns of Table 2 provide the conditional effectiveness estimates for a variety of social distancing policies: (i) Any 4, which represents that a state on day j had any 4 of the social distancing policies; (ii) All 4, which represents that a state on day j had all 4 of the social distancing policies; (iii) S/B/L, which represents that a state on day j had a SIPO, non-essential business closure, and a ban on large gatherings; and (iv) SIPO, which represents that a state on day j had a SIPO.

Focusing on the column Any 4, we find that, on average, conditioning for any of the four policies led to an average effectiveness improvement of 1.30 percentage points. This corresponds to 2000 lives cumulatively saved by September 1, 2020, by the heterogeneous state-level social distancing policies. The biggest gain was found in Massachusetts where nearly 300 lives (cumulatively) were saved by the polices enacted by September 1, 2020. For the 21 states that enacted all 4 policies on at least some days during the COVID-19 pandemic, results show that mean effectiveness improved by 0.50 percentage points. Cumulatively, this suggests that having all 4 policies reduced U.S. COVID-19 deaths by September 1, 2020 by nearly 600 total deaths. It infers that multiple social distancing policies put in place may have had (at least) two deleterious effects: (i) they confused businesses and individuals who may have violated restrictions inadvertently because there were a myriad of different state-level laws to comply with; or (ii) they led to behavioral noncompliance by businesses and individuals who felt that multiple policies were overly restrictive. We see similar results for states that had S/B/L, which are SIPO, non-essential business closure, and ban on large gathering policies at the same time. Here, effectiveness improved by only 0.393 percentage points. These results may also be a result of the U.S. focusing on a “flatten the curve” strategy, rather than “suppression”, where flatten the curve spreads out cumulative COVID-19 cases over a longer period of time, but does not reduce the number (Anderson et al., 2020, Sachs et al., 2022, Thunstrom et al., 2020). Given the political nature of the COVID-19 pandemic in the U.S. outlined by Sachs et al. (2022), there could be another cause of the seeming lack of effectiveness of All 4 policies on substantially reducing COVID-19 death rates: the premature lifting of public health and social distancing measures. Sachs et al. (2022) note that premature lifting led to the omicron variant, and that premature lifting of policies were problematic because vulnerable groups were not able to achieve immunity as well as vaccine coverage (and effectiveness) waning over time (Moayyedi, 2022). Grafton et al. (2021) note that the benefits of social distancing policies worsened public health and economic outcomes.

If we focus on states that passed at least a SIPO policy, we find that having a SIPO policy improved conditional effectiveness by 0.61 percent corresponding to almost 700 fewer cumulative deaths by September 1, 2020, which is higher than the reduced death toll from implementing all 4 policies. Results also show that the earliest adopter of a SIPO, California (March 19, 2020) had the highest percentage point improvement for conditional effectiveness, improving by 2.04-percent or a cumulative reduction in total deaths on September 1, 2020 of nearly 270.

One could broadly conclude that a national SIPO order only would likely have been the most effective policy set if there was an incentive to employ one. However, the findings that conditional mean effectiveness improved the most by allowing states the ability to create their own heterogeneous policy sets suggests that the variety of state-level actions were optimal, even if they were not uniform. Though this is counter to the optimal recommendations in Sachs et al. (2022), it likely takes into account the political realities in different states, as well as the willingness of the population to comply with mandates.

As noted in a variety of papers (Dave et al., 2020, Friedson et al., 2020), the benefits of many of these policies were likely short-lived if they were not maintained (Grafton et al., 2021, Moayyedi, 2022, Sachs et al., 2022). Both Kompas et al. (2021) and Grafton et al. (2021) note that policies were more effective the sooner they were implemented, as long as they were stringent enough. In Table 3, we restrict our sample to the 100 days after March 1, 2020, as the “critical period” during which the COVID-19 pandemic ramped up.

Table 3.

Efficiency estimates, conditional and unconditional, first 100 days.

State	Unconditional	Conditional (Exogenous variables labeled)
	Eff.	Any 4	All 4	S/B/L	SIPO	SIPO & Dist.
Alabama	0.6231	0.6290	–	0.6269	0.6269	0.6356
Alaska	0.7341	0.7517	0.7509	–	0.7389	0.8625
Arizona	0.6481	0.6666	0.6640	–	0.6551	0.6880
Arkansas	0.6000	0.6096	–	–	–	–
California	0.7937	0.8057	–	0.8031	0.8088	0.8164
Colorado	0.8649	0.8700	–	0.8715	0.8715	0.8828
Connecticut	0.9775	0.9951	0.9775	–	0.9908	0.9936
Delaware	0.8689	0.9064	0.8974	–	0.8830	0.8851
DC	0.9774	0.9957	–	0.9907	0.9907	0.9963
Florida	0.6852	0.7108	0.7031	–	0.6902	0.7224
Georgia	0.6983	0.7057	–	0.7018	0.7029	0.7040
Hawaii	0.9175	0.9620	0.9530	–	0.9336	0.9835
Idaho	0.7595	0.7729	0.7701	–	0.7664	0.8013
Illinois	0.7575	0.7674	–	0.7698	0.7698	0.7715
Indiana	0.7654	0.7722	–	0.7728	0.7738	0.7753
Iowa	0.9826	0.9960	–	–	–	–
Kansas	0.9244	0.9575	0.9516	–	0.9320	0.9346
Kentucky	0.6000	0.6227	–	–	–	–
Louisiana	0.6877	0.6972	–	0.6962	0.6962	0.7046
Maine	0.8185	0.8473	0.8379	–	0.8315	0.8679
Maryland	0.9055	0.9481	0.9404	–	0.9154	0.9179
Massachusetts	0.8462	0.8897	–	–	–	–
Michigan	0.7132	0.7206	–	0.7185	0.7248	0.7267
Minnesota	0.9737	0.9861	–	0.9737	0.9876	0.9882
Mississippi	0.6795	0.6879	–	0.6834	0.6834	0.6877
Missouri	0.6337	0.6393	–	0.6364	0.6379	0.6391
Montana	0.8223	0.8507	0.8412	–	0.8281	0.8329
Nebraska	0.9732	0.9893	–	–	0.9732	0.9772
Nevada	0.7194	0.7287	–	0.7258	0.7291	0.7323
New Hampshire	0.8539	0.8666	–	0.8629	0.8682	0.8877
New Jersey	0.8275	0.8374	0.8275	–	0.8425	0.8455
New Mexico	0.6582	0.6860	0.6851	–	0.6673	0.6692
New York	0.7818	0.7904	0.7818	–	0.7960	0.8034
North Carolina	0.7469	0.7569	–	0.7538	0.7562	0.7619
North Dakota	1.0000	1.0000	–	–	–	–
Ohio	0.7616	0.7616	0.7797	–	0.7720	0.7736
Oklahoma	0.6481	0.6481	0.6613	–	0.6545	0.6577
Oregon	0.7650	0.7650	–	0.7746	0.7788	0.7851
Pennsylvania	0.8111	0.8111	–	0.8182	0.8182	0.8196
Rhode Island	0.8302	0.8638	0.8581	–	0.8385	0.8570
South Carolina	0.6767	0.6908	0.6840	–	0.6802	0.7025
South Dakota	0.8898	0.8956	–	–	–	–
Tennessee	0.6067	0.6133	–	0.6109	0.6109	0.6120
Texas	0.7707	0.7930	0.7889	–	0.7758	0.7769
Utah	0.8316	0.8412	–	–	0.8386	0.8551
Vermont	0.8441	0.8711	0.8593	–	0.8545	0.9139
Virginia	0.8097	0.8203	–	0.8186	0.8224	0.8259
Washington	0.7427	0.7539	–	0.7515	0.7547	0.7573
West Virginia	0.5738	0.5909	0.5897	–	0.5792	0.5835
Wisconsin	0.8412	0.8498	–	0.8505	0.8509	0.8536
Wyoming	0.8505	0.8620	–	–	–	–
		% Point improvement
Mean efficiency	0.7857	2.1	0.93	0.76	1.0	2.8
Max change	–	5.2 (MA)	4.1 (NM)	1.6 (IL)	1.9 (CA)	17.5 (AK)
Min change	–	0.0 (ND)	0.0 (NJ)	0.0 (MN)	0.0 (NE)	0.4 (NE)
Census division	Eff	% Point improvement
New England	0.8617	3.154	1.801	1.061	1.369	4.128
Middle Atlantic	0.8068	1.091	0.000	0.878	1.504	1.987
East North Central	0.7678	1.492	2.372	1.114	1.365	1.615
West North Central	0.9110	1.356	2.946	0.173	0.738	0.587
South Atlantic	0.7713	2.656	2.843	1.010	1.135	2.279
East South Central	0.6273	1.739	–	0.629	0.629	1.108
West South Central	0.6766	2.081	2.223	1.247	0.958	1.253
Mountain	0.7693	2.007	2.502	0.824	0.983	2.739
Pacific	0.7906	2.351	3.167	1.201	1.562	6.368

Note: To interpret the efficiency estimates, a value of less than 1 for the unconditional and conditional order-m estimators correspond to being “inefficient”, while a value of greater than 1 corresponds to “super-efficient”. SIPO refers to shelter-in-place orders. Business refers to non-essential business closures. Large refers to large gathering bans. Travel refers to mandatory quarantine for travelers. Any 4 refers to any of the 4 policies in place on day j. All 4 refers to all 4 of the policies in place on day j. S/B/L refers to a SIPO, business, and large policy in place on day j. SIPO & Distance refers to having a SIPO social distancing policy, as well as SafeGraph social mobility environmental variables.

Focusing on conditional effectiveness, we find that having any of the COVID-19 social distancing policies leads to a 2.10 percentage point improvement in effectiveness; cumulatively, if the effectiveness gains from these social distancing policies were the same for the first 100 days of the COVID-19 pandemic as they were for our full sample, this would equate to 3711 fewer (cumulative) deaths, compared to 2000 for our full sample. This suggests a conditional productivity regression of 0.8 percentage points. This implies COVID-19 social distancing fatigue over time, as both individuals and businesses likely relaxed their behaviors as the pandemic and social policies continued. It also hints that effectiveness regressed between June 9, 2020 and September 1, 2020, suggesting that long-lasting social distancing policies may not only be ineffective, but could actually be harmful. This seems to be consistent with the “go early, go hard” approach advocated by Kompas et al. (2021). In fact, for the “suppression” strategy to truly be effective, it would have to be done earlier, rather than later (Mendez-Brito et al., 2021).

Similarly, findings suggest that social distancing policies have productivity regression between June 9, 2020 and September 1, 2020 of 0.43 (0.37; 0.40) percentage points if we focus on All 4 (S/B/L; SIPO Only). Again, it infers that while long-lasting social distancing policies are improvements over no social distancing policies, there are costs, including noncompliance, that may offset the positive gains. Given the tremendous economic costs that can occur from long-lasting policies (Acemoglu et al., 2020, Mulligan, 2020), individuals are likely change their preference set to weigh economic concerns more heavily than virus concerns. Again, we see that the state with the earliest SIPO policy, California, by implementing the policy, had 250 fewer deaths (if the productivity returns stayed constant between June 9, 2020 and September 1, 2020), relative to not implementing a SIPO. This suggests that over 90-percent of the effectiveness of California’s SIPO was in the first 100 days of implementing the policy.

Broadly, Table 3 suggests that a majority of the benefits of social distancing policies were in the first 100 days, which supports the “go early, go hard” approach advocated by a number of papers (Grafton et al., 2021, Kompas et al., 2021, Mendez-Brito et al., 2021, Sachs et al., 2022). In states such as California, over 90-percent of the effectiveness of the policies were in the first 100 days. Overall, the effectiveness of the social distancing policies put in place actually decreased between June 9, 2020 and September 1, 2020. This suggests that confusion over the state of the social distancing laws, or behavioral noncompliance from the economic impacts of long-lasting COVID-19 social distancing policies reduced the effectiveness of the guidelines over time. This supports the findings in Czeisler (2020), who found via survey that a majority of respondents supported social distancing policies early in the pandemic. Another potential explanation is that the decrease in conditional effectiveness after 100 days was due to the early expiration of social distancing policies. As noted in Grafton et al. (2021), Moayyedi (2022), and Sachs et al. (2022), the premature lifting of public health and social measures worsened outcomes. For instance, Arizona, Connecticut, Louisiana, Maryland, Minnesota, New Mexico, North Carolina, Ohio, Oklahoma, Pennsylvania, Rhode Island, Vermont, and Wisconsin all ended their SIPO policies in mid-May. Therefore, deaths would have started to accelerate after the end of the 100 day sample. This suggests that the premature ending of social distancing policies eliminated much of the positives of these policies that were initially in place, as conditional effectiveness worsened. However, the fact that conditional effectiveness was still, on average, higher than unconditional effectiveness, even in these states that ended SIPO adoption, suggests that the policies had long-lasting benefits by bending the curve early, and reducing total growth of the COVID pandemic. Though it may have been politicized, the “flatten the curve” strategy did help to reduce total COVID-19 deaths, even if they were ended prematurely. The fact that conditional effectiveness worsened even in states that had long-lasting social distancing policies suggests that early expiration of policies and behavioral noncompliance both contributed to the regression in effectiveness.

We provide a basic test of the hypothesis (as well as empirical evidence from Grafton et al. (2021), Moayyedi (2022), and Sachs et al. (2022)) that states which social distancing policies too early worsened conditional effectiveness weeks after the revocation of these policies in Table A.1. Given Dr. Fauci’s comments that deaths lag cases by 4 weeks, we code the end of a policy as being 30 days after the social distancing policy was formally revoked or expired. We then create 7 (or 14) day rolling averages before and after this date, then subtract conditional effectiveness before this window from conditional effectiveness after this window. To net out any pre-existing trends, we also subtract from this the difference between the 7 (or 14) day average of the unconditional effectiveness estimates after the policy change, minus the 7 (or 14) day average of the unconditional effectiveness estimates before the policy change. On average, the 7 (14) day rolling average suggests effectiveness worsens by 0.24 (0.42) percentage points after the revocation or expiration of the policy; the median change is no effectiveness regression. This suggests that premature revocation of COVID-19 social distancing policies only played a part (along with behavioral noncompliance), and was unlikely to be a causal factor for the increase in deaths.

While we can explore the impact of social distancing policies on effectiveness measurements, we also utilize the regression of conditional to unconditional effectiveness scores on social distancing environmental variables, shown by Cazals et al. (2002), Daraio and Simar, 2005, Daraio and Simar, 2007a, Daraio and Simar, 2007b, and De Witte and Kortelainen (2013) for a variety of environmental variables. Table 4 examines the impact of our different social distancing specifications (Any 4; All 4; S/B/L; SIPO Only) for our full sample.

Table 4.

Conditional efficiency regressions, various social distancing policies, full sample.

Panel A: Any Social Distancing Policy
Variable	p-Value	Better efficiency occurs when…
SIPO	0.29	–
Non-essential business closures	0.001***	A state has a non-essential business closure policy on day j.
Mandatory quarantine for travelers	0.001***	A state has a mandatory quarantine for travelers policy on day j.
Large gathering ban	0.001***	A state has a large gathering ban policy on day j.
Panel B: All Social Distancing Policies
All 4 policies	0.001***	A state has a SIPO, non-essential business closure, mandatory quarantine for travelers, and large gathering ban policy on day j.
Panel C: SIPO/Business/Large
SIPO/Business/Large	0.001***	A state has a SIPO, non-essential business closure, and large gathering ban policy on day j.
Panel D: SIPO Only
SIPO	0.001***	A state has a SIPO policy on day j.

*** refers to significance at the 1-percent level; ** refers to significance at the 5-percent level; * refers to significance at the 10-percent level.

Note: Panel A: Any Social Distancing Policy refers to if a state has any of the following policies on day j: (i) SIPO; (ii) non-essential business closure; (iii) mandatory quarantine for travelers; and/or (iv) ban on large gatherings. Panel B: All 4 Social Distancing Policies refers to if a state has all 4 of the following social distancing policies on day j: (i) SIPO; (ii) non-essential business closure; (iii) mandatory quarantine for travelers; and/or (iv) ban on large gatherings. Panel C: SIPO/Business/Large refers to if a state has all of the following policies on day j: (i) SIPO; (ii) non-essential business closure; and (iii) ban on large gatherings. Panel D: SIPO Only refers to if a state has a SIPO order on day j.

For any social distancing policy in Panel A (Any 4), we see that the effective policies in limiting COVID-19 deaths were non-essential business closures, mandatory quarantines for travelers, and bans on large gatherings. This suggests that there may have been more optimal ways for early adopters of SIPO policies to (further) fine tune their COVID-19 social distance restrictions to achieve reduced COVID-19 death rates. Fig. 2 provides the partial regression plot for Panel A.

Fig. 2.

Partial regression plot of non-parametric regression of efficiency ratio to any social distancing variables. Note: effratio refers to the ratio of conditional efficiency to unconditional efficiency. This figure analyzes the impact of having any of the social distancing policies in place on day j. We have set up the figures so that an upward-sloping diagram indicates productivity improvement, while a downward-sloping diagram indicates productivity regression.

We see that the most effective policies were non-essential business closures followed by bans on large gatherings. Although Abouk and Heydari (2020) found that non-essential business closures were an important source at limiting mobility, bans on large gatherings were not. This suggests that the mobility of individuals during COVID-19 may be only one component of prosocial behaviors (Sachs et al., 2022).

Panels B, C, and D provide our regression estimates for states with all social distancing policies (All 4), states with SIPO, non-essential business closure, and bans on large gatherings (S/B/L), and states with SIPO policies only (SIPO only). We again see that the ability to reduce COVID-19 spread to reduce COVID-19 deaths by was improved in each case by having COVID-19 social distancing policies in place. Our partial regression plots are presented in Fig. 3.

Fig. 3.

Partial regression plot of non-parametric regression of efficiency ratio to all social distancing variables (a), SIPO/Business/Large (b), and SIPO Only (c). Note: effratio refers to the ratio of conditional efficiency to unconditional efficiency. Fig. 2(a) represents have all 4 of the social distancing policies in place on day j; Fig. 2(b) represents having all of the SIPO, non-essential business closures, and bans on large gatherings policies in place on day j; and Fig. 2(c) represents having a SIPO policy on day j only. We have set up the figures so that an upward-sloping diagram indicates productivity improvement, while a downward-sloping diagram indicates productivity regression.

Similar to our analysis in Table 3 where we restricted our sample to the first 100 days of the COVID-19 pandemic, we regress the ratio of conditional to unconditional effectiveness estimates on social distancing policies. Our results are presented in Table 5.

Table 5.

Conditional efficiency regressions, various social distancing policies, first 100 days.

Panel A: Any Social Distancing Policy
Variable	p-Value	Better efficiency occurs when…
SIPO	0.07*	A state does not have a SIPO policy on day j.
Non-essential business closures	0.001***	A state has a non-essential business closure policy on day j.
Mandatory quarantine for travelers	0.001***	A state has a mandatory quarantine for travelers policy on day j.
Large gathering ban	0.001***	A state has a large gathering ban policy on day j.
Panel B: All Social Distancing Policies
All 4 policies	0.001***	A state has a SIPO, non-essential business closure, mandatory quarantine for travelers, and large gathering ban policy on day j.
Panel C: SIPO/Business/Large
SIPO/Business/Large	0.001***	A state has a SIPO, non-essential business closure, and large gathering ban policy on day j.
Panel D: SIPO Only
SIPO	0.001***	A state has a SIPO policy on day j.

*** refers to significance at the 1-percent level; ** refers to significance at the 5-percent level; * refers to significance at the 10-percent level.

Note: Panel A: Any Social Distancing Policy refers to if a state has any of the following policies on day j: (i) SIPO; (ii) non-essential business closure; (iii) mandatory quarantine for travelers; and/or (iv) ban on large gatherings. Panel B: All 4 Social Distancing Policies refers to if a state has all 4 of the following social distancing policies on day j: (i) SIPO; (ii) non-essential business closure; (iii) mandatory quarantine for travelers; and/or (iv) ban on large gatherings. Panel C: SIPO/Business/Large refers to if a state has all of the following policies on day j: (i) SIPO; (ii) non-essential business closure; and (iii) ban on large gatherings. Panel D: SIPO Only refers to if a state has a SIPO order on day j.

The results are qualitatively similar to the full sample, with one notable exception. Panel A shows the estimates for any social distancing policy (Any4); we find that compared to states with other types of social distancing policies, those that implemented SIPO policies actually saw a decrease in the effectiveness from having that policy in place, though significant at the 10-percent level only. This result is confirmed via the partial regression plot in Fig. 4.

Fig. 4.

Partial regression plot of non-parametric regression of efficiency ratio to any social distancing variables, first 100 days. Note: effratio refers to the ratio of conditional efficiency to unconditional efficiency. This figure analyzes the impact of having any of the social distancing policies in place on day j. We have set up the figures so that an upward-sloping diagram indicates productivity improvement, while a downward-sloping diagram indicates productivity regression.

Our findings are seemingly at odds with Dave et al. (2020) and Friedson et al. (2020), who found that SIPO’s are effective in reducing COVID-19 cases, though we focus on COVID-19 deaths. Friedson et al. (2020) did find that there were 1661 fewer COVID-19 deaths during the SIPO in place. Given that our analysis focusses on how effective COVID-19 cases were turned into deaths, we can reconcile the results by hypothesizing that there was a greater cumulative reduction in cases than in deaths, which would appear as productivity regression. We also note that there were other policies in place during this time in California, where Dave et al. (2020) and Friedson et al. (2020) limited their study to, that may have amplified the impacts of a SIPO.

Estimates in Panels B through D in Table 5 are identical to those in Table 4. Our results again suggest that during the first 100 days, social distancing policies were broadly effective in improving the efficiency of ensuring that COVID-19 cases were not turned into COVID-19 deaths. Our partial regression plots for Panels B through D are presented in Fig. 5.

Fig. 5.

Partial regression plot of non-parametric regression of efficiency ratio to all social distancing variables (a), SIPO/Business/Large (b), and SIPO Only (c). Note: effratio refers to the ratio of conditional efficiency to unconditional efficiency. Fig. 2(a) represents have all 4 of the social distancing policies in place on day j; Fig. 2(b) represents having all of the SIPO, non-essential business closures, and bans on large gatherings policies in place on day j; and Fig. 2(c) represents having a SIPO policy on day j only. We have set up the figures so that an upward-sloping diagram indicates productivity improvement, while a downward-sloping diagram indicates productivity regression.

5. Conclusion

Overall, we find that though social distancing policies were mildly effective in reducing COVID-19 deaths, long-lasting social distance policies were actually less effective in reducing COVID-19 deaths compared to policies that lasted less than the first 100 days. This suggests that, especially in the United States, policymakers have to consider three competing interests: (i) economic costs of imposing social distancing policies; (ii) health costs of not imposing social distancing policies; and (iii) behavioral noncompliance of a population to long-lasting social distancing policies. This also suggests that the “go early, go hard” approach advocated by Kompas et al. (2021) will have diminishing returns, given the highly politicized nature of COVID-19 found in Sachs et al. (2022).

Though not large, the average conditional effectiveness improvement of having any COVID-19 social distancing policy of 1.3 percentage points suggests that social distancing policies were effective in reducing COVID-19 deaths, though certainly not as large as the impacts on COVID-19 cases, as found in Dave et al. (2020), Friedson et al. (2020), and Gearhart et al. (2022). Though Friedson et al. (2020) only suggested that SIPOs reduce COVID-19 mortality, our results provide empirical evidence that they did limit COVID-19 mortality.

Importantly, our results suggest that early adopters of COVID-19 social distancing policies (similar to Dave et al., 2020) saw the biggest conditional efficiency gains, suggesting that early action was necessary to curb the spread of COVID-19; at least initially. This is consistent with the literature, where social distancing policies needed to be implemented early to have the largest impact (Grafton et al., 2021, Kompas et al., 2021, Moayyedi, 2022, Sachs et al., 2022). Our results also suggest that a uniform federal set of social distancing policies would have been ineffective. For instance, North Dakota was considered fully effective in combating COVID-19 without conditioning for COVID-19 policies, while West Virginia, with all four major COVID-19 social distancing policies, was least effective. Our results also suggest that more finely targeted COVID-19 policies, such as quarantining of sick individuals and closure of specific industries, may be the most effective policy tools that we have to fight pandemics. These targeted policies, like the optimal policies suggested by Acemoglu et al. (2020), may have induced more behavioral compliance, as well as minimized the economic costs of the policies.

What we consistently find is that after 100 days, which is about the time for deaths to start trending upwards after the expiration of a variety of social distancing policies, conditional effectiveness worsened. While we cannot rule out either explanation, it appears likely that early expiration of social distancing policies and behavioral noncompliance from longer-lasting social distancing policies both contributed to this productivity regression. In fact, given the findings in Cui et al. (2020), it is likely that once some states started to terminate their social distancing policies, a new Nash equilibrium may have occurred, where the population of a state began to expect their governments to reduce (or eradicate) their social distancing policies; behavioral noncompliance may have worsened in states, such as California, that did not take this step. But, as noted in Sachs et al. (2022), the premature lifting of public health and social distancing policies led to the omicron variant.

Importantly, research should be focused on the optimal length of social distancing policies. Our results in Table A.1 are only indicative that states which revoked their social distancing policies (or allowed them to retire) saw conditional effectiveness that was worse for 7 to 14 days after the lagged revocation or expiration date, which is empirically suggestive of the results in Sachs et al. (2022). Though not conclusive, we believe that there is evidence that some states which ended social distancing policies too early led to higher COVID-19 death rates later. Though our results confirm that COVID-19 social distancing policies reduced cumulative death rates, by 2.1 percentage points, which is rather small in magnitude. This suggests that social distancing policies have a small in numerical value impact. We believe that the impact may be larger and more definitive if we were to apply the same analysis to COVID-19 cases.

Lastly, while our focus on COVID-19 deaths is important, we are seeing that the consequences of COVID-19 may be more impactful in terms of the morbidity aspect, rather than the mortality aspect. So called “long COVID”, which is the persistence (or emergence) of symptoms at least 3 months after COVID infection, irrespective of viral status, is a growing public health threat (Crook et al., 2021, Raveendran et al., 2021). Given that long COVID is poorly understood, and impacts people regardless of the underlying severity of the initial infection, this has massive public health implications (Brackel et al., 2021, Decary et al., 2021, Yong, 2021). As noted in Carfi et al. (2020) and Tenforde et al. (2020), 35 percent of patients treated for COVID-19 and 87-percent of patients hospitalized with COVID-19 have long COVID, this has the potential for considerable long-term economic, social, and health costs, given the relatively low mortality rate underlying COVID-19. Though long COVID symptoms run the gamut from sore throat, abnormalities of taste and smell, and cough to neurocognitive issues, mental health disorders, and fatigue, further research may be used to examine how well social distancing policies helped avoid COVID-19 cases from turning into long COVID (Aiyegbusi et al., 2021, Graham et al., 2021, Lamb et al., 2022, Nabavi, 2020, Stefano, 2021). Though this may be hard to ascertain given data limitations on what classifies long COVID, this is an intriguing and important public health topic for the future.

Appendix.

See Table A.1.

Table A.1.

Conditional efficiency change after expiration or revocation of social distancing policy, 7 and 14 day rolling average.

State	7 day average	14 day average		7 day average	14 day average
Alabama	0.00	0.00	Montana	−0.31	−1.34
Alaska	0.00	0.00	Nebraska	−2.31	−2.52
Arizona	0.00	0.00	Nevada	0.00	0.00
Arkansas	−0.50	−1.10	New Hampshire	0.00	0.00
California	0.00	0.00	New Jersey	0.00	−0.11
Colorado	0.00	0.00	New Mexico	0.00	0.00
Connecticut	0.00	−0.12	New York	−0.12	−0.10
Delaware	−0.20	−1.22	North Carolina	0.00	0.00
District of Colombia	0.00	0.00	North Dakota	0.00	0.00
Florida	−1.30	−1.63	Ohio	−0.63	−1.12
Georgia	0.00	−0.52	Oklahoma	0.00	0.00
Hawaii	0.00	−0.43	Oregon	0.00	0.00
Idaho	0.00	0.00	Pennsylvania	0.00	0.00
Illinois	−0.10	−0.13	Rhode Island	−0.30	−1.44
Indiana	0.00	0.00	South Carolina	0.00	0.00
Iowa	−1.50	−1.64	South Dakota	0.00	0.00
Kansas	0.00	−0.74	Tennessee	−0.21	−0.61
Kentucky	−0.10	−0.40	Texas	0.00	0.00
Louisiana	0.00	0.00	Utah	0.00	0.00
Maine	−1.90	−2.12	Vermont	0.00	−0.30
Maryland	−1.40	−2.03	Virginia	0.00	−0.74
Massachusetts	0.00	0.10	Washington	0.00	0.00
Michigan	−0.90	−1.11	West Virginia	0.00	0.00
Minnesota	0.00	0.60	Wisconsin	0.00	0.00
Mississippi	−0.30	−0.81	Wyoming	0.00	0.00
Missouri	0.00	0.00

Note: We define the impact of a revocation (or expiration) of a social distancing policy as occurring 30 days after the date was social policy was removed (so, a policy removed April 30, 2020 would be coded as May 30, 2020). We do this as COVID-19 deaths data tends to lag, by 4 weeks, changes that lead to a rise in cases. The 7 Day Average represents the percentage point difference in the Any 4 conditional efficiency measure in the 7 days after the day a state revokes (or lets expire) any of the four social distancing policies, minus the Any 4 conditional efficiency measure in the 7 days before a state revokes (or lets expire) any of the social distancing policies. The 14 Day Average represents the percentage point difference in the Any 4 conditional efficiency measure in the 14 days after the day a state revokes (or lets expire) any of the four social distancing policies, minus the Any 4 conditional efficiency measure in the 14 days before a state revokes (or lets expire) any of the social distancing policies.