Systematic review of empirical studies comparing the effectiveness of non-pharmaceutical interventions against COVID-19

Abstract

Objectives

To evaluate which non-pharmaceutical interventions (NPIs) have been more and less effective in controlling the COVID-19 pandemic.

Methods

We performed a systematic review of published and unpublished empirical studies, either observational or interventional, analysing the comparative effectiveness of NPIs against the COVID-19 pandemic. We searched Embase/Medline and medRxiv to identify the relevant literature.

Results

We identified 34 studies. During the first wave of the COVID-19 pandemic, school closing was the most effective NPI, followed by workplace closing, business and venue closing and public event bans. Public information campaigns and mask wearing requirements were also effective in controlling the pandemic while being less disruptive for the population than other NPIs. There was no evidence on the effectiveness of public transport closure, testing and contact tracing strategies and quarantining or isolation of individuals. Early implementation was associated with a higher effectiveness in reducing COVID-19 cases and deaths, while general stringency of the NPIs was not.

Conclusions

In this systematic review, we found that school closing, followed by workplace closing, business and venue closing and public event bans were the most effective NPIs in controlling the spread of COVID-19. An early response and a combination of specific social distancing measures are effective at reducing COVID-19 cases and deaths. Continuous monitoring of NPIs effectiveness is needed in order to adapt decision making.

Introduction

In December 2019, a pneumonia-like disease caused an outbreak in the city of Wuhan, China.¹ This disease, later named COVID-19, spread globally and was declared a pandemic in March 2020 by the World Health Organisation. By April 2021 it has already affected around 145 million people and resulted in more than three million deaths globally.² Until effective treatments are available and vaccines are extensively accessible and administered, governments rely on non-pharmaceutical interventions (NPIs) to control the epidemic. The positive effects of implementing NPIs in controlling the COVID-19 pandemic have been widely studied both at the national3, 4, 5 and the international level.⁶ However, due to the high social and economic costs of many of the interventions implemented, it is essential to understand their individual effectiveness to optimize implementation and lifting strategies.^{7 , 8} A wide range of responses has been implemented worldwide, relying on previous knowledge of NPIs in controlling other epidemics.⁹ Several intervention types have been implemented, including containment measures such as domestic or international travel bans, individual protection measures like mask wearing requirements, social distancing measures such as school closing and gathering bans and health system measures like testing and contact tracing strategies. Evidence on the effectiveness of NPIs is largely based on mathematical modeling, with a limited number of empirical studies, either observational or interventional, exploring this topic. The assessment of empirical studies provides real world effectiveness estimations that do not rely strongly on assumptions as do simulations in modeling. In this review, we summarize the current evidence from empirical studies on the comparative effectiveness of NPIs that have been implemented worldwide to control the current COVID-19 pandemic.

Methodology

In this review we followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)¹⁰ statement. We searched Embase (including Medline resources) and MedRxiv for published literature and preprints, respectively. We restricted the search to articles available in English from January 1, 2020. The search was conducted on March 4 2021, for Embase and on March 9 for MedRxiv. The search strategies used for both databases are available in Annex 1.

Studies were included in the review if they:

• •Assessed NPI effectiveness only in the context of COVID-19.
• •Were either observational or interventional (i.e. quasi-experimental or experimental) studies of empirical data.
• •Included at least one of the following NPIs, as described and categorised in the Oxford COVID-19 Government Response Tracker (OxCGRT):¹¹ school closing, workplace closing, public event cancelation, social gathering restrictions, public transport closure, stay-at-home requirements, internal movement restrictions, international travel restrictions, public information campaigns, testing policies, contact tracing policies and facial covering policies.
• •Compared the effectiveness of at least two NPIs.
• •Analysed NPI effectiveness in the general population of any geographical area.
• •Measured any health outcome.

Studies were excluded from the review if they:

• •Were based on forecasts or simulations.
• •Did not assess the direct link between NPIs and the health outcome (for example, if the link was based on mobility).
• •Analysed the impact of adherence or compliance to NPIs.
• •Did not pre-specify the NPIs explored before the analysis (for example, breaking point analysis of epidemic curves were excluded).

To perform the quality assessment of the studies, we used a risk of bias tool based on a bibliometric review of ecological studies¹² and previously used in two published systematic reviews.^{13 , 14} The tool assesses the study design, statistical methodology and quality of reporting. We added one question to the tool to expand the methodological assessment of the studies included. The risk of bias tool checklist and the final risk of bias rating of the studies are available in Annex 2.

One reviewer (AMB) screened the records, selected the studies for the review, extracted the data and assessed the risk of bias. A second reviewer (FPM) screened ten percent of the total records, all the records that were selected by abstract, and verified risk of bias judgments.

Results

34 studies were included in the review, from which 28 have been published, one of them in a journal without peer review, and six were preprints. The PRISMA diagram flow is presented in Fig. 1 . An overview of the characteristics of all studies included in the review is provided in Annex 3.

Fig. 1.

PRISMA flow diagram for the selection of studies.

Methodological characteristics of the studies

Table 1 presents a summary of the setting, outcomes and NPIs assessed in the 34 studies.

Table 1.

Setting, outcomes assessed, and NPIs included in the studies.

Authors and outcome(s) assessed	Setting	NPIs studied
Banholzer et al. (a)24 Confirmed cases	20 countries: 15 EU countries, USA, Canada, Australia, Norway and Switzerland	(1) School closing / (2) Border closures / (3) Public event bans / (4) Gathering bans / (5) Venue closing / (6) Lockdowns prohibiting public movements without valid reason / (7) Work bans on non-essential business activities
Banholzer et al. (b)25 Confirmed cases	20 countries: 15 EU countries, USA, Canada, Australia, Norway and Switzerland	(1) School closing / (2) Border closures / (3) Public event bans / (4) Gathering bans / (5) Venue closing / (6) Lockdowns prohibiting public movements without valid reason / (7) Work bans on non-essential business activities
Bo et al.17 Reproduction number	Worldwide, 190 countries	(1) Mandatory face mask in public / (2) Isolation or quarantine (3) Social distancing / (4) Traffic restrictions
Brauner et al.30 Reproduction number	Worldwide, 41 countries: 34 European and 7 non-European	(1–3) Gatherings limited to 1000/100/10 people or less / (4–5) Some/ All but essential shops closed / (6–7) Schools or universities closed / (8) Stay-at-home orders with exemptions
Chaudhry et al.46 Confirmed cases, recovered cases, critical cases and deaths per million	Worldwide, 50 countries	(1) Travel restrictions: None/ Partial/ Complete border closure / (2) Containment measures: None/ Partial / Complete lockdown / (3) Curfew implemented (stay-at-home orders limited to specific hours)
Chernozhukov et al.39 Cases and deaths growth rate	USA, all states	(1) Stay at-home / (2) Closed nonessential businesses / (3) Closed K-12 schools / (4) Face mask mandates for employees in public facing businesses
Courtemanche et al.42 Cases growth rate	USA, 3138 counties	(1) Shelter-in-place orders / (2) Public school closing / (3) Bans on large social gatherings / (4) Closures of entertainment-related businesses
Deb et al.32 Confirmed cases and deaths	Worldwide, 129 countries	(1) School closing / (2) Workplace closing / (3) Public event cancellations / (4) Gathering restrictions / (5) Public transportation closures / (6) Stay-at-home orders / (7) Restrictions on internal movement / (8) International travel bans
Dreher et al.40 Reproduction number and case fatality rate	USA, all states	(1) Stay-at-home orders / (2) Educational facilities closure / (3) Non-essential business closure / (4) Limitations on mass gatherings
Duhon et al.21 Case growth rate	Worldwide, unclear number of territories	(1) School closing / (2) Workplace closing / (3) Cancellation of public events / (4) Restrictions on gatherings / (5) Public transit closures / (6) Stay-at-home requirements / (7) Restrictions on internal movement / (8) International travel controls
Ebrahim et al.22 Case growth rate and reproduction number	USA, 1320 counties	(1) Closure of nonessential workplaces / (2) Shelter-in-place/stay-at-home orders / (3) Enforcement of shelter-in-place or stay-at-home / (4) Size restrictions on public gatherings / (5) School closing / (6) Public transport closures / (7) Publicly available testing
Esra et al.27 Reproduction number	Worldwide, 26 countries and 34 US states	(1) Quarantine and isolation policies / (2) Limits on gatherings / (3) School closing (primary, secondary and tertiary educational institutions) / (4) Mask policies / (5) Household confinements (stay-at-home-orders, shelter-in-place orders and lockdowns)
Flaxman et al.29 Reproduction number	Europe, 11 countries	(1) Lockdown / (2) Cancel public events / (3) School closing / (4) Self-isolation / (5) Social distancing encouraged
Fountoulakis  et al.28 Death rate	Europe, 40 countries	(1) School closing / (2) Workplace closing / (3) Public events ban / (4) Gathering ban / (5) Public transport closure / (6) Lockdown implementation / (7) Domestic travel ban / (8) International travel ban
Haug et al.47 Reproduction number	Worldwide, 79 territories, 56 countries	Different categories of NPIs in their hierarchical levels (42,151 measures)
Hsiang et al.15 Case growth rate	China, South Korea, Italy, Iran, France, US	(1) Restricting travel / (2) Social distancing / (3) Quarantine and lockdown / (4) Additional policies
Hunter et al.44 Incident risk ratio of NPIs on the number of cases and deaths	Europe, 30 countries	(1) Mass gathering restrictions / (2) Initial business closure / (3) Educational facilities closed / (4) Non-essential services closed / (5) Stay-at-home order / (6) Travel severely limited - none European country
Islam et al.33 Incidence rate ratio	Worldwide, 149 countries	(1) Closures of schools / (2) Workplace / (3) Public transport / (4) Restrictions on mass gatherings and public events / (5) Restrictions on movement (stay-at-home regulations and restrictions on movements within a country)
Jalali et al.43 Case rates, mortality rates and case-fatality rates	USA, 30 most populous counties and 10 most populous counties of CA, FL, NY and TX	3 broad categories: (1) Restrictions on mass gatherings / (2) Stay-at-home orders / (3) Face mask requirements
Jüni et al.16 Epidemic growth	Worldwide, 144 territories, (not China, South Korea, Iran, Italy)	(1) School closing / (2) Restrictions of mass gatherings / (3) Measures of social distancing
Koh et al.34 Reproduction number	Worldwide, 142 countries	(1) International travel controls (including screening, quarantine and bans on international movement) / (2) Restrictions on mass gatherings (including public event bans and size restrictions on gatherings) / (3) Lockdown-type measures (including workplace closure, internal movement restrictions)
Leffler et al.35 Mortality	Worldwide, 200 countries	(1) School closing / (2) Workplace closing / (3) Cancel public events / (4) estrictions on gatherings / (5) Close public transport / (6) Stay-at-home requirements / (7) Internal movement restrictions / (8) International travel restrictions / (9) Income support / (10) Public information campaigns / (11) Testing policy / (12) Contact tracing policy / (13) Public wearing of masks
Li et al. (a)23 Reproduction number	Worldwide, 131 countries	(1) Closure of schools / (2) Closure of workplaces / (3) Public events bans / (4) Restrictions on the size of gatherings / (5) Closure of public transport / (6) Stay-at-home orders / (7) Restrictions on internal movement / (8) Restrictions on international travel
Li et al. (b)26 Case growth rate and death case growth rate	USA, all states	(1) School closing / (2) Workplace closures / (3) Public event cancellations / (4) Public information campaigns / (5) Public transport closures / (6) Stay-at-home orders / (7) International/national travel controls
Liu et al.36 Reproduction number	Worldwide, 130 countries	(1) Internal containment and closure (School and workplace closure, public event cancelation, limits on gathering sizes, public transport closure, stay-at-home requirement, internal movement restriction) / (2) International travel restrictions / (3) Economic policies / (4) Health systems policies (Public information campaign, testing policy, contact tracing)
Olney et al.31 Reproduction number	USA, all states	(1) Social distancing encouraged / (2) Schools or universities closing / (3) Public events (ban for more than 100 people)/ (4) Lockdown / (5) Self-isolating ill / (6) Sports (public event ban of more than 1000 people)
Papadopoulos et al.37 Total cases and total deaths	Worlwide, 137 countries	(1) School closing / (2) Workplace closing / (3) Cancelling of public events / (4) Restriction on gatherings / (5) Closure of public transport / (6) Stay-at-home restrictions / (7) Domestic travel restrictions / (8) International travel restrictions / (9) Public information / (10) Testing framework / (11) Contact tracing
Piovani et al.45 Mortality	OECD countries	(1) Mass gathering ban (more than 1000 people) / (2) School closures
Pozo-Martin et al.19 Case growth rate	OECD countries	(1) School closing requirements / (2) Workplace closing requirements / (3) Public events cancelling requirements / (4) Restrictions on gatherings / (5) Public transport restrictions / (6) Stay-at-home requirements / (7) Restrictions on internal travel / (8) International travel controls / (9) Public health information campaigns / (10) Mask wearing requirements / (11) Testing policy / (12) Contact tracing policy
Siedner et al.41 Case growth rate and deaths growth rate	USA, all states	(1) Social distancing measures (closures of schools, closures of workplaces, cancellations of public events, restrictions on internal movement, closures of state borders) / (2) Internal movement restrictions (shelter-in-place orders, lockdowns)
Stokes et al.49 Mortality rate	Worldwide, 130 countries	(1) School closing / (2) Workplace closing / (3) Public event cancelling / (4) Gathering restrictions / (5) Public transport closure / (6) Stay-at-home requirements / (7) Restrictions on internal movement / (8) International travel controls / (9) Public information campaigns
Wibbens et al.18 Case growth rate	40 territories: 17 countries and 23 US states	(1) Closing of schools / (2) Closing of workplaces / (3) Public event cancelling / (4) Gathering bans / (5) Public transport closure / (6) Shelter-in-place orders and home confinement / (7) Restrictions on internal movement / (8) Restrictions on international travel / (9) Public information campaigns / (10) Testing access / (11) Contact tracing
Wong et al.38 Cumulative incidence	Worldwide, 139 countries	(1) School closure / (2) Workplace closure / (3) Public event cancelation / (4) Restrictions on gathering size / (5) Public transport closure / (6) Staying at home requirements / (7) Internal movement restrictions / (8) International travel restrictions / (9) Public information campaigns
Zhang et al.20 Case growth rate	USA, all states	(1) Implementation of shutdowns / (2) Mask mandates

Study type, timeframe and geographical scope

All studies identified were ecological studies with data aggregated at population level. While most of the studies analysed country level data, some included more granular analysis at regional,^{15 , 16} or city level.¹⁷ Most of the studies were based on data from the first wave of the pandemic. Only Wibbens et al¹⁸ assessed the impact of NPIs until November 2020 and Pozo-Martin et al.¹⁹ performed a first analysis of the initial phase of the pandemic and a second from October until December 2020. Zhang et al.²⁰ analysed another relatively long study period until August, and several authors performed an analysis until July 2020.21, 22, 23 Some authors standardised the start and/or end of the study period in order to be able to compare the effectiveness of NPIs across units of analysis at similar stages of the epidemic.24, 25, 26, 27, 28, 29, 30, 31

With respect to the geographical scope, twelve studies analysed more than 65% of the world's territories.^{16 , 17 , 21 , 23 ,} 32, 33, 34, 35, 36, 37, 38 ⁴⁹ Some focused on specific areas, with nine studies analysing data from the United States, performing mostly state-level,^{20 , 26 , 31 ,} 39, 40, 41 but also county-level^{22 , 42 , 43} analyses, three studies estimating the effects of NPIs in Europe^{28 , 29 , 44} and two studies in all OECD countries.^{19 , 45} Five studies selected specific countries that belong to different world regions^{15 , 24 , 25 , 30 , 46} and three included both countries and US states.^{18 , 27 , 47}

Outcomes of interest

The authors have reported on different epidemiological parameters to assess the effectiveness of the NPIs studied. More than half of the studies reported on the reproduction number and the case growth rate, daily or weekly (detailed in Annex 3). For all of these studies, excluding Ebrahim et al.²², Li et al. (a)²³ and Liu et al.³⁶ the outcome was estimated by the authors through calculations assuming different epidemiological characteristics of SARS-CoV-2, like time until symptom onset and the distribution of the serial interval.⁴⁸ The number of confirmed cases, mortality or death rate, confirmed deaths and deaths growth rate come next as the most reported outcomes.

NPIs assessed

We analysed 16 NPIs that were consistently assessed in the studies included in the review. More than three quarters of the studies analysed the effectiveness of two NPIs: lockdowns (also called stay-at-home mandates or shelter-in-place orders) were analysed by 29 studies and school closing was analysed by 27. Half of the studies analysed in this review assessed the effectiveness of international travel or border restrictions and social gathering restrictions. Several NPIs were analysed in less than a quarter of the studies: business or venue closing, testing strategies, mask wearing requirements, social distancing, contact tracing strategies and isolation or quarantine (see Table 1).

Statistical methodology

Several authors used sophisticated and flexible methods, like Bayesian longitudinal models^{18 , 19 , 24 , 25 ,} 29, 30, 31 or event studies,^{15 , 20 , 36 , 42} while others used linear regression^{16 , 21 , 23 , 28 , 34 , 35 , 38 , 49} and simple correlation coefficients.^{37 , 43} The quality of the statistical methodology is evaluated in the risk of bias tool in Annex 2.

Risk of bias analysis

The maximum score from the tool was 18, the highest ranked study had a score of 17 and the lowest had a score of 11. The studies were grouped in three categories based on their rating: lower (with a rating of 11 or 12), intermediate (with a rating of 13–15) and higher (with a rating of 16 or 17) quality. Overall, thirty studies were considered to have to have intermediate and higher quality. The detailed quality assessment tool is provided in Annex 2.

Results of the studies

Results on the reproduction number, epidemic growth and incidence-related outcomes

A heatmap of the findings of all studies (excluding Hsiang et al.¹⁵, who compared different NPIs from different countries) studying the reproduction number, epidemic growth and incidence-related outcomes is shown in Fig. 2 . Pozo-Martin et al.¹⁹ performed independent analyses of two different timeframes, each analysis is represented separately in the heatmap.

Fig. 2.

Heatmap of the assessment of NPIs effectiveness in the studies analysing epidemic growth and incidence-related outcomes. The color grading is given according to the effectiveness ranking of each article. Darker green means higher effectiveness among the NPIs studied in the same article. Gray means no significant association with the outcome. White means the NPI was not studied. If no ranking was established, all the NPIs studied in the same article have the same shade of green. The rating provided is the result of the risk of bias analysis.

Rt = time varying Reproduction number; g = growth rate; IRR1 = incident risk ratio of NPIs on the number of cases; IRR2 = incidence rate ratio; Growth* = epidemic growth expressed as ratios of rate ratios; CI = cumulative incidence. Pozo-Martin et al. 1 and 2: first and second waves.

Most effective interventions

Overall, school closing was found to be the most effective measure: 14 out of 24 studies (58%) that analysed this NPI found an association with reduced number of cases and its implementation. It was estimated to be the most effective policy in four studies^{18 , 36 , 44 , 47} and the second most effective in four studies^{23 , 30 , 31 , 33}. Brauner et al.³⁰, estimated a mean reproduction number reduction of 39% after the closure of schools and universities. Haug et al.⁴⁷ found robust evidence of a mean reduction of 73% in the reproduction number associated with school closures.

Other NPIs that were consistently found among the most effective in reducing cases were: workplace closing, business or venue closing and public event bans. Workplace closing was associated with an improvement of the outcome in 12 out of 14 studies that analysed this NPI (86%). Among the studies that ranked the NPIs, four concluded that it was the most effective measure^{18 , 34 , 36 , 19} and three found it was the second most effective.^{23 , 26 , 38}

Regarding business and venue closing, four out of seven studies assessing its effectiveness (57%) concluded that this measure had a significant impact on the outcome, from which two studies found it the most effective intervention^{24 , 30} and another two the second most effective.^{40 , 42}

For public event bans, six out of 12 studies (50%) found that it was a predictor for the outcome. Li et al. (a)²³ found it was the most effective measure, reaching a peak effect of 25% reduction in the reproduction number 28 days after its implementation. Islam et al.³³ found that greater reduction of the incidence was always observed when public event bans were included in the combination of measures implemented. Two studies found it the second most effective control measure.^{26 , 36}

Intermediate effectiveness interventions

There are NPIs consistently found effective among the studies, which were not generally ranked as the most effective measures. These NPIs are lockdowns, movement limitations through national or international travel restrictions, social gathering bans ranging from 10 people to mass gathering bans, social distancing, public information campaigns and mask wearing requirements.

Twenty-seven studies analysed the relation between lockdown and the reproduction number, case growth rates and cases, and 18 out them (67%) found an association with their reduction. Five studies found lockdowns to be the most effective control measure.^{18 , 27 , 31 , 40 , 42} Flaxman et al.²⁹ found it to be the only measure reducing the reproduction number below one. In contrast, five studies found it to be the least effective measure studied.^{26 , 30 , 33 , 37 , 47} This disparity could be a result of the contrasting definitions of lockdown used by different authors. While most authors studied the additional impact of lockdown after the implementation of other NPIs,^{24 , 25 , 30} others estimated the effect of lockdown including several other measures.^{15 , 29 , 34} Brauner et al.³⁰ suggest, that in some countries the reproduction number may have decreased below one without enforcing lockdowns by issuing other NPIs. Li et al. (b)²⁶ studied lockdown for more than two months in the United States and concluded that it proved to be effective, but that its effect in reducing the growth rate of the number of cases decreased with time.

Both domestic and international travel restrictions have been associated with a reduction in the reproduction number, growth rate or incidence-related outcomes. Five out of 12 studies (42%) assessing domestic travel restrictions and nine out of 15 (60%) studying international travel restrictions found an association. International travel restrictions were shown to be more effective than domestic or national travel restrictions in the studies analysing both policies. Haug et al.⁴⁷ estimated that international border restrictions reduced the reproduction number by 56%, while individual movement restrictions reduced it by 42%. In contrast, Wibbens et al.¹⁸ concluded that even recommendations of avoiding movement across regions and cities were more effective than bans on all international regions and total border closures in reducing weekly growth rates.

The definition of social gathering restrictions ranged from mass gathering bans to banning gathering of less than ten people. While mass gathering bans were associated with a reduction of incidence-related outcomes in 7 out of 14 studies (50%), social gathering bans were associated in 11 out of 15 (73%). Consistently in higher and intermediate quality studies, restrictions of smaller social gatherings have been found more effective than mass gathering restrictions, Haug et al.⁴⁷, and Banholzer et al. (a)²⁴ reached the same conclusion.

While some authors consider social distancing to be a combination of certain other restrictive measures^{16 , 17 , 41} sometimes it is considered as the official encouragement by the government to keep social distance.²⁹ Flaxman et al.²⁹ evaluated the effectiveness of officially encouraging social distance and did not observe an impact on the reproduction number. However, several studies that defined social distancing as a mix of several policies found an association between these measures and a reduction in the reproduction number,¹⁷ the growth rate⁴¹ and the epidemic growth.¹⁶

Two health system measures found to be effective in reducing COVID-19 cases are public information campaigns and mask wearing requirements. Six out of eight studies (75%) analysing public information campaigns and six out of seven (86%) analysing mask wearing requirements found an association with the outcomes of interest. Several studies found public information campaigns highly effective.^{18 , 26 , 37 , 38} Wibbens et al.¹⁸ found it the most impactful measure when there was a coordinated public information campaign. Li et al. (b)²⁶ found that the growth rate reduction induced by public health information campaigns increased during the study period, reaching its peak two months after implementation. However, some high rated studies found public information campaigns to be among the least effective policies.^{36 , 47} With respect to mask wearing requirements, three intermediate and high-quality studies^{17 , 19 , 27} found it to be among the most effective measures. Chernozhukov et al.³⁹, found that the only significant measure reducing the case growth was implementing mask wearing requirements for employees in public facing businesses.

Least effective interventions

There was no consistent evidence on the effectiveness of public transport closure or of the following three health system NPIs: testing strategies, contact tracing strategies and isolation or quarantine strategies.

Only one of the 12 studies that analysed its effect found an association between public transport closures and the reproduction number, growth rate or case related outcomes.³² Neither of the six studies analysing the effect of testing policies found testing policies to improve the outcome. Pozo-Martin et al.¹⁹ found in an analysis of the measures between October and December that both testing symptomatic and asymptomatic individuals was a predictor for a higher growth rate. None of the three studies analysing contact tracing strategies found a significant association with COVID-19 cases. Two out of four studies analysing isolation or quarantine strategies found an association with the outcome. However, in these studies these interventions were among the least effective. The findings regarding testing strategies, contact tracing strategies and isolation or quarantine strategies were consistent across intermediate and high-quality studies as rated in our review.

Results on mortality and death-related outcomes

A heatmap of the findings of all studies assessing mortality and death-related outcomes is shown in Fig. 3 . Note that some studies can be found in both Figs. 2 and 3, since several authors assessed more than one outcome. Twelve studies considered mortality or death-related outcomes in their analysis: four studied mortality, three death growth rate, three studied confirmed deaths, one studied the incident risk ratio on the number of deaths and one the case fatality rate. The evidence of the comparative effectiveness of NPIs with respect to mortality-related outcomes is not clear. School closing seemed to be the most effective NPI, six out of ten studies (60%) analysing this measure found it was significantly associated with the outcome. International travel restrictions are associated with a decrease in mortality in 4 out of 7 studies (57%). Mask wearing requirements, public event bans and mass gathering bans show consistent association with the outcome in all of the studies assessing these interventions. However, mask wearing requirements and social distancing were only studied in two and one article respectively. In line with studies analysing epidemic growth and incidence-related outcomes, testing strategies and contact tracing strategies show no evidence of being associated with COVID-19 deaths.

Fig. 3.

Heatmap of the assessment of NPIs effectiveness in the studies analysing mortality and death-related outcomes. The color grading is given according to the effectiveness ranking of each article. Darker green means higher effectiveness among the NPIs studied in the same article. Gray means no significant association with the outcome. White means the NPI was not studied. If no ranking was established all the NPIs studied in the same article have the same shade of green. The rating provided is the result of the risk of bias analysis.

g = growth rate; IRR = incident risk ratio of NPIs on the number of deaths; CFR = case fatality rate.

Eight studies assessed both a death-related outcome and an incidence-related outcome, while four studies focused only in mortality related outcomes.^{28 , 35 , 45 , 49} Several studies that explored the effectiveness of NPIs in both types of outcomes found associations between them and the number of cases, reproduction number or growth rate that were not relevant for mortality.^{26 , 40 , 44 , 46} Hunter et al.⁴⁴ found that mass gathering bans were relevant for reducing cases and deaths, but business closures only for deaths. Piovani et al.⁴⁵ reached a similar conclusion regarding mass gathering bans, and also due to school closing. Both Papadopoulos et al.³⁷ and Chernozhukov et al.³⁹ found school closing to be the most effective NPI in reducing the number of deaths. In contrast, two authors found that the closure of schools decreased the number of deaths, but not the number of cases.^{37 , 39} Leffler et al.³⁵ concluded that in countries with cultural norms or government policies supporting public mask-wearing, per-capita coronavirus mortality increased on average 16.2% each week, as compared with 61.9% each week in remaining countries.

Dose-response effect of NPIs

The OxCGRT stringency index is a score for each country that provides information about the overall intensity of mostly social distancing policies implemented by the Governments in a certain moment of time.¹¹ There are contradictory results about stringency being a predictor of improved outcomes. While Leffler et al.³⁵ could not find any association between the stringency index and mortality, Deb et al.³² estimated that countries that have put in place stringent measures have reduced the number of confirmed cases and deaths by more than 200 percent relative to the absence of measures.

Different levels of business closing were studied by several authors. Hunter et al.⁴⁴ observed no additional value to closing all non-essential services in comparison to only initial business closure. In contrast, Brauner et al.³⁰ estimated that closing some high-risk businesses reduced the reproduction number by 31% while closing most nonessential businesses reduced it by 40%. Brauner et al. also found a dose-response effect for gathering bans: they estimated a 36% and 21% reduction in the reproduction number when limiting gatherings to 10 people or less, and to 100 people or less, respectively. Liu et al.³⁶ reached similar conclusions, adding that restrictions on gatherings of more than 1000 people were not effective.

Pozo-Martin et al.¹⁹ in an analysis of the first COVID-19 wave and Koh et al.³⁴ reached a similar conclusion regarding workplace closure. Recommended workplace closure or staying at home had been effective, implying that voluntary physical distancing has played an important role. However, Pozo-Martin et al.¹⁹ also observed improved outcomes, when all‑but-essential workplaces were closed. Regarding mask wearing requirements, Pozo-Martin et al.¹⁹ found that the effectiveness increased when they were mandated for all public places in all geographical areas within a country.

With respect to lockdown measures, Koh et al.³⁴ suggest that early on in the outbreak complete lockdowns may be unnecessary to control viral transmission, because partial lockdowns show to be equally effective. This finding is supported by the analysis of Papadopoulos et al.³⁷, which concluded that the maximum stringency of individual lockdown policies was not associated with reduced case numbers or mortality.

Wibbens et al.¹⁸ performed a detailed analysis of the individual OxCGRT intensity levels of NPIs in the United States and concluded, that, in general, the higher the policy intensity, the higher the relative impact on reducing the growth of infections. However, the difference in some cases might not be sufficiently relevant to upscale the level of the measure, taken the socioeconomic burden associated. They found that school closure and travel restrictions needed to be implemented at maximum stringency to reach a high impact and public information campaigns are most impactful at the lowest recorded level. Stokes et al.⁴⁹ associated stricter measures with reduced mortality.

Timeliness of implementation

Regarding the effect of time delays in NPI implementation in the incidence and incidence-related outcomes, most studies found an association between the time delay and worse outcomes. Koh et al.³⁴ concluded that all NPIs have to be implemented early to be effective. Chaudhry et al.⁴⁶ found that days to travel restrictions was positively associated with the number of cases. Both Islam et al.³³ and Jalali et al.⁴³ concluded that earlier implementation of lockdowns was associated with a greater reduction in incidence of COVID-19 and in the latter case they also found an association with early introduction of face mask requirements. Papadopoulos et al.³⁷ concluded that early timing of lockdown introduction is of greater importance than its stringency. In contrast, Pozo-Martin et al.¹⁹ did not find that a delay in the response was a predictor of epidemic growth in the OECD countries.

All the studies that analysed the effect of timeliness in mortality or mortality-related outcomes found an association between early implementation and improved outcomes.^{28 , 35 , 37 , 45 , 49} Papadopoulos et al.³⁷, concluded that early generalised school closure, early generalised workplace closure, early restriction of international travel and early public information campaigns were independently associated with reduced national COVID-19 death rate. Similarly, Leffler et al.³⁵ found an association between early international travel restrictions and a reduction of COVID-19 per-capita mortality. Piovani et al.⁴⁵ found that the early application of mass gatherings and school closures was associated with an important reduction in COVID-19 mortality. Fountoulakis  et al.²⁸ found that early implementation of public events bans was a crucial factor for reducing deaths.

Effect of number of NPIs

Some authors reported on the effects on the outcome depending of the number of NPIs implemented. Islam et al.³³ concluded that the implementation of any physical distancing intervention was associated with an overall reduction in COVID-19 incidence of 13%. Bo et al.¹⁷ and Jüni et al.¹⁶ determined that the implementation of two or more types of NPIs was more effective for containing the spread of COVID-19 than implementing only one type. Bo et al.¹⁷ also found that all NPI implementations involving social distancing were associated with a greater decrease in the reproduction number than those not involving distancing and concluded that combinations with more types of NPIs seemed to be associated with slower epidemic growth.

Discussion

Based on 34 ecological studies identified, this systematic review found that, among the 16 NPIs studied, school closure has been the most effective in reducing COVID-19 cases during the first wave of the pandemic. Workplace closures, business or venue closures and public event bans were also consistently considered among the most effective measures in reducing the number of cases. Public information campaigns and mask wearing requirements also proved to be effective in controlling the pandemic, while having less disruptive effects on the population than other NPIs. In contrast, public transport closure, testing strategies, contact tracing strategies and isolation or quarantine strategies showed no evidence of being effective in the studies assessed. Most of the studies assessing mortality were not able to estimate a comparative effectiveness of the interventions. While early implementation was consistently associated with a higher effectiveness in reducing COVID-19 cases and deaths, the stringency of the interventions was not. NPIs are effective in controlling the spread of COVID-19. An early response and a combination of specific social distancing measures are effective at reducing COVID-19 cases and deaths.

We found that the most effective NPI was school closing. This NPI has been widely used since the beginning of the pandemic due to its effectiveness against influenza outbreaks.50, 51, 52 However, school closing carries a heavy socioeconomical burden, hindering education and social interactions for children and causing additional child-care obligations for parents, linked to work absenteeism.⁵³ There are conflicting results in the literature regarding the effectiveness of this NPI in mitigating the COVID-19 pandemic since it is still not clear if SARS-CoV-2 transmission occurs differently among children.⁵⁴ Some authors have concluded that school closing is not an effective NPI and that COVID-19 control can be reached without this measure.55, 56, 57 Viner et al.⁵⁸ performed a systematic review on the effects of school closing on the spread of respiratory diseases and included four modeling studies at the beginning of the COVID-19 pandemic. They found that the effect of school closures was comparatively lower than other measures. In a recent systematic review of observational studies analysing school closing and reopening Walsh et al.⁵⁹ found that half of the studies with lower risk of bias concluded that school closing reduced community transmission while half of the studies found no effect. Other analyses60, 61, 62, 63, 64, 65 suggest that school closing may have been a more important factor during the first wave of the COVID-19 epidemic than initially thought. As the epidemiological situation has improved, reopening schools has become imperative. Walsh et al.⁵⁹ found that there is no increase in community transmission of COVID-19 after reopening schools in a low transmission context with appropriate mitigation measures, as it was observed in Norway and Denmark.⁶⁶ However, in countries like Germany, that reopened schools when community transmission was still high, it can lead to an increase in the growth rate.⁶⁶ In the United Kingdom schools have been closed twice, first during the first round of restrictions starting March 2020, reopening between June and August 2020. Second, schools closed in December 2020 or January 2021 and reopened in March and April 2021.⁶⁷ Mitigation measures have been established for schools to reopen across the United Kingdom, including mask wearing and testing,⁶⁸ and after the reopening in both instances cases still went down.⁶⁹ Indeed, several studies worldwide have found that mitigation measures allow to reopen schools safely.70, 71, 72, 73 Lessler et al.⁷⁰ found that even implementing low levels of in-school mitigation measures COVID-19 outcomes were reduced. On average, each measure implemented was associated with a 9% decrease in the odds of COVID-19-like illness.⁷⁰ In conclusion, a cautious approach for reopening should be adapted to each context, with specific mitigation measures, stepwise opening and monitoring the effects of reopening for in-school and community transmission.

We found other social distancing NPIs to be consistently effective in controlling the COVID-19 pandemic: workplace closing, business and venue closing and public event bans. As reflected in the results of this review, there are differences in the effectiveness of closing all businesses or targeted ones. Further, business and venue closures affect the economy disproportionally⁷⁴ and therefore, careful consideration for implementing these measures needs to be taken. A tailored approach is necessary for each context as workplace closures can pose different social or psychological problems to workers.⁷⁵ Nonetheless, strategies can be adopted to reduce their potential negative effects.⁷⁵ Another widely used measure is the ban of public events. At the beginning of the pandemic several public events were considered “super spreader events” in China⁷⁶ and in other countries.^{77 , 78} A super spreader event describes a situation when only one or some positive cases infect many people. Sun et al.⁷⁹ estimated that 15% of the people accounted for 80% of the infections in the Hunan province in China in early 2020. Estimations for local transmission in Hong Kong⁸⁰ and other locations⁸¹ back up these findings. The existence of super spreaders is considered to be a common characteristic of coronaviruses, and it is related with several factors, like prolonged indoor gatherings with poor ventilation.⁸² However, the exact reasons why some individuals are able to infect many people and other individuals only a few remain unclear. Note that super-spreader events are not avoidable only via public event bans, but also through venue closures and gathering restrictions, which were found to be effective NPIs in our analysis. In a mathematical model developed by Chang et al.⁸³ using mobility data, they observed that restaurants, cafes and gyms could account for most COVID-19 infections in US cities, with 80% of the predicted transmission being linked to 10% of the locations.

Our results regarding several health system interventions need to be cautiously interpreted. For instance, testing and contact tracing policies and isolation or quarantining, which are standard public health activities, were not found to be effective in controlling the COVID-19 pandemic. This can be attributed to several factors. First, if the outcome of the study is related to the number of cases, when case detection improves through more efficient testing and tracing the case number reported will rise, without representing a real rise of cases. Second, countries that have implemented successful control strategies strongly relying on these interventions, like China,^{84 , 85} South Korea,⁸⁶ Singapore⁸⁷ and New Zealand⁸⁸ have not been specifically addressed among the studies included in this review. In our analysis, testing and contact tracing policies and isolation or quarantining were only assessed in few studies among those that met the inclusion criteria. However, the findings were consistent across intermediate and high-quality studies. In the existing literature, testing and contact tracing strategies, followed by quarantining or isolation, have been considered essential in controlling COVID-19 spread. It has been observed that higher testing volume or testing coverage are correlated with improved control of the pandemic.^{19 ,} 89, 90, 91 However, in this review we assessed the impact of different testing strategies, meaning which individuals get tested and how, not the number of tests performed. Several studies have highlighted the importance of a comprehensive test, trace and quarantine approach in different contexts.^{92 , 93} Hellewell et al.⁹⁴ found through a modeling study that the pandemic could be controlled under certain testing and tracing strategies and concluded that the most important factor in determining whether an outbreak was controllable or not was the delay between symptom onset and isolation. A recent systematic review comparing mass testing and contact tracing with conventional test and trace strategies concluded that mass testing could be more effective in controlling the pandemic.⁹⁵

Interestingly, another effective health system measure is the adoption of public information campaigns by the governments. Although this intervention is not among the most effective measures in our analysis, it is consistently associated with a reduction of COVID-19 cases. Mask wearing requirements were also consistently effective in reducing COVID-19 cases in our review. The use of masks can be associated to individual discomfort, but it does not present such an important disruption for daily activities as most of the other measures analysed in this review. Much research has been produced around the use of face masks and its effectiveness, mostly confirming its positive impact in controlling the virus spread.^{96 , 97} Through mask wearing requirements workplaces, schools and businesses have been allowed to open. Considering the disruptive effects of most NPIs and their high societal and economical cost, the implementation of effective public information campaigns and mask requirements can present large benefits with less efforts and socioeconomical consequences than other NPIs.

Similarly, due to the socio-economic burden associated with scaling up some NPIs, the specific context needs to be considered. As reflected in the results of this review, upscaling the level of some measures does not always imply improved outcomes. Besides, the difference in the reduction of cases or deaths might not be sufficiently relevant to implement more stringent measures, considering its socioeconomical impact.

Our findings rely mostly on the analysis of studies published based on data from the first wave of the COVID-19 pandemic. However, several studies analysing the effectiveness of NPIs during the second wave have been released as preprints recently. Sharma et al.⁹⁸ studied the implementation effects of 17 NPIs in 114 subnational areas from 7 European countries. They found that the combined effect of general NPI implementation was smaller during the second wave, which can be attributed, among other factors, to the influence of maintained individual protective behaviours after the first wave. In line with the results of our review, they concluded that in the second wave closing specific businesses was highly effective, together with strict small gathering restrictions. In contrast, they estimate that school closing was not as effective during the second wave in comparison to the first, which could be linked to the control measures adopted in schools after reopening. In another preprint, Ge et al.⁹⁹ analysed the effect of NPIs from the first wave until March 2021 in 133 countries, assessing vaccine rollouts as well. Consistent with Sharma et al.⁹⁸, Ge et al.⁹⁹ observed differences in the effectiveness of NPIs during the different waves; they found that school closing was the most effective measure during the first wave, but not among the most relevant in the second. Gathering restrictions and facial covering requirements were consistently considered effective among waves, whereas international travel restrictions played a more important role in the control of the second wave. Regarding vaccine rollout, they considered that vaccination was increasingly contributing to the pandemic control, despite its effect having a significantly lower impact than the NPIs by the time of the study.

As in every evidence review, the comparability of the studies analysed depends on their design and methodological heterogeneity. First, different outcomes were studied to assess the effectiveness of NPIs among the studies. These were mainly the reproduction number, the growth rate, the number of cases and the number of deaths. Mortality data gives only information about severe cases, but is less influenced by testing strategies and testing capacity than case counts.^{100 , 101} Testing and contact tracing capacity has been an essential constraint during the pandemic, especially at the beginning. Growth rate is more easily calculated than the reproduction number and avoids many inferential difficulties in estimating the latter.¹⁰¹ For calculating the reproduction number several epidemiological assumptions, like time until symptom onset, time until death or serial interval, need to be estimated.^{100 , 101} However, the reproduction number provides more information than the growth rate on the impact of control measures given the non-linear epidemic curve of COVID-19.¹⁰¹ If the proportion of cases that are unreported remains constant throughout the study time, estimates of the reproduction number are unaffected by underreporting.⁴⁸

Second, the number of NPIs analysed and their definitions differ among the studies included in our review. Some homogeneity is expected for studies with the same NPIs data source, like the 14 studies using the OxCGRT dataset (details in Annex 3). However, there are some differing definitions of specific NPIs among the studies. For instance, school closure can include or exclude secondary schools and/or universities, the amount of people assessed for social gathering restrictions varies and there are differences in the definition of lockdown, business closure or social distancing requirements. Furthermore, some authors consider the onset of a policy when it is officially recommended and others when it is enforced. Several studies consider both options by analysing different stringency levels instead of applying a binary approach.

There are some limitations in the body of evidence of this review. First, all the studies use retrospective and observational data to draw inferences about the effectiveness of NPIs. Conclusions from these studies are limited to the specific time and places studied and may be affected by confounding effects from unobserved factors. However, the broad geographical scope of the studies and the very different methodological approaches used to answer the study questions increase the robustness of our findings. Second, only studies that established a direct link between the implementation of NPIs and an outcome were included in the review. Therefore, it is assumed by all studies that the effect on the outcome depends only on the implementation of the measures. Nonetheless, individual behavior, even before the implementation of measures,^{39 , 102} and the compliance with the NPIs^{103 , 104} have played an essential role in controlling the pandemic. Third, all studies included in this review assess the effectiveness of NPIs during the first wave of the pandemic. This improves comparability among the studies: during the first months of the pandemic people were still adapting to protective behaviours, almost all the population was susceptible to the virus and no vaccine immunization had started, so that the effect of NPIs implementation could be more directly linked to the outcomes. However, these studies are less relevant to understand the effect of “controlled openings”, for instance businesses opening with improved hygiene conditions, adapted workplaces and schools with social distancing protocols and openings relying on testing. The worldwide availably of personal protective equipment, hospital equipment and testing material has also remarkably improved since the beginning of the pandemic. Finally, there is an overrepresentation of high-income countries in the region-targeted studies, mainly the USA and Europe, but also most OECD countries. There are neither studies focused on low- or middle-income countries nor on Asia or Oceania, even though some of these countries are included in the worldwide studies. Comparing measures within Europe and the USA leads to analysing territories with more similarities in pandemic control approaches. Many Asian countries have had recent experience with pandemics before COVID-19^{105 , 106} and, as stated before, some of them relied strongly on health system interventions. Regarding low- and middle-income countries, fewer economic resources translate to less surveillance and testing capacity, which increases underreporting and hinders the assessment of effectiveness of NPIs. However, in the first pandemic wave many Sub-Saharan African countries implemented social distancing measures before the first detected case,³⁶ which may have played a role in the low burden of disease encountered in the region during that time.¹⁰⁷ Nonetheless, in Africa, the second wave has been more severe, and despite this situation the stringency of the measures implemented in the continent is decreasing.¹⁰⁷ The knowledge of NPIs effectiveness for specific regions should not only be limited to high-income counties in order to allow policy makers worldwide to make tailored decisions.

This systematic review has certain limitations. First, we did not include in our review the evidence coming from mathematical studies simulating the impact of NPIs on epidemic control. While they generate high quality evidence, these studies rely on assumptions about the type and intensity of the NPIs being implemented rather than on the actual policy implementation. In contrast, this review provides information from data-driven studies that have explored the real-life impact of NPI implementation. Second, due to the novelty of the topic and the urgency to answer the study question we included preprints in the review. Even though these studies have not yet undergone peer review, a risk of bias tool was used to assess their quality. All of the preprints included were considered to have sufficient quality and add relevant evidence to the review. Regarding the quality assessment, there is no consolidated risk of bias tool for evaluating ecological studies. Nonetheless, we used a published tool that has been validated in several systematic reviews of ecological studies before.^{13 , 14}

To the best of our knowledge, at the time of writing there is no other review of empirical studies assessing the comparative effectiveness of more than two NPIs against COVID-19. We found that school closing, followed by workplace closing, business and venue closing and public event bans were the most effective NPIs in controlling the spread of COVID-19. Public information campaigns and mask wearing requirements, less disruptive to the population than other NPIs, were also effective measures. An early response and a combination of specific social distancing measures are effective at reducing COVID-19 cases and deaths. Since scientific knowledge, individual behavior and resources keep on changing and adapting throughout the pandemic, more research needs to be targeted to understand changes in the effectiveness of NPIs and whether controlled openings and lifting of restrictions are compatible with epidemic control.

Declaration of Competing Interest

None.