Table 2.

Selected examples of quasi-experimental studies evaluating real-world interventions to prevent COVID-19.

Policy/intervention	Example	Strengths	Weaknesses	Ways to strengthen
Pre/Post
School closures (Mandate by the Qatari government)	•Compared rate of positive tests for respiratory viruses other than SARS-CoV-2 in a pediatric emergency department before and after school closures in Qatar (1).	•Specified lag period for influenza A. •Compared to trends in 2019 to rule out that seasonal variations could explain the results.	•No control group. •Only captures children who were sick enough to go the emergency room (problematic, for example, if proportion of children with influenza A whose parents take them to the hospital changed over time). •Depending on the hospital catchment area, may be unclear who is in the target population the study sample represents. •Unclear what other interventions or policies to reduce spread of COVID-19 were enacted during the study.	•Comparison group would improve validity. •Consider stratification on relevant characteristics (i.e. age group).
Interrupted Time Series: Without Comparison Groups
Physical distancing (Closures of schools, workplaces, and public transport, restrictions on mass gatherings/public events, and restrictions on movements [lockdowns])	•Assessed incidence of COVID-19 before and after implementation of physical distancing interventions in 149 countries or regions, synthesized using meta-analysis (4).	•Compared effect of five physical distancing interventions overall and in smaller subsets of policies to attempt to determine the most effective combination and sequence. •Specified lag period a priori. •Restricted post-intervention period to address temporal concerns and reduce bias given limited pre-intervention time period. •Allowed for country-level variation using random effects models in random effects meta-analysis to synthesize effect estimates. •Assessed and controlled for country-level characteristics.	•No control group that was not subjected to at least one intervention.	•Comparison of “similar” clusters of countries (i.e., East African nations, Scandinavian nations) could improve analyses & interpretation.
Mask mandate (Universal mask wearing required by health system for healthcare workers and patients)	•Compared SARS-CoV-2 infection rate among healthcare workers before and after implementing universal masking in one health care system in the US (5).	•Allowed for non-linear functional form of SARS-CoV-2 positivity rate.	•Testing was implemented for healthcare workers, but didn't fully account for lags in development of symptoms after implementation of policy in their division of time. •Didn't account for statewide trends (e.g., the reduction observed could be due to other policies outside the healthcare system). •External validity is a concern—healthcare workers not generalizable to other high-risk exposure settings (e.g., food service sector jobs).	•Add comparison group. •Would benefit from statistical adjustment for other interventions external and internal to the hospital. •Analyzing trends during the implementation period could assist with assessing changes in slopes/trends.
Social distancing measures (closures of schools, closures of workplaces, cancellations of public events, restrictions on internal movement, and closures of state borders)	•Estimated change in COVID-19 case growth and mortality before and after implementation of first statewide social distancing measures (2).	•Specified an event-study design as a robustness check. •Conducted sensitivity analyses with multiple incubation periods and to address weekly periodicity.	•The type of the first social distancing measure may have differed across states. •It is not possible to identify which policy was most effective. •Biased if amount of testing (and therefore identification of cases) differed before and after intervention.	•Exploration of how lifting of policies, as compared to those who kept policies (i.e., duration of intervention), could improve interpretation. •Imbalance of time between the pre- (17 days) and post-periods (25 days); post-period is longer than pre-period.
School closures (State government mandates)	•Assessed whether school closures impacted incidence of COVID-19 at the beginning of the pandemic in the US (3).	•Included other non-school related policies (e.g., stay at home orders) in models. •Clear justification for lag period and conducted sensitivity analyses with multiple lag periods. •Adjusted in models for important covariates, such as testing rates, urban density, comorbidities, and age. •Included interaction effects between school closure & covariates.	•No control group. •Median time from school closure to last enacted other intervention was 5 days in states in highest quartile of COVID-19 incidence at time of school closure and 12 days in lowest quartile of incidence—may be difficult to separate out effects of other interventions, despite controlling for them.	•Localized nature of policies could provide advantage for cluster ITS comparisons, as compared to state-level data used in the study. •States implemented other interventions at the same time as or shortly after school closures, making it difficult to completely isolate the effect of school closure, despite controlling for other interventions.
Interrupted Time Series: Integrating Comparison Groups
Stay-at-home orders (State government mandates)	•Compared COVID-19 cases in border counties in Illinois (where a stay-at-home order was issued) to border counties in Iowa (where such an order was not issued) (6).	•Comparison of border counties potentially less likely to be biased than comparison of larger geographic area. •Sensitivity analyses to account for differences in timing of closing schools/non-essential businesses and to assess whether there were differential trends by population density and poverty rates.	•Only one pre-period, as compared to six post-periods.	•Inclusion of analyses of sequencing of orders in Iowa could strengthen analysis. •Control for county-level COVID-19 testing trends.
Social distancing measures (Bans on large social gatherings; school closures; closures of entertainment venues, gyms, bars, and restaurant dining areas; and shelter-in-place orders)	•Assessed effect of social distancing measures on measures of growth rate of confirmed COVID-19 cases in US counties using an event study design (7).	•Event study design (including fixed effects for county and time) allowed testing of parallel trends assumption in pre-policy period. •Tried to separate out effects of different policies. •Multiple robustness checks.	•Relying on administrative boundaries such as counties may not reflect how people live their lives (e.g. working across county lines), making it more difficult to interpret findings. •Longer post-period, as compared to pre-period.	•Could have used localized data to make comparisons over time, comparing similar states (clusters) with more or less restrictive orders. This is particularly important given that controlling for number of tests was done at the state-level, not locally. •Extension of study period after April 27, when orders were being lifted could have provided additional evidence of changes. Particularly of concern given that April 7th was when 95% of the U.S. population was covered by shelter-in-place orders.
				•Inclusion of a longer pre-intervention period would improve the study; could have used excess mortality as a marker of COVID-19 cases. •Could have used state politics as a covariate, which influences policy decision making.
Face mask mandates (State government policies to wear face masks or covers in public)	•Assessed effect of state government mandates for face mask use on changes in daily US county-level COVID-19 growth rates using an event study design (8).	•Event study design allowed testing of parallel trends assumption in pre-policy period. •Compared state-wide face mask mandates and employee only mandates. •Controlled for other policies implemented (e.g., social distancing policies) and state-level COVID-19 tests, including growth rate. •Adjusted for other state characteristics (e.g., population density). •Multiple robustness checks.	•Some states did not have state-wide mandates, but counties within them enacted mandates. •Few data points available pre-intervention.	•Local-level variation in adherence to mandates could alter results, comparison of county adherence measures (e.g., fines) could strengthen analyses.