Do school closures halt SARS-CoV-2 transmission? The centrality of the question, contrast, and context

One-sentence summary

In shaping school-related policy for the COVID-19 pandemic, a useful body of evidence can only be built if we ask the right questions regarding appropriate causal contrasts in relevant settings

Policy questions such as how best to re-open schools for in-person learning during a pandemic are incredibly important - but also incredibly hard to answer in an evidence-based fashion. As with many other policy decisions during the COVID-19 pandemic, school re-opening policies were generally crafted with minimal direct evidence. Recently, however, an increasing number of empirical studies are shaping an evidence base about the risks and benefits of in-person schooling. For example, Ertem et al. (2021) use data from the United States to examine the effects of in-person schooling in selected districts on SARS-CoV-2 cases in surrounding counties. In a similar vein, Fukumoto et al. (2021) provide data comparing SARS-CoV-2 cases in Japanese municipalities with schools that opened for in-person learning to similar municipalities that kept schools closed. As school policymakers must now weigh multiple studies on pandemic-related closure policies (often with apparently conflicting results), it is important to keep three considerations front-and-center: the causal question being asked, the treatment conditions being contrasted, and the context.

For policy decisions we are almost always interested in a causal question - that is, one that compares outcomes under two different potential states of the world. In one, a (well-defined) group experiences the “treatment” of interest (for example, re-opening schools for in-person learning); in the other, it experiences a “comparison” condition (for example, continued virtual learning). This comparison immediately raises the “fundamental problem of causal inference” -- that one can only observe outcomes under one condition, while the other is unobserved, or “counterfactual” (Holland, 1986). Thus, we are forced to use data from different groups to estimate differences that would be seen in the same group under different conditions (a “causal contrast”). Appropriate causal inference therefore requires strong study designs -- such as randomization, difference-in-differences/event study approaches (as in Ertem et al., 2021), and/or well-selected comparison groups (as in Fukumoto et al., 2021) -- to estimate the expected difference in outcomes for a given population under different policy choices.

While others have appropriately highlighted the importance of study design in answering pandemic-related causal policy questions (e.g., Bonvini et al., 2021; Goodman-Bacon and Marcus, 2020; Haber et al., 2021), we argue that policymakers should “keep it simple.” Specifically, most causally-focused studies can be evaluated in terms of their question, contrast (comparison), and context. By asking whether these three components of a given study seem reasonable - and the degree to which they apply to a current decision - policymakers without extensive methodological expertise can make a rapid assessment as to the relevance of a particular study.

Take, for example, a county school board evaluating the results of Ertem et al. (2021) in deciding whether to restrict in-person learning in the face of a new pandemic wave. This study speaks to county-level decisions to open or close in-person learning - not, for example, decisions at the state or national level. The contrast studied would only be directly applicable to an “all-or-nothing” closure decision, as few counties adopted an approach of keeping elementary schools open but middle and high schools closed. And in terms of context, different results were seen in the South than in other regions - and results in the United States might or might not be transportable to other countries. But by focusing on the question, contrast, and context, non-expert decision-makers could reasonably assess the relevance of this study to their policy decision. We should encourage this sort of thinking - and make it more accessible by highlighting these three elements in any analysis that seeks to estimate a policy-relevant causal effect.

Sadly there is often a disconnect between the questions, contrasts, and contexts addressed in research studies, and those that policymakers must consider. Regarding the question being asked, Ertem et al. (2021) and Fukumoto et al. (2021) both consider area-level policy decisions; other work on in-person schooling (e.g., Lessler et al., 2021) has focused on individual household behaviors. But these may not be the questions that local policymakers are being forced to answer; as highlighted by Orr et al. (2019), even randomized trials in schools are not always immediately relevant for local decision making. In terms of contrasts, some studies have compared “school reopening” to “school closure”overall, while others have aimed to estimate the effects of specific mitigation strategies. But rarely are the strategies formally studied the only policy options on the table. For example, Viner and Koirala (2021) compared daily testing versus isolation of contacts - but many school systems might be interested in less frequent testing, or different strategies for children versus staff members. With respect to context, analyses can - and should - evaluate differences in estimated effects across contexts, but these explorations are often limited by the available data. For example, Ertem et al. (2021) highlight interesting variation in estimated effects of school closures across regions of the United States, but also note an inability to differentiate between potential explanations for these differences, such as different mitigation strategies in place, weather-related differences, or differences in underlying incidence. And as a final consideration, all research studies use data from the past, whereas policy decisions must be made in the present. These challenges highlight the importance of performing research that is as close as possible in question, contrast, and context to actual policy decisions that are being considered. If these diverge too greatly, policymakers will default to decision-making in the absence of evidence, thus invalidating the considerable effort being made to bring an evidence base to bear in this process. Rarely are blanket conclusions such as “reopening schools does not fuel SARS-CoV-2 transmission” appropriate.

It is therefore critical that, in informing evidence-based decision-making, we maintain focus on the causal questions, contrasts, and contexts - while making use of the most appropriate data and study designs available. In the social sciences the UTOSTi (units, treatment, outcomes, settings, and times) framework (Cook, 2014) has helped articulate some of these considerations; we need a similarly simple guide for scientists and decision-makers asking policy-relevant questions regarding the COVID-19 pandemic. No one study will be relevant to all policy questions; we therefore must urgently build a diverse evidence base that mirrors the causal questions, contrasts, and contexts most likely to be encountered - and communicate those results to decision-makers in real-time, using language that can be broadly understood.