Lack of correlation between school mask mandates and paediatric COVID-19 cases in a large cohort

Abstract

Objectives

To expand upon an observational study published by the Centers for Disease Control (CDC) showing an association between school mask mandates and lower pediatric COVID-19 cases. We examine whether this association persists in a larger, nationally representative dataset over a longer period.

Method

We replicated the CDC study and extended it to more districts and a longer period, employing seven times as much data. We examined the relationship between mask mandates and per-capita pediatric cases, using multiple regression to control for observed differences.

Results

We successfully replicated the original result using 565 counties; non-masking counties had around 30 additional daily cases per 100,000 children after two weeks of schools reopening. However, after nine weeks, cases per 100,000 were 18.3 in counties with mandates compared to 15.8 in those without them (p = 0.12). In a larger sample of 1832 counties, between weeks 2 and 9, cases per 100,000 fell by 38.2 and 37.9 in counties with and without mask requirements, respectively (p = 0.93).

Conclusions

The association between school mask mandates and cases did not persist in the extended sample. Observational studies of interventions are prone to multiple biases and provide insufficient evidence for recommending mask mandates.

Introduction

There is ongoing scientific debate around mask requirements in schools in the United States and other countries during the COVID-19 pandemic. To date, there have been no randomized controlled trials of mask requirements in children. All analyses of the effectiveness of school mask mandates have relied on observational studies. Some of these studies report a negative association between mask wearing and case rates, while others fail to identify any association.

Studies of this subject face the challenge of controlling for fundamental differences between districts and communities that choose to require or not require masking in schools. For example, a study released by the U.S. Centers for Disease Control (CDC) from Arizona found that mask requirements were associated with a reduction in COVID-19 outbreaks whilst another multi-district U.S. study reported lower in-school transmission associated with universal masking^{1 , 2}. It has, however, been difficult to rule out the possibility that these associations were a result of confounding variables rather than an effect of the masks themselves, as some authors have suggested^{3 , 4}.

Importantly, too, the findings of these studies were inconsistent with more rigorously designed observational studies5, 6, 7 including a study using a regression discontinuity design which reported no significant effect of masking on SARS-CoV-2 transmission in Catalonian schools⁸. Furthermore, one randomized study in adults found no impact of community cloth masking and only a marginal impact of surgical masking, though only in those over 50 years of age⁹. Another randomized study in adults failed to find a 50% reduction in acquisition of SARS-CoV-2 by the wearer of surgical cloth masks in community settings and was not powered to find smaller reductions¹⁰. The scientific literature prior to the COVID-19 pandemic also found low certainty evidence on the effectiveness of masks in various settings¹¹.

Considering millions of schoolchildren have been required to comply with mask mandates since the start of the pandemic and may be required to do so again in the future, it is incumbent upon society to determine if there is any effectiveness of masking children in schools against COVID-19 or other respiratory illnesses and, if so, that the benefits outweigh the harms.

Our study has two goals: First, we replicate and then extend, a highly cited observational study by the CDC on school mask mandates by Budzyn et al.¹². The second goal is to identify possible sources of statistical bias in observational studies, particularly those that are limited in population size, diversity, and duration¹³. This is important because studies identifying rather than rejecting an association are more likely to be published¹⁴ and consequently inform public health interventions despite a potential lack of true efficacy.

In their study, Budzyn et al. found that, following school openings in the Summer of 2021, pediatric COVID-19 case rates increased faster in U.S. counties that did not have mask mandates in schools, compared to those that did. While the results do not show a causal relationship, the study has been cited by the CDC in its policy recommendations and in the news media as evidence that mask mandates in schools can lower community spread.1 Our study replicates the same analysis and then extends it, using a larger sample of school districts and a longer study duration.

Methods

We follow the methodology from Budzyn et al.¹² as closely as possible. Data on pediatric and adult COVID-19 case rates, by county, were obtained from the October 25 release of the CDC's Restricted Case Dataset. Data on school enrollments and mask policies were obtained from the data company MCH, the same source used by Budzyn et al. County level demographic variables and school district to county mappings were obtained from the U.S. Census Bureau.

Following Budzyn et al., counties that met the following criteria were selected for the analysis: (1) a valid school start date and a known mask requirement for at least one school district in the county, (2) in districts that have made mask policies available, such policies apply uniformly to all students (counties containing districts with conflicting mask policies were discarded), (3) at least three weeks of data were available with seven full days of case data since the start of the 2021—22 school year (the median school start date across districts within a county was used).

Using September 4, 2021 as the cut-off, as chosen by Budzyn et al., these criteria resulted in the inclusion of 565 counties. Using the more recent data release of November 30, 2021 resulted in a larger sample of 1832 counties. This study uses the smaller sample to compare with Budzyn et al, and the larger sample to evaluate robustness.

As in Budzyn et al., we conducted a multiple linear regression of per-capita pediatric case rates for these 1832 counties, including the following controls: median age, race and ethnicity, and population density; pediatric COVID-19 vaccination rates; adult transmission rates in the corresponding county and week; percentage uninsured and percentage living in poverty; social vulnerability index (SVI) score; Covid-19 community vulnerability index (CCVI) score; and fixed-effects for each week after school opening.

Demographic variables were obtained from the U.S. Census Bureau. The SVI score was obtained from the CDC and the CCVI score from Surgo Ventures. Community transmission is defined as the per-capita rate of adult cases in the county in the corresponding week. The pediatric vaccination rate is defined as the fraction of children in the county ages 12—18 who received two doses of a Covid-19 vaccine. Per-capita pediatric case rates are defined as the weekly sum of confirmed cases in the 0—19 age group that was reported to the CDC for the corresponding county, divided by the county population in that age group.

Following the empirical analysis, we use statistical methods to reconcile the difference between our results and those of Budzyn et al., We note various biases that can arise when using observational data on jurisdictions that choose whether to adopt particular public health recommendations. We explore potential cross-sectional and temporal biases that can affect observational studies of this nature.

Results

We examine the relationship between school mask requirements and pediatric cases of COVID-19, replicating the analysis of methods of Budzyn et al., as closely as possible. We identified 565 counties that met the criteria, compared to 520 counties in the original study. The discrepancy occurred for at least two reasons. First, MCH obtains data on school start dates and mask policies from phone surveys of school districts and updates this information frequently. This study uses MCH data that were current as of October 15, 2021. Budzyn et al., used information from earlier, which was likely to have been different as school district information changes regularly. Second, Budzyn et al., selection criteria for counties is ambiguous about districts that cross county lines.2

Fig. 1 displays results from 565 counties that fit the criteria used in the original study and analyzes data from three weeks prior to schools opening to nine weeks following opening. After two weeks of schools being open, which was the endpoint for Budzyn et al., the results align with the findings of the original study: non-masking counties had, on average, around 30 additional daily pediatric cases per 100,000 children, compared to counties with mask mandates.

Fig. 1.

Pediatric COVID-19 case rates and school mask requirements: 565 counties.

Thus, during August—September 2021, mask mandates were associated with lower pediatric cases of Sars-CoV-2 in districts where schools started by August 15. However, Fig. 1 also shows that cases quickly declined in later weeks and did so faster in counties without mask mandates. In fact, the Budzyn et al., study ended at exactly the peak of school case numbers for this sample of counties. By the end of the ninth week after reopening average daily pediatric case rates in counties without mask mandates were 15.8 per 100,000 while counties with mandates averaged 18.3 per 100,000; the difference was not statistically significant (p = 0.12).

Fig. 2 shows a larger sample of 1832 counties, including counties which were not included in Budzyn et al., analysis due to later school district start dates. Note that there are 3142 counties in the United States, but many cannot be classified with regard to school mask policies, due to either insufficient data from MCH or the specific criteria used by Budzyn et al.

Fig. 2.

Pediatric COVID-19 case rates and school mask requirements: 1832 counties.

The larger sample of counties enables us to employ a much broader geographical representation of districts, in addition to increasing the population size of the study. This is important for full geographic representation as southern states are more likely to have early start dates. The figure shows the lack of a clear relationship between mask mandates and pediatric cases. Counties that required masks in schools saw slightly larger increases in cases in the weeks immediately before and after school opening, but by the second week after the start of school there was no statistical difference. At the end of week 9, cases per 100,000 capita were 21.8 in counties without mask mandates and 24.3 in counties with mandates; the difference was not statistically significant (p = 0.057).

Fig. 2 also shows that the change in case rates, across various weeks, was virtually identical for counties with and without school mask mandates. From three weeks prior to the start of school to two weeks after school opening, average daily cases rose by 39.0 per 100,000 in counties without mask requirements, and by 40.2 per 100,000 in counties with mask mandates. The difference was not statistically significant (p = 0.74). Similarly, between weeks 2 and 9, cases per 100,000 fell by 38.2 and 37.9 in counties with and without mask requirements, respectively. This difference was also not statistically significant (p = 0.93).

Next, we estimated a multiple regression of pediatric case rates controlling for observable differences across counties (Table 1 ). The first column of that Table uses the same specification as Budzyn et al. and confirms the lack of association between school mask mandates and case rates: after accounting for covariates, pediatric cases in counties with mask mandates were slightly higher, by 1.27 weekly cases per 100,000, though the effect was not statistically significant (p = 0.058).

Table 1.

Regression of pediatric COVID-19 case rates per 100 K.

	From Budzyn et al (1)	From Budzyn et al minus CCVI (2)
School Mask Requirement	1.279	3.507
	(0.058)	(0.000)
Adult Cases per 100K	1.194	1.196
	(0.000)	(0.000)
Percent Uninsured	-0.558	-0.353
	(0.000)	(0.000)
Percent in Poverty	0.531	0.312
	(0.000)	(0.000)
Population Density	-0.001	0.001
	(0.162)	(0.329)
Social Vulnerability Index	-13.558	8.911
	(0.000)	(0.000)
Community Vulnerability Index	27.154
	(0.000)
Percent Non-Hispanic White	-0.892	-1.836
	(0.667)	(0.380)
Median Age	0.768	0.738
	(0.000)	(0.000)
Pediatric Vaccination Rate	1.320	-1.352
	(0.546)	(0.540)
Intercept	-40.113	-35.973
	(0.000)	(0.000)
R2	0.673	0.666
Observations	12824	12824

Notes: Values in the table are coefficient estimates obtained from a regression of county level reported per-capita pediatric cases of Sars-Cov-2 on the control variables listed in the table. p-values are in parentheses. Regressions also include week fixed effects, that are not reported. The specification in Column 1 follows Budzyn et al. The specification in Column2 omits the CCVI index due to its high correlation with the Social Vulnerability Index. See text for sample construction and other details.

In column 2 of Table 1, we show the effect of removing the COVID-19 community vulnerability index (CCVI). The CCVI is derived from, and is highly correlated with, the social vulnerability index. Omitting this variable results in a significant positive association (p < 0.0001) between school mask requirements and pediatric case rates (see Supplementary Appendix for additional details).

Discussion

The study by Budzyn et al. examined 520 U.S. counties and found that counties without mask requirements in school experienced larger increases in pediatric COVID-19 case rates following the start of school compared with those that had school mask requirements. Specifically, the increase in pediatric case rates per 100,000 children starting from one week prior to school opening until one week afterwards was 16.32 for the former group and 34.85 for the latter.

We successfully replicated the main result of the original study by Budzyn et al. Further analysis, however, shows that the results do not hold in a larger sample of school districts, or even in the original sample of districts when extended to a longer time interval. Specifically, when we extend the sample to districts that began later in Autumn 2021, which encompasses a much larger portion of the country, we find no association between mask requirements and pediatric cases.

Thus, using the same methods and sample construction criteria as Budzyn et al., but a larger sample size and expanded time frame for analysis, we fail to detect a significant association between school mask mandates and pediatric COVID-19 cases. The discrepancy between our findings and those of Budzyn et al. is likely attributable to the inclusion of more counties, a larger geographic area and extension of the study over a longer time period. By ending the analysis on September 4, 2021, Budzyn et al. excluded counties with a median school start date later than August 14, 2021. According to the MCH data, this heavily over-samples regions that open schools by mid-August including Florida, Georgia, Kentucky and other southern states. The original study would not have incorporated data from New York, Massachusetts, Pennsylvania, and other states that typically start schools in September. While this does not necessarily bias the results, it calls into question whether the results of that study can be representative of the entire country and suggests at least one important geographic confounding variable affects observational studies of school-based mask mandates in the United States.

If mask wearing were assigned or promoted at random, as, for example, in the Bangladesh cluster-randomized trial by Abaluck et al.⁹, then the relationship between these mandates and case outcomes may have a causal interpretation¹⁵. However, school mask mandates in the United States are not only non-random, but likely to be correlated with unobserved factors in systematic ways, making it inappropriate to infer causality.

First, school districts that mandate masks are likely to invest in other measures to mitigate transmission and may differ by testing rates and practices. Second, the choices made by school districts reflect the attitudes and behavior of their community. Communities that are concerned about the spread of SARS-CoV-2 are also likely to implement other measures, even outside of schools, that may eventually result in lower spread in the community and including within schools. Finally, the timing of public health interventions is likely to be correlated with that of private behavioral changes. Public health measures are typically introduced when case counts are high, which is precisely when community members are likely to react to media coverage and change their own behavior.

Our study also uses observational data and does not provide causal estimates either. However, there is an important difference: while the presence of correlation does not imply causality, the absence of correlation can suggest causality is unlikely, especially if the direction of bias can be reasonably anticipated¹⁶.

In the case of school mask mandates, the direction of bias can be anticipated quite well. Past research suggests that behavioral changes are likely to be positively correlated with public health measures to reduce cases, both cross-sectionally and temporally¹⁷. If so, the bias in the estimated coefficient from a naïve regression of case outcomes on public health measures will be negative. In other words, an analysis that omits the extent and timing of private behavioral changes, even one that controls for covariates, will tend to overstate the effect of mask mandates, and run the risk of a spurious correlation between mask mandates and declines in case rates.

To see this, consider estimating the following linear regression:

$$y_{it}=b_0+b_1{x}_{it}+b_2{u}_{it}+e_{it}$$

where y _it denotes case rates in community i at time t, x denotes observed public-health measures such as mask mandates, and u denotes variables that can affect case rates, but are unobserved or difficult to measure, such as private behavioral changes in the population. If we expect that x and u have a correlation δ > 0, but that u is negatively correlated with y, the bias in the estimated coefficient from a naive regression of case outcomes on observed public-health measures is b ₂ δ which is negative¹⁶.

In principle, it is possible for the bias to operate in the other direction. One mechanism for this would be if districts that require masks in schools implement fewer other mitigations, perhaps because they believe that mask mandates are sufficient to decrease the risk of infection. However, evidence from some large school districts does not support this hypothesis. For example, during the 2021-22 school year, New York City Public schools—by far the largest school district in the country—required masks for all students from September 2021 until March 2022, both indoors and outdoors. During this time, both the school district and the city imposed some of the strictest mitigation measures in the country. Additional measures in New York schools included daily symptom screening checks, contact tracing of exposed students in conjunction with whole or partial class quarantines, staggered lunch and recess times, and an extensive ventilation system with two HEPA filters in every classroom.3 New York City had mask and vaccine requirements for most places, monetary incentives to encourage adult and child vaccinations and many other interventions.4 Regardless of their individual efficacies, these interventions could have lowered infection rates in areas with mask mandates, confounding the data in such a way that would give the false appearance of high mask effectiveness.

Importantly, too, districts with mask mandates were more likely to have higher vaccination rates which, at the time of the Budzyn study during the Delta variant wave, might have had more of an effect on infection and transmission rates¹⁸ than later on, with the emergence of the more transmissible and immune-evasive omicron variant¹⁹. Higher vaccination rates at the time of the initial analysis by Budzyn et al. could have been conflated with the effect of masks. This would further support the hypothesis that the correlation between school mask mandates and other interventions is likely to create a spurious negative, rather than positive, bias in the estimated effect of mask policies on case rates, as explained above.

An additional possible confounder in our study may be differences in testing rates across counties with and without school mask requirements, especially as research suggests that Republican-led states have historically had lower testing rates²⁰. This limitation would have affected the Budzyn et al study as well. In the Supplementary appendix, however, we present both cross-state and within-state data which suggest that differences in test rates do not affect our results.

We also demonstrated that removing redundant socioeconomic data from the regression analysis actually resulted in a significant positive association between school mask mandates and COVID-19 cases. No causality can be inferred with the increased cases, but this demonstrates how controlling multiple times for similar variables can alter results and including the CCVI appears to strengthen the association identified by Budzyn et al. As shown in the Supplementary appendix, removing CCVI from Budzyn et al. initial analysis would have rendered their initial finding insignificant. Indeed, it is unclear what the purpose was of including CCVI in the regression analysis when it contains redundant information to the social vulnerability index (SVI) which the authors also controlled for.

An important caveat to our study is that it does not exclude some effectiveness of mask usage or school mask mandates. We do, however, show why relying on observational studies can be misleading. Such studies will tend to systematically overstate, rather than understate, the effectiveness of interventions for two reasons: selection bias caused by particular jurisdictions choosing to implement interventions; and omitted variable bias, due to such jurisdictions also being likely to implement other, unobserved interventions.

Furthermore, studies with significant findings are more likely to be published than those with non-significant or negative findings¹⁴. This is particularly important in the context of the current pandemic where publishing studies that fit a certain narrative can become a self-fulfilling prophecy rather than an unbiased pursuit of truth.

In summary, expanding upon a widely cited CDC study, and employing the same methodology but with a larger, more representative dataset over a longer time period, we fail to find the same evidence that school mask mandates are associated with a reduction in county pediatric COVID-19 cases. We demonstrate how observational studies can be misleading when used to guide public health policy.