Effect of School Reopenings on Children’s Mental Health during COVID-19: Quasi-Experimental Evidence from California

Abstract

Background:

School closures during the COVID-19 pandemic disrupted children’s education, socialization, and access to mental health resources, raising concerns about long-term effects on childrens’ mental health. The objective was to evaluate impacts of pandemic-era school reopenings on children’s mental health and healthcare expenditures. Variation in timing of school reopenings created a unique quasi-experiment.

Methods:

We used difference-in-differences analysis to examine how staggered implementation of school reopenings affected diagnoses with depression, anxiety, and attention-deficit/hyperactivity disorder, and related healthcare spending among school-aged children during March 2020-June 2021 across 24 California counties. Data were drawn from medical claims from the second largest private health insurer in the state (N=185,735).

Results:

School reopening was associated with a 1.2%-point drop in monthly prevalence of mental health diagnoses (95% CI: −1.59, −0.74), and a 10.6% (95%CI: −13.4%, −7.8%) drop in related healthcare spending. The mental health conditions that saw the largest differential between in-person and remote school districts were anxiety and depression. Effects were strongest among girls.

Conclusions:

In-person learning is an important component of children’s mental health. These results are informative for future policymaking during public health crises, to balance infection risk with the need for socialization and other critical resources that schools provide to children.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic caused disruptions to schools during the 2019–2020 and 2020–2021 academic years. By March 16, 2020, more than half of all United States (US) students were affected by school closures, and on March 25, 2020, all US public schools were closed to prevent infectious spread, with most transitioning to remote learning formats.¹ Schools began reopening and discontinuing remote learning in August 2020 at varying capacities, although many remained closed until 2021. These changes disrupted the routines of children, as well as their education and socialization.^2–7

By disrupting routines and posing challenges to educational quality, remote learning may have contributed to stress, disengagement, and emotional difficulties in children. Declines in children’s mental health were documented during the COVID-19 pandemic and the effects were reflected in health care settings.^8–18 A study found the fraction of youth with mental health emergency department visits increased 7% during the second year of the pandemic but there has been limited research on the role of school closures.¹⁵ More recent studies found school closures and canceled school activities were associated with increased ADHD visits and high risk of developing depressive symptoms.^19–20 Despite concerns about youth mental health during the COVID-19 pandemic, there is limited evidence on how school learning modes affected children’s well-being.

This study addresses that gap by leveraging a natural experiment created by California’s staggered school reopenings, to estimate the impact of school reopenings on mental health diagnoses and related healthcare expenses among children and adolescents in California. We used data from a large health insurer’s claims database linked with administrative school data from the state’s Department of Education between March 2020 and June 2021. During this time, districts started to reopen in a staggered fashion, creating a natural experiment to examine the impact of school reopening on children’s mental health. The findings of this study have important implications to guide future policymaking to balance the risks and benefits of school closures for children’s wellbeing during future public health emergencies or other crises.

Methods

Data Sources and Study Population

We drew data from Healthcare Integrated Research Database, a large commercial administrative database with individual-level claims data; and administrative school-level data from California Department of Education. The healthcare database includes health insurance claims for individuals enrolled in health insurance plans offered or managed by a national private insurer, and it has been utilized for research for almost 2 decades.²¹ The data used in this study cover the period from March 2020 to June 2021 and included enrollees from 24 counties in California with available linked school district data. The database overrepresents individuals from higher-income households and those with employer-sponsored insurance, and therefore does not fully capture populations with lower socioeconomic status or public insurance.

We first matched school district identifiers associated with each school in the COVID-19 School Data Hub dataset with local education agency identifiers using the Geographic Relationship Files in 2021 from the National Center for Education Statistics, which also has census block group numbers.²² We then merged this file with school district identifiers using the block group numbers in the claims database to obtain the school district information associated with each individual and month. While it is possible that a small fraction of children may attend a school outside of their district of residence, this is expected to have a negligible effect on our analysis. California law generally requires attending schools within their district of residence, and only about 0.2% of students participated in the interdistrict “District of Choice” program in 2018–2019.²³ Thus, district of residence likely approximates actual school attendance for the vast majority of children in our sample. More details on sample creation and linkage are provided in the eAppendix; http://links.lww.com/EDE/C286.

Our sample consisted of children and adolescents between the ages of 5 and 18 living across 24 counties (in 224 school districts) in California between March 2020 and June 2021 [Table 1]. We excluded children who moved counties or did not have continuous medical eligibility during the study period. We included all insured children, regardless of any pre-existing conditions or insurance claims. There was no missingness in outcomes or covariates in our sample. While private schools may have had different reopening timelines than public schools, we were unable to determine which students attended public school; nevertheless, over 90% of US students attend public schools, resulting in a low level of misclassification.²⁴

Table 1.

Distribution of children and districts by reopening status during the study period

	Children exposed to reopening		School districts exposed to reopening
	N	%	N	%
Mar-20	0	0	0	0
Apr-20	0	0	0	0
May-20	0	0	0	0
Jun-20	0	0	0	0
Jul-20	0	0	0	0
Aug-20	4,507	2	6	3
Sep-20	11,420	6	12	5
Oct-20	21,733	12	32	14
Nov-20	40,901	22	55	25
Dec-20	47,648	26	58	26
Jan-21	49,916	27	60	27
Feb-21	50,221	27	63	28
Mar-21	81,868	44	107	48
Apr-21	157,765	85	201	90
May-21	181,561	98	220	98
Jun-21	185,735	100	224	100

Exposure

We created a binary district-level reopening variable, following the method of the COVID-19 School Data Hub, in which the predominant learning method among schools in a district was assigned as the learning method for the whole district. In other words, students were considered to be “exposed” to school reopening if they resided in a school district in which most students had at least some amount of in-person learning (either fully in-person or hybrid). A school district was classified as remote if most students’ schools were closed or remote. The exposure for “reopening” for a given month was determined by whether a school district was in-person or not on the first day of the month. As all school districts in our sample returned to in-person or hybrid learning by the 2021–2022 academic year, our study period ran through June 2021. We use the term “reopen” to indicate the period when the majority of schools in a district were either in-person or hybrid learning models throughout this study. If a school experienced a short-term closure and then re-opened, the school was still classified as having reopened. These infrequent short-term closures were caused by small outbreaks that affected individual schools and lasted for less than one month. The proportion of children “exposed” to school reopening increased from 2% to 100% between August 2020 and June 2021 [Table 1].

Outcomes

Our primary outcomes represent monthly prevalences, defined as whether a child had any mental health diagnosis or filled a prescription for a mental health condition within a given month. To measure prevalence of mental health conditions, we extracted monthly data for diagnoses with mental health conditions and related spending from the claims database. The primary outcome was a binary variable representing whether a child received healthcare with or filled a prescription for any mental health diagnosis (including anxiety, depression, attention-deficit/hyperactivity disorder (ADHD), bipolar disorder, eating disorder, sleep disorder, mania, mood disorder, other emotional disorders, and schizophrenia) in a given month. We also examined three of the most commonly diagnosed conditions separately as secondary outcomes: depression, anxiety, and ADHD. We additionally examined cost outcomes, representing the sum of patient and health plan paid amounts associated with the diagnosis and the prescription medicines respectively, presented in 2022 USD. Medical costs included expenditures incurred in inpatient, outpatient, and emergency department settings, while medicine costs refer to the costs of prescription medications related to a mental health diagnosis. We examined ADHD medicine expenditures separately because a significantly increasing trend in ADHD medicines has been observed among children for decades.^25,26 We separated the analysis on ADHD medicines to clarify whether any observed changes could attributed to school reopenings rather than ongoing secular trends. . Diagnoses were abstracted using International Classification of Disease s (ICD-10) codes from healthcare encounters, and prescription drugs were identified using Generic Product Identifiers (GPI) [eTable 1 and eTable 2; http://links.lww.com/EDE/C286]. We obtained child demographics such as age, gender, and urbanicity from administrative claims data. Area-level income quartile information was obtained by linking each child’s residential census block in claims to the American Community Survey.

We note that any misclassification in exposure or outcome is likely non-differential and would likely bias the results toward the null, suggesting that the observed associations found in this study may underestimate the true effects.

Statistical Analysis

We first examined child demographic and health characteristics before and after school reopening. Next, to study the effect of school reopening status on mental health diagnoses and related spending among school-aged children, we used a difference-in-differences (DiD) design, a quasi-experimental analytic method increasingly used in public health research that is well-suited to examining the effects of policy changes.²⁷ Students living in districts that had reopened were in the treatment group, while students living in school districts that were still remote served as the control group. As the timing of reopening occurred in a staggered fashion, we implemented a DiD estimation developed by Callaway and Sant’Anna that is robust to the aggregation biases that can arise in traditional two-way fixed effects models when treatment effects vary across groups and time.²⁸ . Because school reopening occurred in a staggered fashion across districts, the number of children contributing to each event-time estimate varies. For example, estimates for 9 months post-reopening are drawn from children in school districts that reopened early enough for nine months of follow-up (i.e., the earliest reopening cohorts), while estimates closer to the reopening month include a larger share of the full sample. This cohort structure is a feature of event-study DiD designs under staggered adoption and is accounted for in the estimation. We used Stata’s csdid command. DiD analysis relies on the parallel trends assumption, which requires that the outcomes in the control and treatment groups would have been parallel in the absence of the differences in school reopening. While this counterfactual scenario cannot be directly tested, we assessed the validity of the unconditional parallel trends assumption by checking the estimated impacts of the reopening in pre-reopening periods and providing visualizations of possible violations in the eAppendix; http://links.lww.com/EDE/C286. The estimates for the pre-reopening months were close to zero and imprecise, supporting the plausibility of the parallel trends assumption [eFigure 1; http://links.lww.com/EDE/C286]. Models included indicator variables for each child (i.e., individual-level fixed effects), which accounted for all time-invariant observed and unobserved characteristics of children. In essence, this results in comparisons within-children rather than between-children, which strengthens causal inference. Children residing in school districts that had not yet reopened served as the comparison group in each period. We did not directly control for gender, age, or race–ethnicity, as the individual-level fixed effects accounted for these time-invariant characteristics. We did not control for any other time-changing variables except the exposure status. Diagnosis outcomes were binary, coded as to 1 in a month with of diagnosis, and 0 otherwise. Expenditure outcomes were log-transformed. Along with DiD estimates, we additionally reported the estimated effect size as a percentage of the pre-reopening mean of the outcome variable, calculated as the ratio of the DiD estimate to the pre-reopening mean. In secondary analyses, we conducted subgroup analyses stratified by gender, since boys and girls may have experienced the effects of school reopenings differently. We also repeated the analyses for adolescents aged 14–18. Race and ethnicity were missing for about three-quarters of this sample, precluding analyses of racial–ethnic disparities.

We estimated 95% confidence intervals using robust standard errors clustered at the district level.

Ethical Approval

This study involved the secondary analysis of de-identified data. It was submitted to the Harvard University institutional review board and did not require review as it did not constitute human subjects research.

Results

Sample characteristics

The final sample included 185,735 children who met all study eligibility criteria [Table 2]. The mean age was 11.6 years at the start of the study period. About half (49.0%) were girls. The sample overrepresents students from high-income areas, as 61.3% lived in Census blocks in the highest income quartile. The proportion of children with a mental health diagnosis was 2.8% pre-reopening, increasing to 3.5% post-reopening. The average monthly medical costs related to mental health were $49.5 and $64.6 before and after school reopening, respectively [Table 2]. These trends likely reflect underlying age, period, or cohort effects. Our DID approach accounts for these shared time trends by comparing within-individual changes in districts that reopened to changes in districts that remained closed, reflecting relative changes attributable to school reopenings.

Table 2.

Sample Characteristics

	Pre reopening		Post reopening
Demographic characteristics
Age, mean (SD)	11.6	3.7	12.2	3.7
Female, %	49.0		48.9
Lives in a rural area, %	7.5		7.5
Census block income quartile, %
Lowest quartile	10.5		10.5
Second quartile	10.0		10.0
Third quartile	18.1		18.1
Highest quartile	61.3		61.3
Outcomes
Any mental health diagnosis, %	2.8		3.5
Depression diagnosis, %	1.0		1.3
Anxiety diagnosis, %	1.7		2.2
Attention deficit/hyperactivity disorder diagnosis, %	1.2		1.3
Medical costs related to mental health diagnoses, mean (SD)a	49.5	3856.0	64.6	7634.4
Prescription costs of mental health medicines, mean (SD)b c	5.5	55.0	6.2	59.2
Cost of ADHD medicine, mean (SD)b	4.3	41.8	4.7	45.2
Observations	2,176,318		795,442
Number of children	185,735		185,735

^aExcludes medicine costs.

^bSamples for medication costs include those with additional pharmacy coverage and sample sizes are different: 640,316 and 234,104 observations pre- and post- reopening respectively from 56,168 children.

^cAll costs are in 2022 USD. Costs are inclusive of out-of-pocket costs and costs paid by the insurer.

Effects of school reopening on children’s mental health diagnoses

Before schools reopened, trends for treatment and control districts were similar (i.e., almost all estimates were zero during the pre-period), supporting the model assumptions. Subsequently, students residing in school districts that reopened had decreased mental health diagnoses relative to students residing in school districts that continued to be closed [Figure 1].

Figure 1:

The graphs show the average treatment effect of school reopening among children according to the time before/after the reopening. The reopening is the first time when the majority of the children in a school district had at least some amount of in-person learning (either fully in-person or hybrid). N=2,971,760 Diagnoses data were from the HIRD and school reopening data were drawn from National Center for Education Statistics. Study period covers March 2020 to June 2021.

By 9 months following school reopening, the probabilities of being diagnosed with any mental health condition fell by 1.2 percentage points (pp) (95% confidence interval [CI], −1.59, −0.74), or 43% of the pre-reopening sample means [Figure 1]. Most of the decrease in mental health diagnoses was driven by decreased depression of 0.6 pp (95% CI, −1, −0.3) or 60% of the pre-reopening sample mean [Figure 1]. Anxiety diagnoses also dropped notably in the 8^th and 9^th months, suggesting a more delayed effect of the reopening for anxiety [Figure 1]. There was also a drop in ADHD diagnoses, but only in the 8th month following the reopening [Figure 1].

Effects of school reopening on children’s mental health costs

Figure 2 shows estimates for the effects of school reopening on the cost of healthcare spending associated with mental health diagnoses, which decreased in a similar manner. Non-drug medical costs, drug costs, and ADHD drug costs all decreased starting in the sixth month and continuing until the ninth month of reopening. By the 9th month, non-drug medical costs decreased by 10.6% (95% CI, −13.4%, −7.8%), drug costs decreased by 7.5% (95% CI, −12.5%, −2.5%), and ADHD drug costs decreased by 5.0% (95% CI, −8.2%, −1.8%).

Figure 2:

The graphs show the average treatment effect of school reopening among children according to the time before/after the reopening. The reopening is the first time when the majority of the children in a school district had at least some amount of in-person learning (either fully in-person or hybrid). Sample in “Medical costs of all mental health condition” analysis (N=2,971,760) includes only medical costs and excludes any pharmacy/medical costs. Cost of medication analyses include children with both pharmacy and medical coverage (N=874,420). All costs were transformed to logs. Healthcare costs were from the HIRD and school reopening data were drawn from National Center for Education Statistics. Study period covers March 2020 to June 2021.

Subgroup analyses

We conducted subgroup analyses to examine whether the effect of school reopenings showed different associations by gender. Among girls, Figures 3 shows decreases in mental health diagnoses overall and Figure 4 in medical and pharmacy costs following school reopenings. Specifically, the probability of having any mental health diagnosis among girls decreased by 0.48 pp (95% CI, −0.83, −0.12) six months after the reopening, and continued to decline further to −1.9 pp (95% CI, −2.6, −1.3) by the ninth month. Depression diagnoses among girls also decreased by the 6^th month following reopening and kept declining until the 9^th month where it dropped by 1.33 pp (95% CI, −1.98, −0.69). There were no changes in ADHD diagnoses after school reopenings among girls. The estimated post-reopening average treatment effect was stronger for girls than boys for any mental health diagnosis and depression, with no gender differences observed for anxiety or ADHD [eTable 3; http://links.lww.com/EDE/C286]. The medical and pharmacy costs of mental health conditions among girls also started dropping by the sixth month. Specifically, by the ninth month medical and pharmacy costs for all mental health-related conditions among girls dropped by 15.34% (95% CI, −18.45, −12.24) and 9.94% (95% CI, −16.43, −3.45) from the pre-reopening mean respectively.

Figure 3:

The graphs show the average treatment effect of school reopening among girls aged 5 to 18 according to the time before/after the reopening. The reopening is the first time when the majority of the children in a school district had at least some amount of in-person learning (either fully in-person or hybrid). Sample sizes in Figure 3 are 1,455,488. Diagnoses were from the HIRD and school reopening data were drawn from National Center for Education Statistics. Study period covers March 2020 to June 2021.

Figure 4:

The graphs show the average treatment effect of school reopening among girls aged 5 to 18 according to the time before/after the reopening. The reopening is the first time when the majority of the children in a school district had at least some amount of in-person learning (either fully in-person or hybrid). Sample size for “Medical costs of all mental health conditions” analysis is 1,455,488 and the sample includes only medical costs and excludes any pharmacy/medical costs. Samples for medication costs analysis include children with both pharmacy and medical coverage (N=430,445). All costs are adjusted for 2022 USD and transformed to logs for difference-in-difference regressions. Diagnoses and health care costs were from the HIRD and school reopening data were drawn from National Center for Education Statistics. Study period covers March 2020 to June 2021.

Figure 5 and 6 estimates that boys were not affected as much as girls, especially with respect to diagnosis outcomes, although there were still decreases in mental health related costs among boys. Specifically, there was a 2.69% drop (95% CI, −4.57, −0.81) in mental health related medical spending by the sixth month following reopening relative to the mean of pre-reopening and the decrease continued until the ninth month, reaching 6%, (95% CI, −10.36, −1.63). Similarly, costs of mental health medicines decreased six months after reopening. Next, we conducted the analyses among adolescents aged 14 to 18. The estimated decreases in mental health outcomes followed a similar pattern among adolescents, with notable decreases starting by the 6^th month and continuing until the 9^th in most of the outcomes. The estimated effects were larger among this subsample, consistent with the hypothesis that the impact of school reopening was more pronounced for adolescents [ eFigure 2; http://links.lww.com/EDE/C286].

Figure 5:

The graphs show the average treatment effect of school reopening among boys aged 5 to 18 according to the time before/after the reopening. The reopening is the first time when the majority of the children in a school district had at least some amount of in-person learning (either fully in-person or hybrid). Sample sizes in Figure 5 are 1,516,272. Diagnoses were from the HIRD and school reopening data were drawn from National Center for Education Statistics. Study period covers March 2020 to June 2021.

Figure 6:

The graphs show the average treatment effect of school reopening among boys aged 5 to 18 according to the time before/after the reopening. The reopening is the first time when the majority of the children in a school district had at least some amount of in-person learning (either fully in-person or hybrid). Sample size for medical cost analysis is 1,516,272 and the sample for it includes only medical costs and excludes any pharmacy/medical costs. Costs of medication analyses samples include children with both pharmacy and medical coverage (N=443,975). All costs are adjusted for 2022 USD and transformed to logs for difference-in-difference regressions. Healthcare costs were from the HIRD and school reopening data were drawn from National Center for Education Statistics. Study period covers March 2020 to June 2021.

Discussion

This study is among the first to examine the impacts of school reopenings during the COVID-19 pandemic on mental health among school-aged children in California using a large sample of commercially insured children, finding that in-person learning was associated with a substantial decrease in mental health diagnoses, mental health related medical costs, and mental health related prescription costs for children and adolescents in California. The impacts typically occurred after 6 months, with stronger cumulative impacts estimated over time.

There are several mechanisms through which school closing might affect mental health.²⁹ Mental health problems may arise from changes in social interaction, irregular sleep patterns, increased screen time, less balanced diets, and learning difficulties.^30–36 Additionally, decreased educational performance during remote learning may have negatively impacted children’s well-being.⁷ Child mental health may also have suffered because of broader familial difficulties due to socioeconomic hardship from unemployment and income loss and increased tensions due to more time spent in the home.³⁷ Finally, in the US, school systems often deliver mental health services, such as one-on-one counseling or therapy, case management, and referrals for care outside of the school—for instance, a 2012–2015 survey found that among all adolescents who used any mental health services in the year, 57% received some school-based mental health services.^37,38 Losing access to these resources may have contributed to the changes in children’s mental health observed in this study.

The larger observed effects of school reopenings on mental health among girls may be due to gender differences in youth behavior and socialization patterns driven by socially constructed gender norms. For example, prior work has found that girls spend more time in social conversation than boys, which may have been limited in the context of school closures, while relationships are thought to be more central for girls’ sense of self compared to boys.³⁹ Similarly, girls are more likely to report socializing with friends as a way to deal with stress.⁴⁰ This may be true of older girls in particular, who are already at increased risk of mental health problems.⁴¹ Importantly, schools that reopened often did so first for the youngest students (i.e., elementary schools reopened earlier than middle schools which reopened earlier than high schools sampled in this study). Thus, even school districts with most children attending in person may have had older students still remote, which may result in misclassification, and an underestimation of the true impacts among older children. Of note, 63% of our sample were younger children, but the estimated effects were larger among adolescents.

These results are consistent with those showing that school closings were associated with worse educational outcomes, including test scores.^7,42,43 However, the small number of existing studies on mental health are mixed. Prior work in the UK has found that mental health deteriorated more for girls than boys during the COVID-19 pandemic, although they did not examine specific policies.⁴⁴ One study found that ADHD diagnoses decreased relative to pre-pandemic levels after school closings, which may be because of decreased access to mental health services and underreporting bias.¹⁷ Another study found that teen suicide rates decreased under remote learning, suggesting that the decline was due to reduced bullying.¹⁸ Results from our study finding that school reopening was associated with decreased depression diagnoses suggest alternative mechanisms, such as closer supervision of children with suicidal ideation when they were at home with parents. Alternatively, it may be that school reopening had heterogenous effects on students’ mental health, depending on pre-existing mental health status, perhaps improving the mental health of those who were most at risk of suicide.¹⁸ Additionally, family dynamics and socioeconomic inequalities may have also contributed to the heterogeneity in mental health impacts.

This study has several strengths, including the use of large administrative data and rigorous quasi-experimental methods. Moreover, our use of a panel of fully insured children reduces the chances of bias due to changes in sample composition or healthcare access. This study also has limitations. First, it is possible that school reopenings coincided with loosening of other COVID-19 policies (e.g., workplace closures) or COVID-19 infection rates, such that other related policies may contribute to the observed results. While our fixed effects approach reduces the chances that time-invariant characteristics of school districts and children bias results, our results should nevertheless be interpreted in light of the possible co-occurrence of time-varying COVID-19-related policies.⁴⁵ Other time-varying factors such as changes in local telehealth access, trends in healthcare seeking, extracurricular offerings, or spillover effects where children in remote districts accessed services in nearby reopened areas, could have influenced outcomes independently of school reopening. Unfortunately, our analytical choice cannot accommodate time-varying covariates, therefore represent a trade-off between different sources of confounding; future studies should examine this research question using different analytic approaches. Second, we could not identify children who were in home school or private school, but this is likely to be less than 10% of the sample.⁴⁶ Another limitation is that the sample only includes children in relatively higher-income areas in California with commercial insurance and relatively better access to care. Evidence from literature estimate differentially elevated impacts of school closures on students of color and students with economic disadvantage compared with their non-disadvantaged counterparts.⁴⁷ More research with richer sociodemographic variation and data on race–ethnicity is needed to examine the consequences of school reopening among those in lower-income areas with more limited healthcare access. Given that students of lower socioeconomic status experienced more negative estimated impacts of school closures on test scores and would also be more likely to have parents who experienced unemployment and income loss, our results likely underestimate the effects among more marginalized groups.^11,48,49

Conclusion

In conclusion, this study demonstrates that school reopenings were associated with decreased mental health diagnoses and related spending among children and adolescents. These findings are consistent with the hypothesis that prolonged school closures may have contributed to the current US mental health crisis among children. Understanding the impacts of school disruptions is critical to inform future public health crisis responses, and future research should continue to explore this topic.