Commentary: School closures, the pandemic, and pediatric mental health: Scrutinizing the evidence

Abstract

The COVID-19 pandemic arrived with significant hardship. The secondary impacts of the pandemic and our response with respect to pediatric mental health has been a subject of significant discussion in the lay public, media, and decision-maker groups. The initiatives to control SARS-CoV-2 have become politicized. A narrative emerged early that strategies to mitigate the spread of the virus were harming children’s mental health. Position statements from professional organizations in Canada have been used to support this claim. The aim of this commentary is to provide a reanalysis of some of the data and research methodology used to support these position statements. Some of the direct claims such as “online learning is harmful,” should be supported by a strong evidence base with significant consensus that speaks directly to causality. We find that the quality of the studies and the heterogeneity of the results does not support the strength of the unequivocal claims made by these position statements. In a sample of the current literature examining the issue, we find that outcomes range from improvements to deteriorations. Earlier studies relying on cross-sectional surveys typically have shown stronger negative effects than longitudinal cohort studies, which often have also shown groups of children experiencing no changes to measured mental health characteristics or groups that have experienced improvements. We argue it is imperative that policymakers use the highest quality evidence in making the best decisions. We as professionals must avoid discussing only one side of heterogeneous evidence.

Introduction

When the SARS-CoV-19 pandemic first emerged into the public’s consciousness, its secondary impacts quickly became a dominant part of the public discourse. On the same day that the virus was named “COVID-19” by the World Health Organization, reports in mainstream media described the mental health impacts of the pandemic (1,2). Countless scientific and non-scientific articles used alarmist language, such as “a tsunami,” to predict the secondary mental health impacts of the pandemic that were to befall the public (3,4). We have witnessed an overriding narrative in scientific and public discourse: the amplification of findings of pandemic-related pediatric mental health harm, and a relative lack of discussion regarding findings demonstrating improvements or no change to pre-pandemic trends. The same trend has occurred for discussions of school closures or remote schooling on pediatric mental health.

The public can reasonably expect to look to scientific and medical publications, especially by professional organizations, to try to understand what the impact of COVID-19 mitigation measures would be on youth mental health. This commentary is designed to answer the question: “Is there evidence of pandemic- or school closure-related pediatric mental health outcomes other than deterioration, and has this evidence been reflected in Canadian professional organizations recommendations and position statements?” To do this, we have conducted a selected reanalysis of the data used to support these position statements, as well as offered examples of methodologically rigorous research that has produced results conflicting with the predominant narrative, to emphasize that the current state of knowledge is still evolving and that this subject should be approached with more nuance.

Critical appraisal of evidence cited in position statements by two major Canadian Institutions

The Canadian Paediatric Society (CPS) released an advocacy letter arguing that “online learning is harmful” and that “increased depression and anxiety, suicidality, eating disorders, learning losses, and delayed development… have been documented by researchers across the country (5).” In it, they provide three citations for this statement, one of which is a publication of the other major Canadian organization, the Royal Society of Canada (RSC). The executive summary of the RSC document states plainly that “[school closures]... have led to a notable deterioration in mental health (6).”

Both the CPS statement and the RSC report cited a study by Cost et al. This was a cross-sectional Canadian survey of 1,013 parents and 347 adolescents conducted between April 15 to June 19, 2020 (7). Parents and youth were asked questions about their mental health across six domains: depression, anxiety, irritability, attention, hyperactivity, and obsessions or compulsions. There are multiple ways to examine the premise asserted in the title of the article that Canadian children and adolescents were “mostly worse.” For the authors, this was determined by the proportion of respondents aged 6–18 who were “faring worse in at least one domain” (70.2%) to the proportion “faring better in at least one domain” (19.5%). Unfortunately, this analysis ignores important scenarios, such as “no change in most domains” or “better in one domain but worse in another.” In our analysis, we grouped results by respondent types (parent proxies versus self-reported by children and youth), and by age categories. Then, we dichotomized the results to “deteriorated” versus “not deteriorated” (summing “unchanged” and “improved”), and added a measure of uncertainty around the estimates using binomial 95% confidence intervals around the proportions. Figure 1 displays this re-organization of the data, which did not indicate predominant deterioration in mental health in any age categories, irrespective of respondent source. While some domains showed an even split between “deteriorated” and “not deteriorated,” (the error measurement spanned the 50% mark), the only domains in which the error measurement did not contain the 50% mark were ones where most respondents reported improvement or no change.

Figure 1.

Survey of Parents and Children April 15-June 19, 2020 as reported by Cost et al, 2021. Dichotomization (“deteriorated” vs “not deteriorated”) and 95% confidence intervals were added to test the hypothesis of “worsening mental health domains.” Because the confidence intervals for both dichotomized positions are identical, they are overlapped and represented in the lighter shaded area.

The authors also used multinomial regression analysis to explore variables associated with parent- and youth-reported mental health change, using “unchanged” status as the reference outcome. Overall, the results are challenging to interpret, given that several variables were associated with significantly improved and deteriorated outcomes depending on the mental health domain examined. For example, pre-pandemic psychiatric diagnoses were associated with both improvement (OR = 3.12, 95% CI 1.81–5.37, p<0.001) and deterioration (OR = 2.04, 95% CI 1.45–2.86, p<0.001) relative to “no change” in the depression domain. At the same time, pre-pandemic psychiatric diagnoses were associated with improvement (OR = 2.42, 95% CI 1.17–5.01, p=0.018), but not deterioration (OR = 1.24, 95% CI 0.89–1.73, p=0.194) in the anxiety domain. Surprisingly, improvements in both anxiety (OR = 5.57, 95% CI 1.64–18.97, p=0.006) and obsessions/compulsions (OR = 5.20, 95% CI 95% 1.27–21.21, p=0.022) were associated with having greater pandemic-associated economic concerns. More intuitively, social isolation-related stress was strongly associated with deterioration in all six domains (OR ranging from 11.12–55.24). However, the surveys sent to children and parents did not mention school, except for the example of “cancelled school trips” in the parent survey.

The CPS statement also describes the results of “online learning,” citing a study by Racine et al, a meta-analysis of 29 cross-sectional survey studies conducted between February and July 2020 that estimated prevalence of anxiety and depressive symptoms in multiple countries (8). During the measured period, the authors report a doubling of pre-pandemic estimates of depressive and anxiety symptoms. Several methodological and analytic observations warrant closer attention. First, the method of schooling (home, hybrid, or remote) was not a variable considered in the studies’ analyses. Second, the point prevalence estimates of both depressive and anxiety symptoms display extremely high heterogeneity (I²=99.47, I²=99.27 for depression and anxiety symptoms, respectively). Third, pooled prevalence was nearly halved when comparing high- and low-quality studies. For depressive symptoms, pooled estimates for high- (n=5) and low-quality (n=21) studies were 18% and 27%, respectively. For anxiety symptoms, pooled estimates for high- (n=4) and low-quality (n=21) studies were 12% and 22%, respectively. When restricted to high quality studies, the confidence intervals for the prevalence of anxiety and depressive symptoms during the pandemic period spanned the reported pre-pandemic prevalence values, indicating no significant change.

In a paragraph about school closure harms, the RSC cites a paper by Duckworth et al as evidence that “students who attended school remotely reported lower levels of social, emotional, and academic well-being than students who attended school in person.” In this study, a computer-based survey was given to students in Grade 8 to 11 in February 2020 (“pre-pandemic”) and October 2020 (“pandemic”), to assess their social, emotional, and academic well-being (9). The study population was stratified by school attendance type: in-person or remote. Differences for each item were regressed, controlling for pre-pandemic scores. We took the survey responses from Table S8 of the article’s supplemental material and presented them in Figure 2. In the original article, this was presented with a cropped y-axis, which visually exaggerated the size of the differences between in-person and remote students. By including the full 100-point range on the y-axis, the small effect size is visually obvious. The differences in measures of social, emotional, and academic well-being for children who participated in remote compared to in-person schooling were statistically significant, but the effect sizes of these differences were clinically meaningless (Hedge’s g of 0.08 to 0.10). The largest individual point difference on the “emotional” scale was in response to the question: “how do you feel about your life these days” where students could select one of 5 “emoticons” from “full frown” (scored as 0) to “full smile” (scored as 4). This was then stretched to a 100-point scale for comparison to other items, but returning it to its original scale shows that the difference between the in-person (2.57 out of 4) and remote schooling (2.49 out of 4) groups is a small fraction (0.08 out of 4) of the scale itself, which would be visually indistinguishable in a visual analogue scale of 5 “emoticons.”

Figure 2.

Social, emotional, and academic well-being differences in high school students receiving in person versus remote learning as reported by Duckworth et al, 2021. The vertical axis has been adjusted to show the full range of scores from 0 to 100, to demonstrate the negligible effect sizes (Hedges g for Social 0.10, Emotional 0.08, Academic 0.07) reported. ES=Effect size.

An analysis of recent findings on children’s mental health, COVID-19, and school closures

The extant literature has shown mixed findings with respect to youth mental health throughout the pandemic. In an October 2020 survey of youth, Hertz et al reported an association between virtual school instruction and increased risk of mental health challenges, compared to students attending school in-person or combined instruction (10). Due to the cross-sectional nature of this study, causality and directionality could not be determined. The authors acknowledge that it could be that children with more severe mental health problems are more likely to participate in virtual or hybrid learning. Around the same time, another survey indicated that parents felt their child’s mental health deteriorated more in those who were “remote learning” (24.9%) compared to “in-person learning,” (15.9%), yet when asked directly about mental health symptoms, the proportion of parents indicating elevated depression (4.4%), anxiety (6.3%), and stress (9.2%) symptoms in their children was much lower, with no differences between the learning modalities (11).

One of the challenges in interpreting cross-sectional surveys is that they provide a snapshot of a population, which creates difficulty in interpreting directionality for causal analysis and the impact of confounding factors. As well, it is difficult to know how the same group would have responded prior to the pandemic when surveyed on the same measures. Cohort studies address this issue by allowing longitudinal outcome measurement over multiple time points in the same group. A meta-analysis of 38 early child and adolescent longitudinal cohort studies showed a small increase (standard mean change of 0.114) in psychiatric symptoms in aggregate, however it was not significant (95% confidence interval −0.030 to 0.257, p=0.120) (12). Longitudinal cohort studies published after this meta-analysis was complete demonstrate more heterogeneous results. In the Netherlands, an adolescent cohort conducted over three time points, one pre-pandemic and two during full school closure (April 2020 and January 2021), found evidence of a small increase in depressive problems, but no increases or decreases in anxiety, oppositional, psychotic, or suicidality problems (13). A German longitudinal study of 1,618 children 7 to 18 years old found small deteriorations during the pandemic (up to October 2021) in quality of life and anxiety symptoms but not depressive symptoms (ϕ coefficient effect sizes of 0.27, 0.18, and 0.08 respectively) compared to a pre-pandemic reference sample, but that reference sample was collected between 2014–2017, which may neglect pre-pandemic trends (14). A Canadian cohort of 168 Ontario youth aged 14–24 years showed stable scores for internalizing and externalizing disorders throughout the pandemic compared to a pre-pandemic baseline, with a significant decrease in substance use disorders (15).

The association between school closure and students’ mental health remains to be robustly evaluated. Viner et al.’s systematic review of international studies from February to July 2020 also found mixed and heterogeneous findings (16). The authors carefully noted that the association between school closure and adverse mental health symptoms during the first wave of the pandemic could not be distinguished from other public health measures. In all included studies, school closures were bundled with broader public health measures, and the isolated effect of school closure on students’ mental health was not described in any study. As well, only 39% of included studies were rated as having higher quality cohort or longitudinal designs. In the absence of rigorous study designs isolating the school closure or attendance type (in person versus online), prevalence or incidence comparison with pre-pandemic figures may offer limited interpretability or equivalence due to well-known variability in outcome measures, study populations, and methodological or analytic approaches.

Potential impacts on vulnerable populations are concerning. An earlier systematic review by Jones et al found pandemic-related increases in adolescent depression and anxiety were particularly prevalent among LGBTQ+ adolescents, female students, or those with previous mental health diagnoses (17). Social support and positive coping skills were important factors for mitigating these negative effects in another meta-analysis (17). The previously mentioned Dutch Cohort analyzed an oversampled subset of youth with more severe emotional and behavioural problems in the pre-pandemic period and showed, contrary to expectations, a decrease in psychiatric symptoms during the two time periods of school closure and pandemic lockdown (13).

Suicides and mental-health related emergency department visits are an emerging source of population-level data on the pandemic’s secondary impacts. In 2020, the number of suicides in Canadians under the age of 20 (n=211) decreased from five years prior to the pandemic (n=238–277), making it the third-lowest year for deaths by suicide in this age group for the past 21 years (18). A Canadian provincial-level study found that self-harm and overdose-related hospitalizations and acute care visits made by youth aged 15–24 years decreased in the first 15 months of the pandemic compared to the prior two years (19). The Centers for Disease Control reported that there were no significant differences in pediatric suicide rates in the United States in 2020, and that pandemic-associated changes in the number of emergency department visits for pediatric mental health problems varied by sex, age, and mental health condition (20, 21). In boys aged 5–17 years and girls 5–11 years, emergency department visits in the United States for all mental health decreased during the first 20 months of the pandemic, whereas an early decrease then subsequent increase occurred for girls aged 12–17. Within that subgroup of 12–17 year-old girls, there was heterogeneity as well, as visits for four conditions did not change (depressive, anxieties, trauma, and obsessive-compulsive disorders), two conditions increased (tic, eating disorders), and three conditions decreased (disruptive behavioural, attention-deficit hyperactivity, and bipolar disorders). It is important to note that changes in access to non-emergency services may be responsible for any increase in emergency visit presentations for mental health conditions, adding to the confounding of any temporal association.

Discussion

We are concerned that there is a moral panic about the negative effects of school closures and other pandemic related responses on children’s mental health. Moral panic is a societal fear that occurs when “a condition, episode, person, or group of persons emerges to become defined as a threat to societal values and interests” (22). While a moral panic may revolve around genuine concern, its claims are often distorted and represented by political ideology and moral values, which, in turn, drives public discourse and influences how research is reported and interpreted.

The distortion of scientific evidence occurs across different public health issues (in the past with fetal alcohol syndrome, and video gaming’s impact on youth violence, for example), where harms are misrepresented or exaggerated with simplistic causal mechanisms proposed, and fear-reassuring research and information confronting the moral panic is ignored (23, 24). True to the moral panic’s use of confirmation bias, alternative explanations are not adequately considered. Fitting these concerns, few news articles have reported on Statistics Canada’s bi-monthly survey (September 2020 to November 2021), which has consistently shown that among Canadians age 12–17 years, similar proportions report improved (20%) and worsened (20%) mental health during the pandemic, while 60% reported no change (25). In the same vein, a survey was conducted with 16,940 UK students, most of which (82%) attended school “not at all” or “once or twice” per week) during the first pandemic wave. The proportion of students who felt more happy, less happy, and about as happy as prior to the lockdown were divided into thirds – yet few major news agencies picked up this reassuring news (26).

There is a significant concern about a delayed impact on mental health measures, and to the best of our abilities in this commentary, we have made every effort to include the latest data. In the time between our original submission and revision, a number of pandemic-related longitudinal pediatric mental health studies have been published. While we cannot provide a full systemic analysis in this commentary, we note that if we restrict our view to the newest studies available, it is very challenging to come up with a single narrative. For example, considering younger children, a longitudinal Canadian study published in 2022 of low-income mothers and their young children (mean age of 5.3 years) showed that the majority of mothers experienced a deterioration in at least one area of mental health during the pandemic, whereas the majority of children experienced no change or improvements (27). A cross-sectional French survey of 4,575 children 8 to 9 years old during school closures reported an association between pre-pandemic lower socioeconomic status and worse emotional symptoms, but only in families that suffered a decline in income during the pandemic (28). As data come in with subgroups and covariates, our appreciation for the complexity of the impact of school closures and the totality of the pandemic on pediatric mental health increases.

Our critical appraisal of the evidence used by two major Canadian professional organizations for their authoritative policy documents, in the context of the broader literature on secondary impacts of the pandemic on children’s mental health, highlights the complexity and inconsistency of findings in the literature. To the extent that we wish the scientific literature to inform policy, we need a better, evidence-critical approach to this topic. A review of 37 COVID-related mental health surveys conducted through June 2020 found that the majority of them were of very low quality, with online publication occurring an average of 18 days after it was received (29). As higher-quality, longitudinal evidence has emerged through the two years of the pandemic, it has revealed a more nuanced picture of pediatric mental health outcomes than the early cross-sectional surveys suggested (30). The use of high-quality evidence allows us to truly understand which groups of children fared worse, better, or the same, and which protective and risk factors influenced those outcomes; this is crucial to developing effective public policies.

The conclusion that online school and school closures are harmful for the mental health of children is not substantiated by the currently available evidence according to our analysis of the literature, including evidence cited by the CPS and RSC. Misleading statements supported by uncritical reviews of the evidence amount to unfortunate misinformation. Further, misinformation that is spread by professional organizations and amplified on social and popular media is not a trivial matter. The public is likely to trust the CPS in its review of evidence, particularly when their position on the harms of online learning is stated as unequivocally as their more evidence-based position that “the COVID-19 vaccine is safe and effective for children 5 years of age and older (31).”

Conclusion

The growing knowledge of the mental health impacts of school closures and other pandemic measures is nuanced and far from fully elucidated. Higher quality evidence is emerging, showing a complex picture of improvements, deteriorations, and within-previous-trend shifts of pediatric mental health outcomes. Understanding both risk and protective factors that predict the full gamut of outcomes for individuals is important to developing the overall picture. As personal and policy decisions are often made on the basis of the state of the science at the time, it is imperative that the scientific findings available be of the highest quality, and it must be reported as critically and broadly as possible to the public and decision makers in order to avoid individual, institutional, or political decisions to be made based upon an incorrect or oversimplified interpretation.