Learning loss during Covid-19: An early systematic review

Abstract

With Covid-19 having caused significant disruption to the global education system, researchers are beginning to become concerned with the impact that this has had on student learning progress and, in particular, whether learning loss has been experienced. To evaluate this, the authors conducted a thorough analysis of recorded learning loss evidence documented between March 2020 and March 2021. This systematic review aims to consolidate available data and to document what has been reported in the literature. Given the novelty of the subject, eight studies were identified; seven of these found evidence of student learning loss among at least some of the participants while one of the seven also found instances of learning gains in a particular subgroup. The remaining study found increased learning gains in their participants. Additionally, four of the studies observed increases in inequality where certain demographics of students experienced learning losses more significant than others. It is determined that further research is needed to increase the quantity of studies produced, their geographical focus, and the numbers of students they observe.

Due to the emergency nature of the Covid-19 pandemic, education systems around the world are facing extreme disruption. At its peak, UNESCO (2020) reported that nearly 1.6 billion learners in more than 190 countries, or 94 percent of the world’s student population, were impacted by educational institution closures. Given the abruptness of the situation, teachers and administrations were unprepared for this transition and were forced to build emergency remote-learning systems almost immediately. In response to this disruption, education researchers are beginning to analyze the impact of these school closures on student learning progress or lack thereof.

The term “learning loss” is commonly used in the literature to describe declines in student knowledge and skills (Pier, Hough, Christian, Bookman, Wilkenfeld, & Miller, 2021). Historic data provides researchers with information regarding where student learning should be year over year and is often measured through regular testing. Learning loss occurs when educational progress does not occur at the same rate at which it has historically compared to previous years (Pier et al., 2021).

Outside of the classroom, these losses may translate to greater long-term challenges. Currie and Thomas (2001) put this into perspective because they observe that a 0.20 standard deviation (SD) decrease in standardized test scores could decrease future employment probability by 0.86 percent. Additionally, Chetty et al. (2014) observe that increasing student achievement by 0.20 SD results on average in a 2.6 percent increase in annual lifetime earnings (Maldonado & De Witte, 2020). Likewise, another year of schooling is, on average, associated with an 8–9 percent gain in future earnings (Psacharopoulos & Patrinos, 2018).

While many researchers have established learning-loss prediction models (Azevedo et al., 2021), formal research and documentation of the actual impact that Covid-19 has had on student learning progress is just beginning to emerge. As the global education system continues to face pandemic-related disruption, a strong understanding of how Covid-19 school closures are impacting student learning progress can better equip educators, policy-makers, and researchers going forward.

Our focus in this article is on an important research question: Have Covid-19 education closures resulted in recorded student learning losses? To answer this, we conducted a thorough analysis of recorded learning-loss evidence documented between March 1, 2020, and March 18, 2021. This systematic review aims to consolidate such data and document what has currently been reported in the literature. To the best of our knowledge, this is the first systematic review of its kind.

Our article makes three important contributions. First, we develop a comprehensive review that consolidates the research that has been presented related to the impact of Covid-19 on student learning progress. Second, we answer the research question, “Has pandemic-related learning loss been recorded in the literature?” Third, based on our review, we identify significant gaps in the literature and provide relevant guidance for further research.

We begin by describing the methods used in this review to identify and collect the articles analyzed. We then present an analytical review where each article was categorized by its geographical region studied, length of school closure, education level of students analyzed, subject analyzed, documented learning impact, presence of increased inequality, and sample size. Next, we move on to our discussion where we review the findings of the analytical review. We then end by discussing areas for future research and summarizing the main ideas of this paper.

Methodology

The initial search was performed using English-language articles published between March 1, 2020, and March 18, 2021. To ensure a comprehensive, multidisciplinary search, the electronic databases included were ECONLIT, Google Scholar, PubMed, Education Resources Information Center, and Cochrane Library. To conduct the search, the keywords “Covid-19”, “coronavirus”, “2019-ncov”, “sars-cov-2”, or “cov-19”, were used in combination with “learning loss”, “learning slide”, “education gap”, or “achievement gap”. Along with this, some studies were identified by reaching out to colleagues and researchers.

When conducting the search analysis, thousands of articles were identified; however, the majority of these pertained to hypothesized or predicted learning loss. To narrow this down to studies with recorded results, article abstracts were then screened. Studies that conducted student analyses and reported impacts on learning progress (either positive, negative, or insignificant) as a result of Covid-19 school disruptions were included. After this screening process, eight articles remained (Table 1). Factors for rejecting studies from our review included the absence of student analyses or recorded impacts on learning progress, analyses occurring before the onset of Covid-19, or hypothesized results.

Table 1.

Studies included

Title	Authors
Learning inequality during the Covid-19 pandemic	Engzell, Frey, & Verhagen (2021)
Influence of Covid-19 confinement on students’ performance in higher education	Gonzalez, de la Rubia, Hincz, Comas-Lopez, Subirats, Fort, & Sacha (2020)
The impact of Covid-19 on student learning in New South Wales primary schools: An empirical study	Gore et al. (2021)
Collaborative for student growth. Learning during Covid-19: Initial findings on students’ reading and math achievement and growth	Kuhfeld, Tarasawa, Johnson, Ruzek, & Lewis (2020)
The effect of school closures on standardized student test outcomes	Maldonado & De Witte (2020)
Learning during the Covid-19 pandemic: It is not who you teach, but how you teach	Orlov, McKee, Berry, Boyle, DiCiccio, Ransom, Reese-Jones, & Stoye (2020)
Educational gains of in-person vs. distance learning in primary and secondary schools: A natural experiment during the Covid-19 pandemic school closures in Switzerland	Tomasik, Helbling, & Moser (2020)
Did students learn less during the Covid-19 pandemic? Reading and math competencies before and after the first pandemic wave	Schult, Mahler, Fauth, & Lindner (2021)

Analytical review

Once articles were selected, they were then coded using the classifications in Table 2.

Table 2.

Classifications used to analyze studies

Classification term	Description
Country	The residing nation of the study’s participants
Closure length	The number of days that the participants were out of in-person traditional schooling prior to assessment
Education level	Education level of participants
Subject	Course subject of participants
Learning loss	Documented level of learning loss experienced by participants. If gains were experienced, “Improved” was listed
Equality impact	Documented differences in the level of loss experienced by certain groups of students
Sample size	Sample size of students analyzed

As indicated in Table 3, we find that seven out of the eight studies identified learning loss among at least some of the students analyzed. For example, Maldonado and De Witte (2020) found Grade 6 students in Belgium experienced losses of 0.19 SD in math and 0.29 SD in Dutch. Engzell, Frey, and Verhagen (2021) find that, overall, Grade 4–7 students in the Netherlands have encountered an average 0.08 SD learning loss in math, spelling, and reading. Tomasik, Helbling, and Moser (2020) found learning progress of primary-school students in Switzerland during in-person learning to be more than twice as high compared to the progress made during the eight-week school closure. Orlov et al. (2020) determined that economics students at four USA universities were 0.19 SD behind. Kuhfeld, Tarasawa, Johnson, Ruzek, and Lewis (2020) found that Grade 3–8 students in the USA scored 5–10 percentile points below historic levels in math. Gore et al. (2021) found Year 3 students studying math in low ICSEA (Index of Community Socio-Educational Advantage) schools in Australia to be two months behind the progress students made in 2019. Last, Schult, Mahler, Fauth, and Lindner (2021) find learning losses of 0.07 SD in reading comprehension, 0.09 in operations, and 0.03 in numbers for Grade 5 students in Germany. At the university level, in a single university, for 458 students in STEM faculties, learning outcomes actually improved (Gonzalez et al., 2020).

Table 3.

Results of literature classification.

Source	Country	Closure length	Education level	Subject	Learning loss	Equality impact	Sample size
Maldonado & De Witte (2020)	Belgium	9 weeks	Primary, Grade 6	Math	0.19 SD	Yes	Not specified
			Primary, Grade 6	Dutch	0.29 SD	Yes	Not specified
			Primary, Grade 6	Social science	Insignificant	Not specified	Not specified
Engzell, Frey, & Verhagen (2021)	The Netherlands	8 weeks	Primary (Age 8)	Math	0.063 SD	Not specified	92,180 students
			Primary (Age 8)	Reading	0.05725 SD	Not specified	76,397 students
			Primary (Age 8)	Spelling	0.09375 SD	Not specified	90,403 students
			Primary (Age 9)	Math	0.07325 SD	Not specified	93,417 students
			Primary (Age 9)	Reading	0.0975 SD	Not specified	79,016 students
			Primary (Age 9)	Spelling	0.07075 SD	Not specified	91,567 students
			Primary (Age 10)	Math	0.0935 SD	Not specified	93,769 students
			Primary (Age 10)	Reading	0.08425 SD	Not specified	68,412 students
			Primary (Age 10)	Spelling	0.0755 SD	Not specified	91,315 students
			Primary (Age 11)	Math	0.05025 SD	Not specified	73,263 students
			Primary (Age 11)	Reading	0.07425 SD	Not specified	48,537 students
			Primary (Age 11)	Spelling	0.07575 SD	Not specified	69,841 students
			Primary (Grade 4–7)	Math, spelling, reading	0.08 SD	Yes	350,000 students
Tomasik, Helbling, & Moser (2020)	Switzerland	8 weeks	Primary (Grade 3–6)	Math, German	0.20 SD	Not specified	13,134 students
			Secondary (Grade 7–9)	Math, German	Insignificant	Not specified	15,551 students
Gonzalez et al. (2020)	Spain	10 weeks	Higher education	Applied computing, metabolism, design of water-treatment facilities	Improved	Not specified	458 students
Orlov et al. (2020)	USA	3.5 weeks	Higher ed	Economics	0.185 SD	No	4 universities
Kuhfeld, Tarasawa, Johnson, Ruzek, & Lewis (2020)	USA	Not specified	Primary (Grade 3–8)	Math	5–10 percentile points	Inconclusive	4.4 million students
			Primary (Grade 3–8)	Reading	Insignificant	Inconclusive	4.4 million students
Gore et al. (2021)	Australia	8–10 weeks	Primary (Year 3, all schools)	Math	Insignificant	Yes	1,427 students
			Primary (Year 3, low ICSEA schools)	Math	2 months less growth	Yes	334 students
			Primary (Year 3, mid ICSEA schools)	Math	Improved, two months additional growth	Yes	813 students
			Primary (Year 3)	Reading	Insignificant	No	1,429 students
			Primary (Year 4)	Math	Insignificant	No	1,498 students
			Primary (Year 4)	Reading	Insignificant	No	1,515 students
			Primary (Year 3–4)	Math and reading	Insignificant	No	3,030 students
Schult, Mahler, Fauth, & Lindner (2021)	Germany	8.5 weeks	Primary (Grade 5)	Math	0.09 SD	Yes	80,000 students
					0.03 SD	Yes	80,000 students
				Reading	0.07 SD	Yes	80,000 students

Discussion

Learning loss is being experienced. As Table 3 indicates, the early findings of seven studies provide evidence of experienced learning losses among students. These observed losses are occurring across a range of subjects, grade levels, and geographical regions. This signals that although robust and empirical research on Covid-19-related student learning loss is limited, learning loss itself may not be.

Not all students are experiencing learning loss. While the majority of the literature analyzed indicates that students are experiencing some level of learning loss, there were also instances where this was not the case. For example, both Maldonado and De Witte (2020) as well as Kuhfeld et al. (2020) found learning losses in certain subjects but insignificant impacts in others. Likewise, while Tomasik et al. (2020) found primary students to be impacted, they found no impact on secondary students. This is consistent with the literature showing that students in the early grades may be more vulnerable than secondary students because of their inability to seek learning on their own, due to the differences in developmental and cognitive abilities. In their Australian study, Gore et al. (2021) found there to be no evidence of overall learning loss in Year 3 and 4 students in math and reading with the exceptions being Year 3 students in math in low-ICSEA schools who experienced losses, while mid-ICSEA students experienced small gains. Last, in the case of Gonzalez et al. (2020) who studied university students in Spain, it was determined that student learning progress actually improved rather than declined during the Covid-19 learning disruption period, but this was for university students in STEM subjects at one university.

Some students are experiencing more learning loss than others. Of the eight studies, four found instances of inequality, while only one found demographics exclusively to have no impact on learning loss. Gore et al. (2021) found instances of increased inequality as well as instances of no change. The other studies did not specify in this area or in the case of Kuhfeld et al. (2020) found inconclusive and minor differences between ethnic or racial groups. In the four studies where increases in inequality were observed, certain demographics of students experienced losses more significant than others. Maldonado and De Witte (2020) observed inequality within schools rise by 17 percent for math and 20 percent for Dutch. Engzell et al. (2021) determined that losses were up to 60 percent larger among students from uneducated homes. Gore et al. (2021) found the only losses to be among students from low-ICSEA schools where the lower the ICSEA level the lower the educational advantage attending students have due to their parents’ occupation and education, their geographical location, and the school’s proportion of indigenous students (Australian Curriculum, Assessment and Reporting Authority, 2014). Schult et al. (2021) found losses in math among Grade 5 students to be more severe in low achieving students. In reading comprehension, Schult et al. (2021) found more severe losses among middle- to high-achieving students.

More research is needed. In general, the literature representing the impact that Covid-19 has had on student learning progress is limited in the quantity of studies available, geographical regions analyzed, and number of participating students. Given the novelty of the subject, it is understandable why education researchers are only just beginning to analyze the learning losses that students have experienced. However, a stronger understanding of how Covid-19 school disruptions have impacted student learning is still needed. To support this, more studies are needed.

Along with this, the current studies that are available are limited in their geographical span. The only limited information that is currently available is from Belgium, The Netherlands, Switzerland, Spain, the United States, and Australia. Given the differences in educational institutions between countries, in terms of quality, length of school closures, and remote-learning strategies, it is crucial that researchers continue to investigate Covid-19-related learning loss in countries where limited research exists.

Last, many of the studies themselves who were analyzed in this systematic review had limited numbers of participants. For example, Gonzalez et al. (2020) analyzed just 458 students at one university. Similarly, Orlov et al. (2020) observed economics students in just seven classes across four universities. While the information these studies presented remains relevant to their observed samples, research that can more accurately represent larger groups of students remains crucial to policy-makers. As such, there is a demand for studies that analyze representative groups of students.

Conclusion

Through conducting a thorough analysis of recorded learning-loss evidence documented between March 2020 and March 2021, this systematic review provided a consolidated audit of available research on Covid-19-related learning loss. Given the novelty of the subject, eight studies were identified. Seven of the eight found evidence of student learning loss among participants while one of these found instances of learning gains in a particular subgroup. The remaining study observed learning gains among university students. Along with this, four of the studies observed increases in inequality where certain demographics of students experienced learning losses more significant than others. Further research is needed to increase the quantity of studies produced, their geographical focus, and the numbers of students they observe.

Biographies

Robin Donnelly

was Consultant to the World Bank and a graduating Bachelor of Commerce student at the University of Ottawa, where she has focused her studies on management and social entrepreneurship. With significant experience managing social enterprises, she currently specializes in topics centered at the intersection of international development and commerce. Robin has managed many grassroots projects focused on addressing social and environmental issues including those related to climate change, food security, and education. In her own time, Robin also successfully launched and exited an enterprise of her own that aimed to create value out of plastic waste. Prior to joining the World Bank’s education sector, Robin worked for the United Nations Office for Partnerships on tasks related to advancing the Sustainable Development Goals through private and public partnerships.

Harry Anthony Patrinos

is Manager for Europe and Central Asia in the World Bank's education global practice. He specializes in the returns to education, quality of education, demand-side financing, and public-private partnerships. Harry has worked on the quality of education and co-created the Harmonized Learning Outcomes, recently published in Nature. He has studied and worked extensively on the socioeconomic status of Indigenous Peoples, and has co-authored books on the subject, including, Indigenous Peoples, Poverty, and Development (Cambridge). He received his doctorate from the University of Sussex.