Skip to main content
PMC home page
Elsevier - PMC COVID-19 Collection logoLink to Elsevier - PMC COVID-19 Collection
. 2021 Nov 18;88:102515. doi: 10.1016/j.ijedudev.2021.102515

Estimation of the fundamental learning loss and learning poverty related to COVID-19 pandemic in Mexico

Felipe J Hevia a,⁎,1, Samana Vergara-Lope b,2, Anabel Velásquez-Durán c,3, David Calderón d,4
PMCID: PMC8599019  PMID: 34812219

Abstract

There is evidence of learning loss due to the COVID-19 pandemic, but there is no related information for Mexico. The objective of the study is to estimate learning loss in reading and numeracy in Mexico. We compared the results of two household surveys conducted in 2019 and 2021. We interviewed 3161 children between 10 and 15 years. We estimated a learning loss according with SES in a range from 0.34–0.45 SD in reading and 0.62–0.82 SD in mathematics by COVID-19 pandemic, and an increase in learning poverty in a range of 25.7%–15.4% in reading and 29.8%–28.8% in numeracy. Gaps in fundamental learning by gender and SES increased. There is an urgent need develop a clear strategy to perform personalized diagnoses and implement remedial courses to address learning loss.

Keywords: Academic achievement, Learning assessment, Educational equality, Learning loss, Learning poverty, Learning gap

1. Introduction

The COVID-19 pandemic, and the associated school closures, have represented one of the most critical educational emergencies of the last 100 years. As we will see below, evidence has been generated regarding the adverse effects prolonged school closures had on fundamental learning. This information is critical for education systems to develop strategies to help children recover from lost learning. For this reason, it is necessary to know in greater detail how much has been lost and how far behind children will be when they return to school.

In this sense, the research problem is the lack of information on the effects of the COVID-19 pandemic on fundamental learning in reading and arithmetic in Mexico. The closure of schools associated with the COVID-19 pandemic is related to learning loss and an increase in learning poverty. In Addition, Mexico is a country that kept its schools closed for many weeks, which has increased the educational inequality gap and deepened the school dropout crisis, making it increasingly difficult to maintain educational quality. There are simulations regarding the impact of this closure on learning, but there is no information based on empirical evidence.

1.1. COVID-19, learning loss and learning poverty

The COVID-19 pandemic has had very adverse effects on education, with school closures that have affected more than 190 million students worldwide (UNESCO, 2021). The regions most affected by the pandemic in terms of health and education are Latin America and the Caribbean (Rieble-Aubourg and Viteri, 2020, World Bank, 2021). This region has not only experienced a very high concentration of deaths concerning its population (John Hopkins University. 2021) but has kept schools closed longer than many other regions (UNESCO, 2021).

As has been widely analyzed, school closures and health and economic crises increase the inequality gap, the risk of dropout, educational exclusion, and, as we will see in this article, learning loss (Fore, 2021, GEM-UNESCO, 2020, United Nations, 2020, World Bank, 2020) and learning poverty (Azevedo et al., 2020, Azevedo and Montoya, 2021, World Bank, 2019a).

Learning loss can be defined as “any specific or general loss of knowledge and skills or reversals in academic progress, most commonly due to extended gaps or discontinuities in a student’s education” (Huong et al., 2020, p. sp). Before the pandemic, learning loss was associated with summer vacations, when a drop in academic performance and a widening achievement gap among students were observed (Kuhfeld, 2019). However, with the pandemic, interest in measuring learning loss has increased, intending to reduce learning loss and inequalities in learning and dropout (Azevedo et al., 2020, Donnely and Patrinos, 2021, Huong et al., 2020, Iqbal et al., 2020, Kaffenberger, 2020, Pier et al., 2021).

At the beginning of the pandemic, statistical estimates were made to predict these losses. For example, Iqbal et al. (2020) estimated a loss of between 0.3 and 0.9 quality-adjusted years of schooling according to data from 157 countries. Azevedo et al. (2020), through a simulation, estimated that after five months of school closure due to COVID-19, there could be a loss of 0.6 quality-adjusted years of schooling, an increase of 25% in “insufficient” level scores on standardized tests, and an increase in learning poverty, especially in the poorest countries.

These projections have been complemented with empirical analyses in developed countries, which have estimated the learning loss in various contexts, including basic and higher education. Donnely and Patrinos (2021) identified studies that show learning loss in at least some of the students analyzed. In basic education, Maldonado and Witte (2021) identified losses of 0.19 SD in mathematics in Belgium and 0.29 SD in Denmark for 6th graders in elementary school. Engzell et al. (2020) analyzed the learning loss in mathematics, reading, and writing in the Netherlands in boys and girls between eight and 11 years old and estimated a loss of 0.08 SD between 4th and 6th grade in these domains. Kuhfeld et al. (2020), in a sample of 4.4 million students aged three to eight years in the United States, estimated losses in mathematics of 5–10% points, with no differences in reading, using an at-home online test (MAP Growth Assessment). Gore et al. (2021) analyzed the closure of schools in Australia and found that after 8–10 weeks of closure, there were losses in mathematics and reading in three elementary schools. The study by Schult et al. (2021) of 5th graders in Germany found that after 8.5 weeks of closure, there was a loss of 0.03–0.09 SD in mathematics and 0.07 SD in reading. Only one study, by Tomasik et al. (2020), found no significant differences in mathematics in a sample of 25,685 students in Switzerland after eight weeks of school closure (Donnely and Patrinos, 2021, p. 149).

These empirical analyses have been published only for high-income countries with a relatively short closure of schools. Thus, except for the simulations, there are no estimates of the learning loss in middle-income countries or with more extended school closings. For this reason, the first objective of this article is to estimate learning losses due to the COVID-19 pandemic in southern Mexico.

1.2. Learning poverty

On the other hand, just before the pandemic, the UNESCO Institute for Statistics and the World Bank coined the concept of learning poverty. Learning poverty is the inability to read and understand a simple text by ten years. This indicator combines schooling and learning indicators: it begins with the proportion of children who have not achieved minimum reading proficiency (as measured in schools) and is adjusted by the proportion of children who are not in school (and are assumed to be unable to read proficiently) (World Bank, 2019a, p. 6)(World Bank, 2019b).

This concept allows the inclusion of some of the results of the Citizen-Led Assessment, which was developed in the global south to generate evidence on the difference between schooling and learning (schooling is not learning) (Pritchett, 2013). Additionally, it allows an understanding of the intensity of the learning crisis in the countries of the global south (Banerji, 2013, Banerji et al., 2013, Hevia and Vergara-Lope, 2020, Munene, 2016) and is a pioneering home-based assessment for investigating the learning of all children, whether they attend a school or not (Banerji, 2017, Mugo et al., 2015, PAL, 2018, Ruto, 2013, Wilson, 2018). The Independent Learning Measurement Project (MIA, for its acronym in Spanish), which conducts Citizen-Led Assessments in southern Mexico, generated information on the learning lag between 2015 and 2019 for various states in southeastern Mexico (Hevia and Vergara-Lope, 2019, Vergara-Lope et al., 2017, Vergara-Lope and Hevia, 2016). However, there is no information on the effects of school closures on learning poverty in Mexico. Therefore, the second objective of this article is to estimate the learning poverty in southern Mexico.

1.3. The COVID-19 pandemic in Mexico

The effects of COVID-19 in Mexico have been profound in health, economics, and education, as we will review in this document. Regarding health, a total of 326,612 excess deaths were recorded in 2020 (Mejía et al., 2021), and as of May 2021, there were 223,068 deaths. In the economic sphere, the GDP fell by 8.5% (INEGI, 2021a).

Regarding education, between March 2020 and May 2021, schools were closed for 48 weeks. According to the National Institute of Statistics and Geography (Instituto Nacional de Estadística y Geografía - INEGI), in the 2019–2020 school year, 740,000 students between 3 and 29 years old (2.2%) did not finish the school year, 58.9% for reasons associated with COVID-19% and 8.9% for lack of money or resources. Of these, 28.8% dropped out of school because they had lost contact with their teachers or could not do homework, 22.4% dropped out because someone in the household was left without a job or had a reduced income, and 17.7% left because they lacked a computer, other device, or Internet connection (INEGI, 2021b, p. 12). Therefore, for the 2020–2021 school year, 5.2 million students between 3 and 29 years old (9.6%) were not enrolled in school due to the COVID-19 pandemic and a lack of economic resources (INEGI, 2021b). This situation is similar to the estimates by the Secretariat of Public Education (SEP, for its acronym in Spanish), which, at the beginning of the 2020–2021 school year, indicated that approximately 10% of students at the primary level and 8% at the high school level had dropped out (Arellano, 2020).

The “Learn at home” strategy continued to deliver its original program and study plan by remote means, initially via the internet and, during the 2020–2021 school year, via television (SEP, 2020b). According to a telephone survey conducted by the Research Institute for Development with Equality (Instituto de Investigaciones para el Desarrollo con Equidad - EQUIDE), in May 2020, only 60% of households reported having seen or heard about this program, and the proportion decreased according to socioeconomic status (SES) (Pérez and Gaitan, 2020, pp. 3–4). Despite this, according to the government, this strategy allowed eight out of 10 students to establish constant communication with their teachers during the 2019–2020 cycle and nine out of 10 students to “maintain their learning” (SEP, 2020a). This last statement, however, was only supported by an opinion poll. As of September 2021, no document or research would allow an estimation of the effect of the pandemic on learning. For these reasons, the objectives of this article are to estimate the learning loss and learning poverty in Mexico.

2. Methodology

To meet the objectives of this research, the results of two household surveys that used the same tools to measure fundamental learning are compared. The first was developed by the MIA Project and conducted in November 2018 in Yucatan, Quintana Roo, and Campeche, and its results were published in February 2019 (MIA, 2019). In this survey, 3438 individuals between 7 and 17 were interviewed, and the MIA tool was applied to measure reading and basic mathematics (Hevia and Vergara-Lope, 2016). The second survey was implemented by the Center for Educational and Social Studies (Centro de Estudios Educativos y Sociales - CEES) and Mexicanos Primero in May 2021 in the states of Campeche and Yucatan. In Addition, in this survey, 2000 individuals between 10 and 15 were interviewed, and a questionnaire adapted from MIA that included the same items that were applied in 2019.

2.1. Participants

For this research, only participants between 10 and 15 years old who lived in the states of Campeche and Yucatan were selected from the 2019 survey, for a total of 1172 individuals. A total of 1989 individuals participated in the 2021 survey. Thus, the sample consisted of 3161 students between 10- and 15-years old living in the states of Campeche and Yucatan, with a mean age of 12.07 years (SD = 1.63); 49.64% (N = 1569) were boys, and 50.36% (N = 1592) were girls. Both samples include children in and out of school. In the 2019 sample, 97.7% were attending, while in 2021, after 14 months of closed schools, it dropped to 92.15%.

2.2. Sample

In the 2019 survey, the total number of households in the states of Yucatan, Quintana Roo, and Campeche was considered the population universe, according to the results of the 2015 inter census survey (INEGI, 2015). Therefore, the sample design was defined with a probability of 95% (z = 1.96), a margin of error of + /- 2%, a power of 0.8, and a type I error of 0.03. Based on this calculation, a sample of 2564 homes was selected.

Multistage, probabilistic, stratified cluster sampling was performed (Hernández Sampieri et al., 2001, Kerlinger and Lee, 2002). The final unit of selection was the home, and the observation units were boys, girls, and young people between 7 and 17 years old who lived in the selected home. For the selection of homes, the sampling procedure involved four steps:

1. Selection of clusters or primary sampling units through stratified systematic sampling and with probability proportional to size, which in this case was the municipalities that make up each electoral district.

2. Selection of the secondary sampling units, which in this case were rural localities and primary geostatistical areas (Áreas Geoestadística Básica-AGEB) for urban areas.

3. Random selection of blocks within the cluster.

4. Systematic selection of homes in the blocks.

All boys and girls between 7 and 17 years old who lived in the selected households and wanted to participate in the study were interviewed. The information was collected from 2564 households in 39 urban and rural locations in 20 different municipalities. The surveys were performed by 293 volunteers who received six effective hours of prior training.

In the 2021 survey, to calculate the representative sample, the universe of observations corresponded to the list of beneficiaries of scholarships from the Benito Juárez Welfare Program (Programa Nacional de Becas para el Bienestar Benito Juárez) for the 2019 fiscal year, which was the last year for which the scholarships were available. Therefore, the sample has a probability of 95% (z = 1.96) with a margin of error of + /- 3%, a power of 0.8, and a type I error of 0.03 and included 2037 boys and girls. In Addition, the selection of municipalities and localities was proportional to size, and individuals from 20 municipalities were interviewed (CEES, 2021).

2.3. Sample-Comparability procedures

Although both surveys are random, they were designed to identify households, and all children living in that household were interviewed. The following procedures were carried out to ensure the comparability of the bases. First, a Socio-Economic Status-SES index was generated composed of the schooling of the head of household and household characteristics: number of rooms, internet connectivity, and the existence of a car. Using Pearson’s correlation, this index has a moderate and significant correlation with the Díaz-Acosta et al. (2015) proposal (0.51 sig at 0.001), and a significant and robust correlation with the Mexican Association of Marketing Research and Public Opinion (Asociación Mexicana de Agencias de Inteligencia y Mercado de Opinión; AMAI, 2018) measurement (0.86 sig at 0.001).

Second, using normal quartiles, four levels were generated: high, medium-high, medium-low, and low SES. Third, we compared means by age between 2019 and 2021 for each SES level, finding no significant differences ( Table 1). For gender we obtained no significance Chi-squared between 2019 and 2021 sample, for each SES level.

Table 1.

SES groups by age and gender.

Socio-Economic Status-SES Group N Age
 Gender
Mean T P % men % women X2 p
Low SES 2019 340 12.1559 1.6797 NS 52.1 47.9 1.06 NS
2021 648 11.9707 48.6 51.4
Middle-Low SES 2019 207 12.1643 1.6808 NS 48.3 51.7 1.396 NS
2021 462 11.9416 53.3 46.8
Middle-High SES 2019 210 12.0571 -0.5509 NS 50.1 49.1 2.261 NS
2021 591 12.1303 48.9 51.1
High SES 2019 336 12.2262 1.7065 NS 52.7 47.3 0.71 NS
2021 300 12.0067 49.3 50.7
Open in a new tab

Source: Authors’ analysis.

2.4. Tools

For reading and mathematics, the MIA tool was applied (Hevia and Vergara-Lope, 2016). The tool is similar to other Citizen-Led-Assessment instruments, like ASER in Asia or UWEZO in Africa, and consists of 10 items: five for reading (Syllables, Words, Sentences, History, Comprehension) and five for mathematics (identification of Numbers, Addition, Subtraction, Division, Problem-solving). For reading, the difficulty of the instruments corresponds to 2nd grade. In mathematics, number recognition, addition and subtraction are aligned to 2nd grade and division and problems to 4th grade. To score the instrument, each item has a value of one point, so the scores range from 0 to 5 in reading and mathematics (Hevia and Vergara-Lope, 2016). The tool’s reliability was calculated using Cronbach’s alpha coefficient; internal consistency indexes of 0.82 in general, 0.78 for reading, and 0.74 for mathematics were obtained, which are considered acceptable levels (Celina Oviedo and Campo Arias, 2005). During the 2021 administration, item 4 (History) was not applied; therefore, the analyses were performed using four reading and five mathematics items.

2.5. Analytic procedures

Diverse descriptive and inferential statistical tests were used for the analysis. First, we used descriptive statics to determine fundamental learning in reading and mathematics. Second, we used Pearson correlations and the Spearman test to identify the factors associated with educational achievement. Third, means were also compared using parametric (Student’s t-test) and non-parametric test (U-Mann-Whitney), and the effect size was calculated using Cohen’s d (Ellis, 2010, Sullivan and Feinn, 2012). These procedures were performed using the statistical package Stata 14.1.

3. Results

The results are presented in three sections: the first shows a big picture of the learning loss and estimates its effect cording by SES; in the second, the increase in learning poverty is estimated; and in the third, comparisons are made according to gender.

3.1. Learning loss in reading and mathematics

In reading, there was a learning loss for all items and at all ages and SES groups analyzed. In this sense, early ages (10 and 11 years) and Low SES show the steepest drops. For example, at age 10, on the Comprehension item, the falls are 25% in Low SES and 15% in High SES. The accumulated results shows that the most critical reduction was on comprehension in every SES quartile (see Table 2).

Table 2.

Comparison of results for each level of reading by year, according to age and SES quartiles (Percentages).

Age Syllable
 Words
 Sentences
 Inferential comprehension
2019 2021 2019 2021 2019 2021 2019 2021
Low SES (N 2019 =340; N 2021 =648)
10 years old 94.3 90.2 94.3 81.7 89.8 65.2 58.0 32.3
11 years old 98.1 91.5 96.2 89.8 88.5 68.6 53.8 38.1
12 years old 98.0 94.2 96.1 89.3 92.2 80.2 66.7 47.9
13 years old 98.2 95.4 98.2 92.6 98.2 82.4 76.8 52.8
14 years old 100.0 100.0 100.0 100.0 98.1 93.5 87.0 75.0
15 years old 97.4 93.3 97.4 88.9 94.9 88.9 76.9 66.7
Mean SES low 97.4 93.7 96.8 89.5 93.2 77.2 68.5 48.1
Middle Low SES (N 2019 =207; N 2021 =462)
10 years old 97.8 95.5 95.6 89.2 93.3 73.9 44.4 43.2
11 years old 100.0 98.9 100.0 95.4 97.1 77.0 74.3 49.4
12 years old 100.0 97.1 100.0 95.1 94.9 92.2 74.4 56.3
13 years old 100.0 100.0 97.3 98.5 97.3 96.9 83.8 73.8
14 years old 100.0 98.6 100.0 97.2 96.2 91.5 84.6 60.6
15 yerars old 95.8 96.0 95.8 96.0 95.8 92.0 87.5 68.0
Mean SES middle-low 99.0 97.6 98.1 94.6 95.6 85.5 72.3 55.6
Middle High SES (N 2019 =210; N 2021 =591)
10 years old 97.6 95.4 97.6 87.7 95.2 76.2 76.2 51.5
11 years old 100.0 95.9 100.0 94.3 96.1 84.4 72.5 53.3
12 years old 100.0 95.6 100.0 94.5 95.1 90.1 82.9 64.8
13 years old 100.0 100.0 100.0 100.0 100.0 97.9 85.7 75.8
14 years old 100.0 98.9 100.0 97.9 96.0 96.8 88.0 76.8
15 yerars old 100.0 96.6 100.0 96.6 100.0 93.1 91.3 75.9
Mean SES Middle high 99.5 97.0 99.5 94.6 96.7 88.5 81.0 64.3
High SES (N 2019 =336; N 2021 =300)
10 years old 98.5 95.8 98.5 93.1 98.5 84.7 72.3 56.9
11 years old 100.0 98.2 100.0 96.5 98.2 87.7 80.7 68.4
12 years old 100.0 100.0 100.0 98.2 97.5 94.5 81.0 67.3
13 years old 100.0 97.9 100.0 97.9 100.0 95.7 88.0 70.2
14 years old 100.0 100.0 97.6 100.0 97.6 98.0 81.0 86.3
15 yerars old 100.0 100.0 100.0 100.0 100.0 100.0 93.0 83.3
Mean SES high 99.7 98.3 99.4 97.0 98.5 92.0 81.8 69.7
Total (N 2019 =1093; N 2021 =2001)
Mean results 98.9 96.3 98.2 93.3 95.6 84.7 75.3 57.9
Open in a new tab

Source: Authors’ analysis.

When we compare reading means, we can first observe a learning loss in all socio-economic levels. And secondly, the losses are largest at the lowest SES ( Fig. 1).

Fig. 1.

Fig. 1

Open in a new tab

Comparison of the reading sum means between 2019 and 2021 by SES level.

Source: Authors’ analysis.

The Table 3 shows a comparison of groups in reading, using the Mann-Whitney test for non-parametric data. There were statistically significant differences between 2019 and 2021 in all the SES levels. Using d-Cohen, we estimate ranging from 0.45 SD at Low SES to 0.34 SD at High SES in reading (see Table 3).

Table 3.

Comparison of reading effect size between 2019 and 2021 by SES level.

SES quartile Groups Obs Mean Diff z d-Cohen 95% conf
Low SES 2019 340 3.56 0.4739*** 6.881 0.4503 0.3175 0.5830
2021 648 3.08
Middle-Low 2019 206 3.65 0.3172*** 4.456 0.3677 0.2022 0.5329
2021 462 3.33
Middle-High 2019 210 3.77 0.3234*** 4.694 0.3797 0.2210 0.5381
2021 591 3.44
High SES 2019 336 3.79 0.2246*** 3.864 0.3466 0.1896 0.5033
2021 300 3.57
Open in a new tab

Note: *** sig. > 0.001. Source: Authors’ analysis.

For mathematics, a similar situation occurred. Losses occurred in all items, at all ages, and all SES levels. The most significant losses were in subtraction and division skills, mainly among low SES, with more than 33 points in Subtraction and 28 points in Division. On average, we observed losses of 16.4% in Addition, 26.9% in Subtraction, 27.5% in Division, and 16.9% in Problem (See Table 4).

Table 4.

Comparison of results for each level of mathematics by year, according to age and SES quartile.

Age Numbers
 Addition
 Subtraction
 Division
 Problem
2019 2021 2019 2021 2019 2021 2019 2021 2019 2021
Low SES (N 2019 =340; N 2021 =648)
10 Years old 96.6 92.7 87.5 59.1 67.0 20.1 34.1 4.3 14.8 0.0
11 Years old 96.2 97.5 86.5 63.6 61.5 32.2 36.5 12.7 11.5 1.7
12 Years old 98.0 95.9 88.2 71.9 66.7 36.4 39.2 17.4 17.6 5.8
13 Years old 98.2 98.1 89.3 69.4 55.4 37.0 41.1 18.5 23.2 9.3
14 Years old 100.0 100.0 96.3 77.2 74.1 40.2 53.7 22.8 35.2 10.9
15 Years old 100.0 95.6 94.9 73.3 79.5 48.9 66.7 26.7 48.7 13.3
Mean SES low 97.9 96.3 90.0 67.6 66.8 33.0 43.2 14.8 23.2 5.4
Middle Low SES (N 2019 =207; N 2021 =462)
10 Years old 100.0 92.8 84.4 65.8 55.6 36.0 26.7 11.7 4.4 5.4
11 Years old 100.0 98.9 91.4 72.4 51.4 39.1 31.4 12.6 8.6 5.7
12 Years old 100.0 97.1 100.0 75.7 64.1 44.7 46.2 27.2 23.1 3.9
13 Years old 97.3 98.5 94.6 80.0 62.2 41.5 45.9 20.0 32.4 6.2
14 Years old 100.0 97.2 92.3 76.1 88.5 46.5 80.8 25.4 38.5 9.9
15 Years old 95.8 96.0 87.5 76.0 70.8 44.0 50.0 24.0 29.2 12.0
Mean SES middle-low 99.0 96.5 91.7 73.4 63.6 41.3 44.2 19.3 20.9 6.3
Middle High SES (N 2019 =210; N 2021 =591)
10 Years old 97.6 90.8 85.7 66.9 66.7 37.7 35.7 11.5 16.7 1.5
11 Years old 98.0 96.7 88.2 74.6 70.6 28.7 45.1 13.1 23.5 7.4
12 Years old 97.6 97.8 87.8 78.0 78.0 48.4 56.1 16.5 17.1 7.7
13 Years old 100.0 98.9 92.9 85.3 71.4 62.1 50.0 40.0 17.9 16.8
14 Years old 100.0 96.8 92.0 83.2 76.0 49.5 68.0 30.5 44.0 12.6
15 Years old 95.7 100.0 87.0 81.0 69.6 55.2 60.9 32.8 47.8 24.1
Mean SES Middle high 98.1 96.3 88.6 77.2 71.9 45.0 50.5 22.3 25.2 10.2
High SES (N 2019 =336; N 2021 =300)
10 Years old 96.9 95.8 93.8 70.8 61.5 37.5 38.5 9.7 13.8 5.6
11 Years old 100.0 96.5 94.7 71.9 77.2 47.4 57.9 29.8 35.1 5.3
12 Years old 97.5 98.2 97.5 85.5 86.1 52.7 63.3 23.6 26.6 5.5
13 Years old 100.0 97.9 92.0 89.4 74.0 57.4 60.0 36.2 32.0 12.8
14 Years old 97.6 100.0 90.5 90.2 73.8 64.7 59.5 37.3 28.6 13.7
15 Years old 100.0 100.0 97.7 88.9 83.7 61.1 58.1 50.0 46.5 22.2
Mean SES high 98.5 97.7 94.6 81.0 76.2 51.3 56.0 27.3 29.2 9.0
Total (N 2019 =1093; N 2021 =2001)
Mean results 98.4 96.6 91.5 73.8 70.2 41.2 48.7 19.9 24.9 7.5
Open in a new tab

Source: Authors’ analysis.

When we compare mathematic means, we can first observe a learning loss in all socio-economic levels. And secondly, the losses are higher at the lowest SES ( Fig. 2).

Fig. 2.

Fig. 2

Open in a new tab

Comparison of the mathematics sum means between 2019 and 2021 by SES level.

(a) Source: Authors’ analysis. (b) Source: Authors’ analysis.

The Table 5 shows a comparison of groups in mathematics, using the Mann-Whitney test for non-parametric data. There were statistically significant differences between 2019 and 2021 in all the SES levels. Using d-Cohen, we estimate ranging from 0.83 SD at Low SES to 0.62 SD at High SES in mathematics (see Table 5).

Table 5.

Comparison of mathematic effect size between 2019 and 2021 by SES level.

SES quartile Groups Obs Mean Diff z d-Cohen 95% conf
Low SES 2019 340 3.2118 1.0405*** 11.4120 0.8275 0.6911 0.9636
2021 648 2.1713
Middle-Low 2019 206 3.1942 0.8262*** 7.2910 0.6550 0.4868 0.8227
2021 462 2.3680
Middle-High 2019 210 3.3429 0.8336*** 7.5370 0.6295 0.4688 0.7897
2021 591 2.5093
High SES 2019 336 3.5446 0.8813*** 8.3610 0.6947 0.5341 0.8548
2021 300 2.6633
Open in a new tab

Note: *** sig. > 0.001. Source: Authors’ analysis.

3.2. Increase in learning poverty

Learning poverty in reading is defined as the inability of a 10-year-old child to read and understand a text adequately (World Bank, 2019a, p. 6). From Fig. 3, three observations can be made: 1) there was a significant increase in learning poverty in all SES quartiles. As Fig. 3 shows, in Low SES, learning poverty increased to 25.7%. In High SES, there was an increase of 15.4% between 2019 and 2021; 2) learning poverty in reading and the differences between in 2019 and 2021 decreased but did not disappear with age: even in 15 years old there was a significant proportion of children who could not read or understand a simple text. For example, in 15 years, increase 10% in both Low SES and High SES, and 33.3% in Low SES and 16.7% in High SES were in a situation of “learning poverty” in 2021; and 3) there was already a significant percentage of poverty in fundamental learning before the pandemic, it increased in 2021 and both, the percentages and the increments are very related with SES conditions. Data from 2019 show gaps in learning poverty between all SES quartiles and the highest learning poverty is in Low SES in 2021.

Fig. 3.

Fig. 3

Open in a new tab

Comparison of percentage of individuals (10 and 15 years old) with learning poverty in reading between 2019 and 2021, by SES levels.

Source: Authors’ analysis.

Although the World Bank does not define poverty of numeracy learning, for this article, we can define it as the percentage of 10-year-old children who cannot solve three-digit division problems without remainders (for example, 210/3). Using this definition, learning numeracy poverty in the analyzed population was ranged from 88.3% to 95.7%, according to socioeconomic quartile. When comparing 2019 and 2021, we see that math learning poverty grew 29.9 points in Low SES, 15% in Middle-Low SES, 24.2% in Middle-High SES, and 28.8% in High SES. Similarly with reading poverty, numeracy poverty decreases with age but at a lower rate. ( Fig. 4).

Fig. 4.

Fig. 4

Open in a new tab

Comparison of percentage of individuals (10 and 15 years old) with learning poverty in mathematics between 2019 and 2021, by SES levels.

Source: Authors’ analysis.

3.3. Differences by gender

Finally, this section presents the comparisons between groups according to gender. The gaps between 2019 and 2021 are compared by gender according to with SES quartiles. As Table 6 shows, in Low SES, both boys and girls presented significant decreases in learning. In mathematics the losses are more unequal, in Low SES girls lost more (0.93 SD), while in High SES boys lost more (0.72 SD) (Table 6).

Table 6.

Comparison of reading and mathematics effect size between 2019 and 2021 by gender and SES level.

Gender Year N Means Diff. z d-Cohen 95% conf
Low SES Reading Girls 2019 163 3.503 0.500*** 4.875 0.450 0.260 0.639
2021 333 3.003
Boys 2019 177 3.610 0.439*** 4.790 0.445 0.258 0.631
2021 315 3.171
Math Girls 2019 163 3.319 1.181*** 8.922 0.931 0.734 1.127
2021 333 2.138
Boys 2019 177 3.113 0.907*** 7.203 0.728 0.538 0.918
2021 315 2.206
High SES Reading Girls 2019 159 3.774 0.260*** 2.926 0.358 0.133 0.581
2021 152 3.513
Boys 2019 177 3.814 0.185* 2.488 0.330 0.110 0.550
2021 148 3.628
Math Girls 2019 159 3.358 0.845*** 5.597 0.663 0.434 0.891
2021 152 2.513
Boys 2019 177 3.712 0.894*** 6.203 0.719 0.493 0.943
2021 148 2.818
Open in a new tab

Note: *** sig. > 0.001; * sig. > 0.05. Source: Authors’ analysis.

4. Conclusion

The results are shown here reflect an educational emergency because of the COVID-19 pandemic and the closure of schools for more than 48 weeks in Mexico. The effects of the pandemic have manifested in a widespread way in both genders, in all the ages studied and for children with different SES.

Although there were already severe lags in fundamental learning in reading and mathematics (2019 measurement), the loss that occurred after more than 12 months of school closures was evident (2021 measurement), and much more severe in the low SES level. The results regarding the learning loss and the increase in learning poverty in reading and mathematics are consistent with both empirical analyses conducted in other parts of the world (Donnely and Patrinos, 2021) and existing estimates and simulations (Azevedo et al., 2020).

These results also show a trend similar to the Comprehensive Learning Assessment (Diagnóstico Integral de los Aprendizajes) conducted by the government of Chile. Furthermore, this study shows that 43% of students are below the expected level in mathematics, and 60% are below the expected level in reading (Agencia de Calidad de la Educación, 2021), confirming that the learning loss due to school closures is a generalized phenomenon.

The magnitude of the learning loss forces governments and society in general to generate an urgent strategy of return to classes that has two priorities: the generation of diagnostic methods, training, and personalized evaluations that give teachers as specific a picture as possible of the diversity of learning levels that they will find when their students return to school. For this purpose, valid and reliable tools, such as those of the MIA Project, can be used to obtain immediate information regarding the reading and mathematics levels of each child with a few minutes of application; such information can be used to make decisions about the needs of a particular child and implement a leveling strategy based on the child’s learning and not on his or her grade in school.

Second, remedial, and leveling courses should be designed and implemented to address this learning loss, as recommended by the United Nations (2020) and prominent international organizations. The “Teaching at the Right Level” (TaRL) strategy seems to be particularly relevant and effective at both the international level (Banerjee, 2016) and in Latin American, where the MIA project has been able to evaluate remedial courses that show positive and significant effects for addressing this lag (Hevia and Vergara-Lope, 2021).

The implementation of remedial courses implies difficulties guaranteeing minimum infrastructure conditions and adequate health safety protocols in each school. In Addition, it is necessary working on comprehensive recovery that does not neglect the socioemotional and mental health situation of teachers, families, and the children themselves.

Comparative experience recommends incorporating modifications to the curriculum to provide greater flexibility to teachers, avoid content saturation and incorporate formative assessment for high-impact decisions, such as student promotion. A recent report from the region highlights that, among the fifteen countries participating in the Latin American Laboratory for Assessment of the Quality of Education, only Brazil, Peru, and Mexico have not modified their curricula in this sense (UNESCO/OREALC, 2021).

In sum, in a comprehensive strategy of recovery, reinforcement, and leveling, diagnostic assessment is the necessary step to identify and remedy setbacks in equity and quality of education and to guide specific, non-homogeneous interventions required to achieve the fundamental learning that was not achieved (UNESCO/OREALC, 2021).

However, the learning lag is not a problem that can be solved in a few weeks or with the use of leveling courses alone; in this sense, together with an emerging strategy, we agree with the United Nations (2020) that there is a need to develop fairer and more egalitarian educational systems that leave no one behind.

Funding

This work was supported by the William and Flora Hewlett Foundation [grant 2018-6968].

Author statement

Conception and design of Study: Felipe J. Hevia, Samana Vergara-Lope. Acquisition of data: Anabel Velásquez-Durán; David Calderón. Analysis and interpretation of data: Felipe J. Hevia, Samana Vergara-Lope. Drafting the manuscript: Felipe J. Hevia, Samana Vergara-Lope. Revising the manuscript critically for important intellectual content: Anabel Velásquez-Duran, David Calderón. Approval of the version of the manuscript to be published: Felipe J. Hevia, Samana Vergara-Lope, Anabel Velásquez-Durán, David Calderón.

Acknowledgments

We would like to thank the "Instituto Campechano" and "IEPAC" who helped recruit and train volunteers for the data collection in 2019.

References

  1. Agencia de Calidad de la Educación, 2021. Resultados Diagnóstico Integral de Aprendizaje 2021. Agencia de Calidad de la Educación. 〈https://www.mineduc.cl/en-ensenanza-media-no-se-alcanzo-el-60-de-los-aprendizajes-minimos/〉.
  2. AMAI, 2018. Nivel Socio Económico AMAI 2018. Asociación Mexicana de Agencias de Inteligencia y Mercado de Opinión. 〈https://www.amai.org/nse/wp-content/uploads/2018/04/Nota-Metodolo%CC%81gico-NSE-2018-v3.pdf〉.
  3. Arellano, S. , 2020, septiembre 17. El inmenso reto de frenar el abandono escolar. Mexico Social. 〈https://www.mexicosocial.org/el-inmenso-reto-de-frenar-el-abandono-escolar/〉.
  4. Azevedo, J.P., Hasan, A., Goldemberg, D., Iqbal, S.A., Geven, K. , 2020. Simulating the Potential Impacts of COVID-19 School Closures on Schooling and Learning Outcomes: A Set of Global Estimates (Policy Research Working Paper Núm. 9284). The World Bank. 10.1596/1813-9450-9284. [DOI]
  5. Azevedo, J.P., & Montoya, S. , 2021, March 24. Building Back Better After COVID-19: The Importance of Tracking Learning Inequality. Data for Sustainable Development. 〈https://sdg.uis.unesco.org/2021/03/24/building-back-better-after-covid-19-the-importance-of-tracking-learning-inequality/〉.
  6. Banerjee P.A. A systematic review of factors linked to poor academic performance of disadvantaged students in science and math’s in schools. Cogent Educ. 2016;3(1) doi: 10.1080/2331186X.2016.1178441. [DOI] [Google Scholar]
  7. Banerji R. The Birth of ASER. Learn. Curve. 2013;2013:85–88. [Google Scholar]
  8. Banerji R. When schooling doesn’t mean learning. Stanf. Soc. Innov. 2017:4. [Google Scholar]
  9. Banerji R., Bhattacharjea S., Wadhwa W. The Annual Status of Education Report (ASER) Res. Comp. Int. Educ. 2013;8(3):387. doi: 10.2304/rcie.2013.8.3.387. [DOI] [Google Scholar]
  10. CEES . Centro de Estudios Educativos y Sociales; 2021. Nota técnica cálculos de muestra Yucatán y Campeche 2021. [Google Scholar]
  11. Celina Oviedo H., Campo Arias A. Aproximación al uso del coeficiente alfa de Cronbach. Rev. Colomb. Psiquiatr. 2005;34(4):572–580. [Google Scholar]
  12. Díaz-Acosta R., Shiba-Matsumoto A.R., Gutiérrez J.P. Medición simplificada del nivel socioeconómico en encuestas breves: Propuesta a partir de acceso a bienes y servicios. Salud Públ. México. 2015;57(4):298–303. [PubMed] [Google Scholar]
  13. Donnely R., Patrinos H. Learning loss during COVID-19: an early systematic review. Covid Econ. 2021;77(30):145–153. doi: 10.1007/s11125-021-09582-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Ellis P.D. The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results. 2010. [Google Scholar]
  15. Engzell, P., Frey, A., & Verhagen, M.D. , 2020. Learning Loss Due to School Closures During the COVID-19 Pandemic. SocArXiv. 10.31235/osf.io/ve4z7. [DOI] [PMC free article] [PubMed]
  16. Fore, H. , 2021, enero 12. Los niños y niñas no pueden permitirse otro año sin escuela. UNICEF. 〈https://www.unicef.org/lac/comunicados-prensa/los-ni%C3%B1os-y-ni%C3%B1as-no-pueden-permitirse-otro-a%C3%B1o-sin-escuela〉.
  17. GEM-UNESCO, 2020. Act now: Reduce the impact of COVID-19 on the cost of achieving SDG 4 (Policy paper Núm. 42; ED/GEM/MRT/2020/PP/42REV). UNESCO.
  18. Gore J., Fray L., Miller A., Harris J., Taggart W. The impact of COVID-19 on student learning in New South Wales primary schools: an empirical study. Aust. Educ. Res. 2021 doi: 10.1007/s13384-021-00436-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Hernández Sampieri R., Fernández C., Baptista P. McGraw-Hill; 2001. Metodología de la investigación. [Google Scholar]
  20. Hevia F.J., Vergara-Lope S. Evaluaciones educativas realizadas por ciudadanos en México: Validación de la Medición Independiente de Aprendizajes. Innov. Educ. 2016;16(70):85–110. [Google Scholar]
  21. Hevia, F.J., Vergara-Lope, S. , 2019. Educational Accountability or Social Accountability in Education? Similarities, Tensions, and Differences (Working Paper Núm. 5; p. 64). Medición Independiente de Aprendizajes; Accountability Research Center. 〈https://accountabilityresearch.org/publication/educational-accountability-or-social-accountability-in-education-similarities-tensions-and-differences/〉.
  22. Hevia F.J., Vergara-Lope S. Particularidades, límites y potencialidades de las evaluaciones dirigidas por ciudadanos en América Latina. Educ. Educ. 2020;15(35):1–20. doi: 10.17648/educare.v15i35.23918. [DOI] [Google Scholar]
  23. Hevia, F.J., Vergara-Lope, S. , 2021. Efectos de cursos remediales en aprendizajes y equidad durante la COVID-19 en México. En dictamen.
  24. Huong, L.T., Na-Jatturas, T., Coronavirus, 2020 |, & Learning | 0. (2020, mayo 18). The COVID-19 induced learning loss – What is it and how it can be mitigated? The Education and Development Forum. 〈https://www.ukfiet.org/2020/the-covid-19-induced-learning-loss-what-is-it-and-how-it-can-be-mitigated/〉.
  25. INEGI, 2015. Encuesta, Intercensal 2015. 〈http://www.beta.inegi.org.mx/proyectos/enchogares/especiales/intercensal/〉.
  26. INEGI, 2021a. Comunicado de prensa núm. 97/21 29 de enero de 2021. Instituto Nacional de Geografía y Estadística.
  27. INEGI, 2021b. Encuesta para la Medición del Impacto COVID-19 en la Educación (ECOVID-ED) 2020. Nota técnica (p. 29). INEGI.
  28. Iqbal, S.A., Azevedo, J.P., Geven, K., Hasan, A., Patrinos, H.A. , 2020) We should avoid flattening the curve in education – Possible scenarios for learning loss during the school lockdowns. World Bank Blogs. 〈https://blogs.worldbank.org/education/we-should-avoid-flattening-curve-education-possible-scenarios-learning-loss-during-school〉.
  29. John Hopkins University , 2021. COVID-19 Map. Johns Hopkins Coronavirus Resource Center. 〈https://coronavirus.jhu.edu/map.html〉.
  30. Kaffenberger M. Research on Improving Systems of Education (RISE); 2020. Modeling the Long-Run Learning Impact of the COVID-19 Learning Shock: Actions to (More Than) Mitigate Loss. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Kerlinger F.N., Lee H.B. McGraw-Hill; 2002. Investigación del comportamiento: Método de investigación en ciencias sociales. [Google Scholar]
  32. Kuhfeld M. Surprising new evidence on summer learning loss. Phi Delta Kappan. 2019;101(1):25–29. doi: 10.1177/0031721719871560. [DOI] [Google Scholar]
  33. Kuhfeld M., Tarasawa B., Johnson A., Ruzek E., Lewis K. NWEA; 2020. Learning During COVID-19: Initial Findings on Students’ Reading and Math Achievement and Growth.〈https://www.nwea.org/research/publication/learning-during-covid-19-initial-findings-on-students-reading-and-math-achievement-and-growth/〉 [Google Scholar]
  34. Maldonado, J.E., Witte, K.D. , 2021. The Effect of School Closures on Standardized Student Test Outcomes (Discussion Paper DPS 20.27; p. 49). Ku Leuvan. Department of Economics.
  35. Mejía L.S.P., Fernández J.L.W., Hernández I.O., Ridaura R.L., Ramirez H.L.-G., Avila M.H., Ávila J.E.H., Mortalidad G., interinstitucional para la estimacion del exceso de Estimación del exceso de mortalidad por todas las causas durante la pandemia del Covid-19 en México. Salud Públ. México. 2021;63(2, Mar–Abr):211–224. doi: 10.21149/12225. [DOI] [PubMed] [Google Scholar]
  36. MIA, 2019. Resultados medición 2018. Yucatán, Campeche y Quintana Roo. CIESAS; UV.
  37. Mugo J.K., Ruto S.J., Nakabugo M.G., Mgalla Z. A call to learning focus in East Africa: Uwezo’s Measurement of Learning in Kenya, Tanzania and Uganda. Afr. Educ. Rev. 2015;12(1):48–66. doi: 10.1080/18146627.2015.1036564. [DOI] [Google Scholar]
  38. Munene I.I., editor. Achieving Education for All: Dilemmas in System-wide Reforms and Learning Outcomes in Africa. Lexington Books; 2016. [Google Scholar]
  39. PAL, 2018. PAL Network – Peoples Action for Learning Network. 〈https://palnetwork.org/?lang=es〉.
  40. Pérez, V.H., Gaitan, P. , 2020. Usos desiguales de “Aprende en casa” (Apunte de política Núm. 18; p. 8). Universidad Iberoamericana. Faro educativo.
  41. Pier, L., Hough, H., Christian, M., Bookman, N., Wilkenfeld, B., Miller, R. , 2021, enero 25. COVID-19 and the Educational Equity Crisis Evidence on Learning Loss from the CORE Data Collaborative. Policy Analysis for California Education. 〈https://edpolicyinca.org/newsroom/covid-19-and-educational-equity-crisis〉.
  42. Pritchett L. Center for Global Development; 2013. The Rebirth of Education: Schooling Ain’t Learning. [Google Scholar]
  43. Rieble-Aubourg S., Viteri A. IDB; 2020. Educación más allá del COVID-19; p. 20. [Google Scholar]
  44. Ruto, S. , 2013. Information on learning outcomes as a catalyst to spur citizens to act: Examples from Tanzania, Uganda and Kenya. In: The Right to Learn. Community Participation in Improving Learning (pp. 19–21). Save The Children.
  45. Schult J., Mahler N., Fauth B., Lindner M.A. Did students learn less during the COVID-19 Pandemic? Reading and mathematics competencies before and after the first pandemic wave. PsyArXiv. 2021 doi: 10.31234/osf.io/pqtgf. [DOI] [Google Scholar]
  46. SEP. (2020a). Boletín No. 136 Logra Aprende en Casa que 9 de cada 10 niñas y niños mantengan su aprendizaje: SEP. 〈http://www.gob.mx/sep/es/articulos/boletin-no-136-logra-aprende-en-casa-que-9-de-cada-10-ninas-y-ninos-mantengan-su-aprendizaje-sep?idiom=es〉.
  47. SEP. (2020b). Iniciará el Ciclo Escolar 2020–21 con el modelo de aprendizaje a distancia Aprende en Casa II: Esteban Moctezuma (Boletín de prensa Núm. 205). SEP. 〈http://www.gob.mx/sep/articulos/boletin-no-205-iniciara-el-ciclo-escolar-2020–21-con-el-modelo-de-aprendizaje-a-distancia-aprende-en-casa-ii-esteban-moctezuma?idiom=es〉.
  48. Sullivan G.M., Feinn R. Using effect size—Or why the P value is not enough. J. Grad. Med. Educ. 2012;4(3):279–282. doi: 10.4300/JGME-D-12-00156.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Tomasik M.J., Helbling L.A., Moser U. Educational gains of in-person vs. distance learning in primary and secondary schools: a natural experiment during the COVID-19 pandemic school closures in Switzerland. Int. J. Psychol. 2020 doi: 10.1002/ijop.12728. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. UNESCO . COVID-19 Impact on Education. 2021. Global monitoring of school closures caused by COVID-19.〈https://en.unesco.org/covid19/educationresponse〉 [Google Scholar]
  51. UNESCO/OREALC . UNESCO-OREALC; Santiago: 2021. A un año del comienzo de la pandemia: Continuidad educativa y evaluación en América Latina y el Caribe en 2021.〈https://unesdoc.unesco.org/ark:/48223/pf0000377802〉 [Google Scholar]
  52. United Nations, 2020. Education during COVID-19 and beyond [Policy Brief]. United Nations. 〈https://www.un.org/es/coronavirus/articles/future-education-here〉.
  53. Vergara-Lope, S., Hevia, F.J. , 2016. La Reforma Educativa: ¿Ha dado resultados? Construcción de línea base para una evaluación independiente de aprendizajes y factores asociados al logro educativo. En Premio Nacional de Investigación Social y de Opinión Pública 2016 (pp. 49–95). CESOP-Cámara de Diputados.
  54. Vergara-Lope S., Hevia F.J., Rabay V. Evaluación alternativa de competencias básicas de lectura y aritmética y análisis de factores asociados en Yucatán, México. Revi. Iberoam. Eval. Educ. 2017;10(1) doi: 10.15366/riee2017.10.1.XXX. [DOI] [Google Scholar]
  55. Wilson, H.-M. , 2018, octubre 26. How Civil Society Can Supply Rigorous Data for the SDGs: The Citizen-Led Assessment Approach. Data for Sustainable Development. 〈https://sdg.uis.unesco.org/2018/10/26/how-civil-society-can-supply-rigorous-data-for-the-sdgs-the-citizen-led-assessment-approach/〉.
  56. World Bank, 2019a. Learning Poverty [Text/HTML]. World Bank. 〈https://www.worldbank.org/en/topic/education/brief/learning-poverty〉.
  57. World Bank . World Bank; 2019. Mexico. Learning Poverty Brief.〈https://www.worldbank.org/en/topic/education/brief/learning-poverty〉 [Google Scholar]
  58. World Bank, 2020. The COVID-19 Pandemic: Shocks to Education and Policy Responses. World Bank. 10.1596/33696. [DOI]
  59. World Bank, 2021. Acting Now to Protect the Human Capital of Our Children: The Costs of and Response to COVID-19 Pandemic’s Impact on the Education Sector in Latin America and the Caribbean. World Bank. 10.1596/35276. [DOI]

Articles from International Journal of Educational Development are provided here courtesy of Elsevier

RESOURCES