COVID-19 and inequalities in educational achievement in Italy

Abstract

We use longitudinal data from over 1.5 million Italian students to examine differences in the mathematics and reading achievement of students who completed primary and lower secondary school in 2020–21 (COVID cohort) and those who completed it in 2018–19 (non-COVID cohort). We also examine the evolution of inequalities by gender, socio-economic condition, and prior academic achievement during the pandemic. On average, the primary school COVID cohort experienced a small increase in reading achievement and a drop in mathematics achievement compared to the non-COVID cohort. The lower secondary school COVID cohort experienced a large reduction in mathematics achievement and a smaller reduction in reading achievement compared to the non-COVID cohort. Previously middle-achieving students suffered the most from the pandemic, while high achievers gained. Socio-economic inequalities in achievement remained stable for secondary school students and somewhat decreased for primary school students between the non-COVID and COVID cohorts. Gender disparities were broadly reduced across domains and school levels, except for primary school math

1. Introduction

The COVID pandemic caused significant disruptions to the everyday lives of people around the world. In an attempt to limit the spread of the disease and protect children’s right to education, governments imposed school closures but maintained teaching and learning through remote learning (OECD, 2021a). Despite marked increases in the past decade in the adoption of Information and Communication Technologies (ICTs) in education (Borgonovi & Pokropek, 2021), many schools struggled to provide adequate online learning solutions (Kuhfeld et al., 2020). Teachers often had little preparation for remote teaching and many children, especially students from socio-economically disadvantaged households and those living in deprived communities, lacked access to equipment or connectivity (World Bank et al., 2021).

Other burdens imposed by the pandemic compounded disruptions from school closures and the difficult transition to distance learning. Economic uncertainty, unemployment, and changing working arrangements had repercussions on the ability of many parents to supervise and provide for their children’s needs (Adams-Prassl et al., 2020). Many children with a disadvantaged condition faced learning disruptions because of social isolation (Zaccoletti et al., 2020) and poor mental and physical health (Golberstein et al., 2020). Finally, there is evidence that school closures led to reduced learning time, especially among low achievers and socio-economically disadvantaged students (Andrew et al., 2020, Bayrakdar and Guveli, 2020, Grätz and Lipps, 2021, Grewenig et al., 2021).

We contribute to the emerging literature on the educational consequences of the pandemic [see (Thorn & Vincent-Lancrin, 2021) and (Hammerstein et al., 2021) for reviews and (Betthäuser et al., 2022) for a meta-analysis of previous studies] by investigating the case of Italy, one of the countries that were hit first and the hardest by COVID (Johns Hopkins, 2022), and where schooling was severely disrupted (World Bank et al., 2021). We examine differences in the academic achievement of primary and secondary school students who experienced pandemic-related disruptions during the first two waves of the pandemic (between March 2020 and May 2021) and the achievement of those who completed primary and lower secondary school just before the pandemic hit using longitudinal population data from the Italian National Institute for the Evaluation of the Education System (INVALSI). Primary and lower secondary schools remained closed from March 2020 until the end of the 2019–2020 academic year and, in some communities, for long stretches of the 2020–21 academic year. Lower secondary schools closed for longer than primary schools.

We are aware of two studies to date that estimate how academic achievement evolved following the pandemic in Italy: Contini et al., (2022) estimated the effects of the pandemic on the mathematics achievement of primary school children in the city of Turin. Bazoli et al., (2022) compared the overall level of reading and mathematics achievement of samples of Italian students from primary, lower secondary and upper secondary schools before and after the pandemic. They also estimated differences across different levels of socio-economic condition (SES). Our contribution differs from these studies in three key respects. First, we use population-level data covering all of Italy. Second, we measure differences in achievement between the cohorts of students who experienced COVID-related disruptions and those who did not, as well as differences by socio-economic status (SES), gender and students’ previous academic achievement. Finally, we explore the outcomes of academically resilient students, such as previously high-achieving but low-SES students (Agasisti et al., 2018), girls with high achievement in mathematics, and boys with high achievement in reading.

Like in many other countries, socio-economically advantaged students in Italy achieve at higher levels than their disadvantaged counterparts. Socio-economic inequalities arise early and are already established before children complete primary school (Mullis et al., 2017, Mullis et al., 2020). Gender differences are also marked in Italy, and they tend to emerge earlier than in other countries (Contini et al., 2017). We investigate SES and gender inequalities in the cohorts who experienced the pandemic and whose who did not also net of students’ previous achievement.

2. Background

2.1. The effect of COVID on reading and mathematics achievement

The pandemic disrupted the learning of virtually all students worldwide (World Bank et al., 2021). In the 2019–2020 academic year, schools in Italy closed in March 2020 and remained closed until the end of the academic year. In this period, children were also exposed to anxiety, stress and uncertainty, factors that the literature indicates severely reduce individuals’ ability to learn (Fegert et al., 2020, Vogel and Schwabe, 2016). Even when schools reopened in September 2020, teaching and learning continued to be disrupted: teaching was often conducted in hybrid formats, and many children and teachers missed classes because of illness or contact with infected individuals.

Evidence from a meta-analysis of studies on learning losses associated with COVID from 12 countries indicates that the pandemic had overall negative effects on achievement during the first phase of generalised school closures, and that, between June 2020 and October 2021 (the latest time point covered by such studies) negative effects persisted (Betthäuser et al., 2022). Evidence from the Northern Italian city of Turin on the learning losses experienced by primary school children in the period of generalised school closures during the first phase of the pandemic identified a strong negative effect of COVID and a larger effect among children with parents with lower educational qualifications and previously highest achievers (Contini et al., 2022). However, such study evaluated only short-term effects when schools and families had little opportunity to put in place remedial interventions and where readiness to engage in online learning varied markedly (Gavosto & Romano, 2020). Furthermore, the quality of schooling before the pandemic hit varied markedly in Italy and Turin is not representative of the broader Italian population (Agasisti, 2013, Matteucci and Mignani, 2014). As a result, the overall effects of the pandemic might have been different in areas where schooling was less conducive to learning before COVID forced schools to close.

We expect the impact of the pandemic to be less pronounced among younger children because primary schools closed for less time than secondary schools (UNESCO, 2022). Furthermore, although it is possible that younger children were less autonomous in the use of remote learning technologies, young children in primary school were more likely to be closely supervised and monitored by their parents than older secondary school students (Scarpellini et al., 2021). Even with schools open, many adults in Italy continued to engage in remote working throughout 2020 and 2021. As such, smaller children might have had more time to interact on a one-to-one basis with an adult, doing a range of activities and practicing their language skills.

We expect the effects of the pandemic on achievement to also differ by domain and age. At young ages, language practice can take place in many settings and instructors require little technical knowledge, so parents can more easily assist their child’s learning. By contrast, teaching mathematics is generally considered to be more formal and structured and parents are less likely to engage in practices designed to promote numeracy development among children (Napoli et al., 2021). Furthermore, particularly for secondary school math, helping a child who struggles with a concept or process requires greater technical knowledge (OECD, 2010). It is therefore possible that school closures weighted more heavily on mathematics achievement than reading achievement, especially for lower secondary school students.

Finally, we expect the effects of the pandemic to differ by levels of prior achievement. On the one hand, given the cumulative nature of knowledge acquisition (DiPrete & Eirich, 2006), we expect the effects to be especially negative for initially lower-performing students. Schools and teachers often help students reduce learning gaps that they accumulated earlier but doing so could have been more challenging with school disruptions. Furthermore, a low level of initial achievement could be an indicator that students had low levels of academic motivation and engagement with learning, which could have been especially penalizing during the pandemic. On the other hand, low achievers could have particularly benefitted from one-to-one supervision from their parents during the pandemic, which they may lack during normal schooling. In normal times, teachers may also tend to target the median learner in the classroom, rather than focusing on the very high and very low achievers, the main argument of proponents of tracking and ability grouping in education (Duflo et al., 2011, Fu and Mehta, 2018). So we could also expect students with average pre-pandemic achievement to be those most negatively impacted by school disruptions.

2.2. The effect of COVID on socio-economic disparities in achievement

The effect of the pandemic on SES disparities in achievement depends on the extent to which schooling was contributing to narrowing these disparities in normal times as well as the extent to which family responses to school disruptions varied by SES. The literature documents that, in normal times, socio-economic disparities in academic achievement partly arise because of disparities in the investments families make in their children’s development (Conger & Donnellan, 2007). An advantaged socio-economic status allows families to invest more material, cultural and social resources in their children’s development. High-SES children often benefit from higher-quality parenting practices (Schaub, 2010), from greater parental involvement in their academic decisions (Domina, 2005), from participation in academic settings of higher quality, and from more enriching out-of-school experiences (Lareau, 2002).

Given disparities in parental investments, schools can contribute to narrowing or magnifying such disparities (Downey et al., 2018, Downey and Condron, 2016, Marks et al., 2006, Passaretta and Skopek, 2021). Historically the achievement of students has reflected, to a large extent, the socio-economic condition of their family of origin [see for example (Coleman et al., 1966, Jencks et al., 1972) and comprehensive reviews such as (Sirin, 2005)]. At the same time, proponents of the compensatory hypothesis maintain that schools play an equalising role and reduce differentials in educational outcomes. Significant empirical evidence supports the compensatory hypothesis (Downey et al., 2018, Downey and Condron, 2016) but recent scholarship has found the equalization effects of schooling to be weak or null (Passaretta and Skopek, 2021, von Hippel et al., 2018).

The pandemic might have equalised the quality of schooling inputs by lowering the quality experienced by the most well-off so that it matched the quality experienced by the least well-off. In this case, students who benefited from higher-quality schooling before the pandemic would be expected to experience the largest learning losses due to the pandemic. However, reduced schooling implies a greater salience of home learning environments, which, in normal circumstances, tends to favour high-SES students (Conger & Donnellan, 2007). The pandemic could therefore have led to less negative outcomes among high-SES students because of their parents’ greater ability to invest time and knowledge in their education (Lareau, 2002, Schaub, 2010) and the greater availability of remote learning equipment among the more well-off (Francis and Weller, 2022, OECD, 2020). The relevance of parental investments might have been especially important among younger children, since parents were more likely to closely supervise and monitor them during the pandemic (Scarpellini et al., 2021).

The pandemic could also have increased SES inequalities through its effect on prior achievement. Higher SES students typically have higher academic achievement than lower SES students (Skopek & Passaretta, 2021). Having acquired a more solid educational foundation (including knowledge and learning habits), higher-achieving students might have been able to continue their learning at a more standard pace during the pandemic. Finally, it is possible that not only the effects of the pandemic on SES disparities reflect underlying differences in achievement across different SES groups, but also that the effects of the pandemic were not the same for groups of different SES and achievement (multiplication of disadvantage).

The theory of compensatory advantage (Bernardi, 2014) states that high-SES parents are better able to compensate for any initial disadvantage their children experience and therefore that SES disparities are widest among low achievers. According to the theory, parental resources during the pandemic might have mattered most for low achievers, while among high achievers parental SES might have made less of a difference leading to the prediction of widening SES disparities especially at the bottom of the achievement distribution. At the same time, some of the compensation mechanisms predicted by the theory occur through schooling and were likely curbed by school disruptions, thus limiting differential effects of SES among the lowest achievers. The theory also implicitly suggests that low-SES students who were at the top end of the achievement distribution before the pandemic were likely to be more positively selected in terms of unobserved talent or non-cognitive skills compared to high-SES students. As a result, low-SES high achievers might have been less likely to have been impacted by school closures than their high-SES peers. Overall, at low levels of achievement, we expect that the pandemic had a larger negative effect for low-SES students while at high levels, we expect less negative effects for low-SES than high-SES students.

2.3. The effect of COVID on gender disparities in achievement

Pandemic-related disruptions could have shaped gender gaps in reading and mathematics by influencing the level of effort boys and girls put in learning different academic subjects. The literature indicates that, in school, girls have higher levels of conscientiousness and intrinsic academic motivation (De Bolle et al., 2015, Matthews et al., 2009) and that they report greater intrinsic satisfaction from doing well at school than boys do (Di Prete & Buchmann, 2013). As such, it is possible that the pandemic caused fewer disruptions to the learning of girls because they were able to organise their work without the strict supervision of teachers, engaging in learning even in the absence of tests and marks. We therefore expect that the overall effects of the pandemic will be less negative for girls than for boys. Moreover, if school disruptions had especially negative effects among low achievers, we expect girls to be impacted less, since they are under-represented among the lowest achieving students (Muthukrishnan & Rohini, 2016).

The beliefs that parents, teachers and peers hold have the potential to shape how well boys and girls do in different academic subjects (Bhanot and Jovanovic, 2009, Cornwell et al., 2012, Fryer and Levitt, 2010, Robinson-Cimpian et al., 2014). By disrupting schooling, the pandemic could have widened boys’ advantage in mathematics and girls’ advantage in reading if teachers and educators were acting in ways that reduced the influence of gender stereotypes before the pandemic hit. By contrast, it might have led to a reduction in the extent to which stereotypes shape achievement differentials if schools reinforced stereotypes through peer influence and teaching practices (Carlana, 2019, Keller, 2001). Peer effects may be particularly significant for boys, because the literature suggests that peer-pressure is a very strong driver of behavior among boys, leading boys to adopt a concept of masculinity founded on a disregard for authority, academic work, and formal achievement (Legewie and DiPrete, 2012, Salisbury et al., 1999, Van Houtte, 2004). Students who defied stereotypes before the pandemic - high-achieving girls in mathematics and high-achieving boys in reading - could have benefited during the pandemic if COVID disrupted negative peer influences children usually experience at school. By contrast, they could have suffered severe negative effects if their achievement was due to the presence of supportive teachers and peers at school.

3. Data and methods

3.1. Data and analytical sample

We use data from the Italian National Institute for the Evaluation of the Education System (INVALSI). INVALSI administers annual population-level evaluations of students’ achievement in reading and mathematics. The assessments take the form of an annual census administered in the Spring, and participation is compulsory for all students attending grades 2, 5 and 8 (as well as grades 10 and 13 which we do not use in this work).1 The INVALSI assessments are low-stakes standardised assessments since results have no bearing for students’ academic progression or grades. We analyse students’ results in grade 5 and 8 assessments, which represent the end of primary and lower secondary (compulsory) education in Italy. We refer to students sitting the grade 5 assessment as the “primary school” cohort and to those sitting the grade 8 assessment as the “secondary school” cohort.

All data used in this work are available for researchers from INVALSI (see https://invalsi-serviziostatistico.cineca.it/). Access to the data is subject to registration and the submission of a research protocol specifying how the data will be used. All codes used in the analyses can be requested from the authors after interested researchers have registered on the INVALSI platform and their access to the microdata has been approved by INVALSI.

The INVALSI assessment was not administered in 2019–2020 because schools were closed. Therefore, we focus on the primary and secondary school cohorts who sat the grade 5 and grade 8 INVALSI tests in June 2021 (henceforth “COVID cohorts”). These cohorts experienced fully remote schooling in the last part of academic year 2019–20 as well as localised closures, possible quarantines, and sickness in academic year 2020–21. We compare the achievement of COVID cohorts with the achievement of the last cohorts of students to be assessed by INVALSI before the pandemic - the primary and secondary school cohorts who sat the grade 5 and grade 8 INVALSI assessment in June 2019 (henceforth “non-COVID cohorts”).

In addition to INVALSI data from 2020 to 2021 and 2018–2019, we also use data from the 2017–2018 and 2015–2016 tests to measure students’ previous scores in INVALSI. Since students normally sit INVALSI assessments every three years (in grade 2, grade 5, grade 8) and INVALSI data are longitudinal, students can be matched with their scores from the assessment they sat three years before. For example, for students in the secondary school COVID cohort, who sat the INVALSI grade 8 assessment in 2021, we obtained their performance in the grade 5 assessment that they had completed in 2018. Table 1 below represents the different cohorts and the INVALSI waves used to measure their outcome and baseline scores.

Table 1.

Description of Cohorts and Designs.

Academic year	COVID cohorts	Non-COVID cohorts
2020–2021	Outcome period (grades 5 and 8)
2019–2020
2018–2019		Outcome period (grades 5 and 8)
2017–2018	Baseline period (grades 2 and 5)
2016–2017
2015–2016		Baseline period (grades 2 and 5)

Our analytical sample only includes students for whom we could obtain prior INVALSI scores, so grade 2 scores for students we observe in grade 5, and grade 5 scores for students we observe in grade 8. In other words, our sample consists of students whose observed academic progress over a three-year cycle matched their expected progression. This information was not available for 12–17% of the students in the original sample (see Table A1 in the Online Supplementary Material). Missing information could be due to the fact that students who are missing on the day of the exam (for example because of sickness) are not asked to re-sit it, since the INVALSI tests are not high stakes for students and the information value of the tests is at the population level. These students can be effectively considered as missing at random. However, missing information could also be due to the fact that in Italy students with failing grades are retained and have to repeat certain grades. Students with missing prior INVALSI scores due to grade repetition would be low-achieving students who were retained at least once, and had sat the prior INVALSI test 4 or more years before rather than the expected 3 years before. In Italy, virtually no student skips a grade so there is no counter-phenomenon of missing information due to students advancing at a faster pace than expected. In our analyses, we drop all students whose prior achievement three years before could not be traced, so we effectively only consider students who were on track between consecutive INVALSI assessment rounds.

Our decision is mainly motivated by concerns around differential grade retention across COVID and non-COVID cohorts. As a response to schooling disruptions, at the end of the 2019–2020 academic year, all students moved up a grade and no grade retention was implemented (MIUR, 2021). Grade retention is prevalent in Italy. Although it is not widespread at the primary and lower secondary level (OECD, 2020), differences in grade repetition across the COVID and non-COVID cohorts could create two issues of selection. The first issue is that some low-performing students who were in grade 5 or grade 8 in the 2019–2020 academic year moved up a grade (i.e. were not retained) as a result of the policy, and therefore did not become part of the COVID cohorts as would have been the case had the policy not been implemented (since they did not end up sitting the INVALSI grade 5 and grade 8 exams in 2021). As a result, the COVID cohorts would be positively selected compared to the non-COVID ones since they should include less students who have been retained (and whose INVALSI scores from three years before could therefore not be traced). This is confirmed in our data (see Table A2 in the Online Supplementary Material).

We address this issue by dropping students whose INVALSI scores three years before could not be traced in the COVID and non-COVID sample. In other words, to make the samples more comparable, we focus only on students who had not been retained between consecutive INVALSI rounds. Our estimates show that, unsurprisingly, students in the sample were positively selected in terms of INVALSI scores and SES compared to those excluded because their performance could not be traced, since the latter were more likely to have been retained in school (see Table A3 in the Online Supplementary Material). We note, however, that the majority of students whose INVALSI scores three years before could not be traced were simply missing on the day of the examination and had not been retained. Most importantly, Table A3 shows that the extent to which in-sample students were positively selected compared to excluded students is comparable between COVID and non-COVID cohorts. The only difference is that the secondary school COVID cohort was slightly more positively selected in terms of math scores.

The second selection issue created by the change in retention policy is that some low-performing students who were in grade 4 or grade 7 in 2019–2020 and would have been retained in the absence of the policy instead moved up a grade (i.e. were not retained) and so became part of the COVID cohorts (since they ended up sitting the INVALSI grade 5 and grade 8 exams in 2021). This implies that among students who were not retained between consecutive INVALSI rounds (and are thus part of our analytical sample), the COVID cohorts should be more negatively selected and include more students with low prior INVALSI scores. To address these compositional differences induced by the policy we control for students’ prior performance in all our analyses. Moreover, on top of restricting the sample to students whose progress we could track and controlling for their prior achievement, we also introduce an additional control for students’ self-reported academic regularity in all our specifications. In this way, we also control for possible differences in the prevalence of students repeating a grade outside of the relevant period of the analyses. The indicator for prior repetition of a grade also allow us to control for potentially unobserved trends across the two cohorts in achievement at the left tail of the achievement distribution.

In addition to comparability concerns, the choice to restrict the sample to students whose prior performance could be traced is driven by the fact that we are interested in investigating heterogeneities across students’ prior achievement. Table A1 in the Online Supplementary Material shows how analytical samples were obtained from initial samples by progressively dropping students with missing information for key variable. Our final samples are composed of about 830,000 primary students and 880,000 secondary students, out of which about half were in the COVID cohort. The size of the samples for analyses on math and reading scores vary slightly because the assessments were administered on different days and some students were absent on one of the two assessment days.

3.2. Variables

Our outcome variables are INVALSI reading and math scores. The psychometric design of the INVALSI tests changed over time. In particular, tests administered to primary school children can be reliably compared over time from academic year 2018–19 onwards while for secondary school children they can be reliably compared over time from academic year 2017–18 onwards. In the anchor year, each scale was set to have a mean of 200 and a standard deviation of 40. This means that on the outcome measure (achievement in academic years 2018–19 and in academic years 2020–21), scores obtained by students in the COVID and non-COVID cohorts are directly comparable (for example an average score of 196 in the COVID cohort and an average score of 200 in the non-COVID cohort would imply that the average score of students in the COVID cohort was 4 points lower – 10% of a SD - than the average score of students in the non-COVID cohort and this difference would be directly comparable to the difference observed across two students from the same cohort obtaining scores of 196 and 200). By contrast, prior achievement scores are not comparable across cohorts since we use data from 2015-16 for the non-COVID cohort and 2017–18 for the COVID cohort, before stable anchors were established.

Because of these changes, the outcome achievement variables and the prior achievement variables are not directly comparable. In our analyses we transformed the outcome measures of reading and mathematics achievement such that they had a mean of 0 and a SD of 1 in the anchor year and differences across the two cohorts would indicate the SD difference in scores before and after COVID. By contrast, we converted prior achievement measures into deciles, defining decile thresholds using the full sample of students sitting the exams in the respective years, rather than the subsample who remained on track when we observe their outcomes after three years. Through this standardization, we ensure that we have comparable measures across the COVID and non-COVID cohorts of students’ prior performance relative to their full baseline cohort, including those who were retained and are thus not part of our outcome period samples. We perform this standardization in order to net out compositional differences between the two cohorts due to differential grade retention when we control for prior performance in our models. By transforming prior achievement into deciles we are also able to examine non-linearities in the role played by prior achievement.

In all of our Models, we control for students’ gender and socio-economic status. As an indicator of socio-economic status, we use the composite ESCS index which was developed by INVALSI. It reflects the educational attainment and occupational status of respondents’ parents, as well as the resources that are available in the respondents' household. We consider differences between the COVID and the non-COVID cohorts across four quartiles of the ESCS index to explore and identify non-linearities, and present results in the main text. We also report results based on the continuous indicator of socio-economic status in the Supplementary Online Annex. We define quartiles using the ESCS distribution in the baseline year for the cohorts of secondary school students (i.e. in grade 5) and in the outcome year for the cohorts of primary students (i.e. in grade 5). Ideally, we would have used ESCS observed at baseline for both cohorts but the ESCS indicator is not available for grade 2 students. The choice of considering ESCS at baseline is motivated by the fact that for some students the pandemic might have had a transitory effect on some dimensions of the ESCS index (such as, for example, parental occupation or availability of a quiet room to study) rendering comparisons between the COVID and non-COVID cohorts problematic. In the Online Supplementary Material we present results based on the indicator of parental educational attainment reported by students in the baseline year in all cohorts, including the primary school ones. Although parental educational attainment is a cruder measure of socio-economic status, it is available in all cohorts, is less susceptible to pandemic-related upheavals and is available for a larger number of students.

School closures affected students in different geographical areas to a different degree, since in the 2020–21 academic year, closures reflected the local evolution of the pandemic and local decision making. Furthermore, achievement levels as well as socio-economic conditions differ markedly across different regions in Italy. We include province fixed effects in all our Models to account for differences in the evolution of the pandemic, pervasiveness of school closures, and other underlying differences across Italian provinces.

Table 2 reports differences between COVID and non-COVID cohorts in terms of basic covariates used in the analyses, as well as other variables that allow to compare sample composition. Table 2 shows that despite the change in policy on grade retention, differences between the cohort in terms of prior performance are small. We only find some slight differences in terms of SES: the primary school COVID cohort was positively selected in terms of the ESCS indicator, while the secondary school COVID cohort was slightly negatively selected compared to the same grade non-COVID cohorts. The difference for primary school cohorts was considerable (almost one tenth of a standard deviation). However, observing the components of the ESCS indicator, we see that this difference was mostly driven by students in the COVID cohort being more likely to have a computer to work. We interpret this as a transitory response to home learning and not a substantive socio-economic difference between cohorts. These results highlight the importance of using indicators of SES at baseline when considering COVID achievement gaps, whenever possible.

Table 2.

Describing the COVID and non-COVID Cohorts.

	Primary school			Secondary school
	COVID cohort	Non-COVID Cohort	Diff.	COVID cohort	Non-COVID Cohort	Diff.
Female	0.49	0.49	0.00	0.50	0.50	0.00
Retained at least once	0.01	0.01	-0.00	0.02	0.02	-0.00
ESCS indicator
Average score	0.14	0.05	-0.09	0.05	0.09	0.04
1st quartile	0.25	0.25	0.00	0.23	0.24	0.00
2nd quartile	0.25	0.27	0.03	0.27	0.26	-0.01
3rd quartile	0.25	0.22	-0.02	0.24	0.24	0.01
4th quartile	0.25	0.25	-0.00	0.26	0.26	0.00
Prior math scores
Average score	0.03	0.02	-0.00	0.04	0.04	0.00
1st decile	0.12	0.10	-0.02	0.10	0.11	0.02
2nd decile	0.10	0.09	-0.01	0.11	0.09	-0.02
3rd decile	0.11	0.11	-0.00	0.10	0.12	0.02
4th decile	0.11	0.12	0.00	0.12	0.09	-0.03
5th decile	0.06	0.12	0.06	0.08	0.09	0.01
6th decile	0.12	0.06	-0.06	0.08	0.09	0.01
7th decile	0.11	0.11	-0.00	0.12	0.13	0.01
8th decile	0.10	0.11	0.01	0.11	0.08	-0.03
9th decile	0.09	0.10	0.01	0.09	0.10	0.01
10th decile	0.08	0.09	0.01	0.08	0.10	0.02
Prior reading scores
Average score	0.03	0.02	-0.00	0.05	0.06	0.01
1st decile	0.11	0.13	0.02	0.09	0.09	-0.00
2nd decile	0.15	0.11	-0.05	0.11	0.11	-0.00
3rd decile	0.09	0.06	-0.03	0.12	0.09	-0.02
4th decile	0.09	0.11	0.02	0.10	0.12	0.02
5th decile	0.09	0.12	0.03	0.11	0.10	-0.01
6th decile	0.09	0.12	0.03	0.12	0.11	-0.01
7th decile	0.09	0.06	-0.02	0.06	0.11	0.05
8th decile	0.16	0.12	-0.04	0.12	0.10	-0.01
9th decile	0.07	0.11	0.04	0.10	0.09	-0.01
10th decile	0.07	0.06	-0.01	0.09	0.09	-0.00
Home possessions
Place to study	0.85	0.84	-0.02	0.85	0.87	0.02
Computer for studying	0.74	0.61	-0.13	0.63	0.69	0.06
Desk to study	0.89	0.86	-0.03	0.86	0.87	0.01
Private room	0.62	0.61	-0.01	0.61	0.58	-0.03
Encyclopedias	0.44	0.50	0.07	1.52	1.51	-0.01
Internet connection	0.92	0.88	-0.04	1.14	1.20	0.06
More than 100 books	0.28	0.27	-0.00	0.29	0.26	-0.04
Parental status
Tertiary educated parent	0.33	0.30	-0.03	0.29	0.27	-0.03
Upper class occupation	0.22	0.22	-0.00	0.21	0.21	-0.00

Notes: Prior scores were measured using students' INVALSI scores in the assessment they sat three years before the outcome was measured. They were standardized for the full baseline period samples before matching them with the outcome period samples, and were then divided into deciles. ESCS is the INVALSI index of Economic, Social and Cultural Status. The COVID cohorts identify students for whom the achievement outcome in grade 5 or grade 8 was measured at the end of the 2020–2021 academic year. For non-COVID cohorts, outcomes were measured at the end of the 2018–2019 academic year.

Source: INVALSI waves of 2015–2016, 2017–2018, 2018–2019 and 2020–2021 (grades 2, 5, 8), own calculations.

3.3. Analytical approach

In the first set of analyses, we investigate overall gaps in academic achievement between the COVID and non-COVID cohorts (henceforth the “COVID achievement gap’’), and differential gaps by SES and gender. For the cohorts of primary and secondary school students, and for each subject (math and reading) we fit the following three Models:

$$\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ikg}}={\mathrm{\beta }}_0+{\mathrm{\beta }}_1\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}+{\mathrm{\beta }}_2\mathrm{ESC}{\mathrm{S}}_{\mathrm{ikg}}+{\mathrm{\beta }}_3\mathrm{FE}{\mathrm{M}}_{\mathrm{ikg}}+{\mathrm{\beta }}_4\mathrm{RE}{\mathrm{T}}_{\mathrm{ikg}}+{\mathrm{\beta }}_5{\mathrm{P}}_{\mathrm{ikg}}+{\mathrm{\beta }}_6\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\mathrm{ϵ}}_{\mathrm{ikg}}$$ (1)

$$\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ikg}}={\mathrm{\beta }}_0+{\mathrm{\beta }}_1\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}+{\mathrm{\beta }}_2\mathrm{ESC}{\mathrm{S}}_{\mathrm{ikg}}+{\mathrm{\beta }}_3\mathrm{FE}{\mathrm{M}}_{\mathrm{ikg}}+{\mathrm{\beta }}_4\mathrm{RE}{\mathrm{T}}_{\mathrm{ikg}}+{\mathrm{\beta }}_5{\mathrm{P}}_{\mathrm{ikg}}+{\mathrm{\beta }}_6\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\mathrm{\beta }}_7\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}*\mathrm{ESC}{\mathrm{S}}_{\mathrm{ikg}}+{\mathrm{ϵ}}_{\mathrm{ikg}}$$ (2)

$$\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ikg}}={\mathrm{\alpha }}_0+{\mathrm{\alpha }}_1\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}+{\mathrm{\alpha }}_2\mathrm{ESC}{\mathrm{S}}_{\mathrm{ikg}}+{\mathrm{\alpha }}_3\mathrm{FE}{\mathrm{M}}_{\mathrm{ikg}}+{\mathrm{\alpha }}_4\mathrm{RE}{\mathrm{T}}_{\mathrm{ikg}}+{\mathrm{\alpha }}_5{\mathrm{P}}_{\mathrm{ikg}}+{\mathrm{\alpha }}_6\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\mathrm{\alpha }}_7\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}*\mathrm{FE}{\mathrm{M}}_{\mathrm{ikg}}+{\mathrm{ϵ}}_{\mathrm{ikg}}$$ (3)

Where $\mathit{SCOR}{E}_{\mathit{ikg}}$ is the math or reading score of individual i from cohort k (COVID or non-COVID) and in grade g (grade 5 for primary students and grade 8 for secondary students), $\mathit{CO}{V}_k$ is a dummy indicating if student i is a member of the COVID cohort, $\mathit{ESC}{S}_{\mathit{ikg}}$ represents quartiles of the INVALSI indicator of Economic, Social and Cultural capital, $\mathit{FE}{M}_{\mathit{ikg}}$ indicates if student i is a girl,$\mathit{RE}{T}_{\mathit{ikg}}$ indicates if student i was retained at least once, $P_{\mathit{ikg}}$ represents province fixed effects, and $\mathit{SCOR}{E}_{\mathit{ik}\left(g-3\right)}$ is a term for deciles of students’ prior INVALSI scores from the assessment they sat in grade (g-3), so grade 2 for primary students and grade 5 for secondary students. We clustered standard errors at the classroom level. We also ran some specifications using school fixed effects to measure within-school differences, which we present in the Online Supplementary Material and comment on in the Results section.

In Eq. (1) we are interested in coefficient ${\beta }_1$, which measures mean differences in achievement between the COVID and non-COVID cohorts. In Eqn 2 we are interested in coefficient ${\beta }_7$, which measures the differential COVID achievement gaps across quartiles of the ESCS indicator. In Eqn 3 we are interested in coefficient ${\alpha }_7$, which measures the differential COVID achievement gaps across genders.

Although our paper is descriptive, our coefficients represent causal estimates under the assumption that, conditional on cohort differences in SES, gender, and our measure of relative prior achievement, the scores of the students in the COVID cohort would have been the same as those in the non-COVID cohort. Since, as previously mentioned, INVALSI scores are not comparable before academic years 2017–18 for secondary school students and 2018–19 for primary school students, we cannot control for a measure of absolute prior achievement and we require a further identifying assumption on the lack of structural trends in performance across cohorts. Since we cannot identify pre-COVID trends using INVALSI data, we gauged the plausibility of our assumption using Italian data from the international large-scale assessments.

In the Online Supplementary Material B we present trends in the math and reading achievement of Italian primary and secondary students, as well as trends by gender and SES in the Programme for International Student Assessment (PISA), the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS). Overall, we find relatively small changes over time: at most a 3% of a standard deviation yearly change assuming linear change in achievement across assessment periods (assessments were administered every 3, 4 and 5 years respectively). As we show below, these trends are inconsistent with our estimates, which makes our identifying assumption more plausible. We comment on the trends in detail in Online Supplementary Material B and we elaborate on them in the Discussion section along with our results.

Another identifying assumption required to interpret our results causally is that the COVID and non-COVID cohorts should be comparable in terms of characteristics that could influence students’ academic achievement. In the previous section we discussed some of the analytical decisions we took in order to increase the comparability of the two cohorts and solve issues of differential retention. Moreover, we have shown that the two cohorts are similar in terms of SES and standardized prior academic achievement, which we also control for in all our regressions. The fact that the COVID and non-COVID cohorts are only separated by two years also increases our confidence in the plausibility of this assumption.

In a second set of analyses, we investigate heterogeneous COVID achievement gaps by students’ prior achievement. Then, we further disentangle this heterogeneity by including two-way and three-way interactions between being in the COVID cohorts, previous performance, and gender or ESCS. For the cohorts of primary and secondary school students, and for each subject (math and reading) we fit the following three Models:

$$\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ikg}}={\mathrm{\beta }}_0+{\mathrm{\beta }}_1\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}+{\mathrm{\beta }}_2\mathrm{ESC}{\mathrm{S}}_{\mathrm{ikg}}+{\mathrm{\beta }}_3\mathrm{FE}{\mathrm{M}}_{\mathrm{ikg}}+{\mathrm{\beta }}_4\mathrm{RE}{\mathrm{T}}_{\mathrm{ikg}}+{\mathrm{\beta }}_5{\mathrm{P}}_{\mathrm{ikg}}+{\mathrm{\beta }}_6\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\mathrm{\beta }}_7\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}*\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\mathrm{ϵ}}_{\mathrm{ikg}}$$ (4)

$$\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ikg}}={\mathrm{\beta }}_0+{\mathrm{\beta }}_1\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}+{\mathrm{\beta }}_2\mathrm{ESC}{\mathrm{S}}_{\mathrm{ikg}}+{\mathrm{\beta }}_3\mathrm{FE}{\mathrm{M}}_{\mathrm{ikg}}+{\mathrm{\beta }}_4\mathrm{RE}{\mathrm{T}}_{\mathrm{ikg}}+{\mathrm{\beta }}_5{\mathrm{P}}_{\mathrm{ikg}}+{\mathrm{\beta }}_6\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\mathrm{\beta }}_7\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}*\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\mathrm{\beta }}_8\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}*\mathrm{ESC}{\mathrm{S}}_{\mathrm{ikg}}+{\mathrm{\beta }}_9\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}*\mathrm{ESC}{\mathrm{S}}_{\mathrm{ikg}}*\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\mathrm{ϵ}}_{\mathrm{ikg}}$$ (5)

$$\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ikg}}={\alpha }_0+{\alpha }_1\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}+{\alpha }_2\mathrm{ESC}{\mathrm{S}}_{\mathrm{ikg}}+{\alpha }_3\mathrm{FE}{\mathrm{M}}_{\mathrm{ikg}}+{\alpha }_4\mathrm{RE}{\mathrm{T}}_{\mathrm{ikg}}+{\alpha }_5{\mathrm{P}}_{\mathrm{ikg}}+{\alpha }_6\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\alpha }_7\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}*\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\alpha }_8\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}*{\mathit{FEM}}_{\mathrm{ikg}}+{\alpha }_9\mathrm{CO}{\mathrm{V}}_{\mathrm{k}}*{\mathrm{FEM}}_{\mathrm{ikg}}*\mathrm{SCOR}{\mathrm{E}}_{\mathrm{ik}\left(\mathrm{g}-3\right)}+{\mathrm{ϵ}}_{\mathrm{ikg}}$$ (6)

To aid interpretation, we present results from Models (4), (5) and (6) visually using the margins and marginsplot commands from STATA 17.

4. Results

4.1. Overall COVID achievement gaps and gaps by gender and SES

Table 3, Table 4 present estimates from Models (1) to (3) for primary and secondary students’ scores in math and reading. Results from Model (1) in Table 3 show that, net of controls, primary students in the COVID cohort had math and reading scores that were respectively 0.13 standard deviations lower and 0.06 standard deviations higher than those in the non-COVID cohort. By contrast, results from Model (1) in Table 4 show that, net of controls, secondary students in the COVID cohort had lower scores in both reading and mathematics compared to students in the non-COVID cohort. On average, the scores of secondary school students in the COVID cohort were 0.17 standard deviations lower and their reading scores were 0.08 standard deviations lower. Given an expected yearly learning gain of around 20% of a standard deviation (Avvisati and Givord, 2021), these results imply that COVID led to a reduction of around 85% of children’s expected lower secondary school yearly learning gain in mathematics and a 40% reduction of their expected lower secondary school yearly gain in reading, while for primary school students it led to a 65% decrease in their expected yearly learning gain in mathematics and a 30% increase in their expected gain in reading. These findings are broadly in line with the estimates reported by Bazoli et al. (2022) using a different sample and analytical strategy. Results from models with school fixed effects presented in Tables C1 and C2 in the Online Supplementary Material are similar to those in Table 3, Table 4. This suggests that most of the effects are driven by differences within schools and not across them.

Table 3.

Predicting primary students' standardized INVALSI scores by cohort, gender and SES.

	Math			Reading
	M1	M2	M3	M1	M2	M3
COVID cohort	-0.130***	-0.107***	-0.115***	0.063***	0.110***	0.095***
	(0.005)	(0.007)	(0.005)	(0.004)	(0.006)	(0.004)
Female student	-0.102***	-0.101***	-0.087***	0.165***	0.165***	0.195***
	(0.002)	(0.002)	(0.003)	(0.002)	(0.002)	(0.003)
ESCS quartile (ref: Bottom)
2nd	0.174***	0.193***	0.174***	0.186***	0.222***	0.186***
	(0.004)	(0.005)	(0.004)	(0.003)	(0.004)	(0.003)
3rd	0.265***	0.276***	0.265***	0.280***	0.306***	0.281***
	(0.004)	(0.005)	(0.004)	(0.004)	(0.005)	(0.004)
Top	0.358***	0.369***	0.358***	0.383***	0.410***	0.383***
	(0.004)	(0.005)	(0.004)	(0.004)	(0.005)	(0.004)
COVID cohort * ESCS (ref: bottom)
COVID cohort * 2nd		-0.041***			-0.077***
		(0.007)			(0.007)
COVID cohort * 3rd		-0.023**			-0.054***
		(0.008)			(0.007)
COVID cohort * Top		-0.024**			-0.057***
		(0.009)			(0.007)
COVID cohort * Female			-0.031***			-0.065***
			(0.004)			(0.004)
Observations	829143	829143	829143	823217	823217	823217
R2	0.276	0.276	0.276	0.260	0.260	0.260
Prior scores control	Yes	Yes	Yes	Yes	Yes	Yes

Notes: The COVID cohort identifies the primary school cohort of students who sat the grade 5 assessment at the end of the 2020–2021 academic year. They are compared to the cohort who sat the same assessment in 2018–2019. ESCS is the INVALSI index of Economic, Social and Cultural Status. Prior scores were measured using students' INVALSI scores in the assessment they sat three years before the outcome was measured. They were standardized for the full baseline period samples before matching them with the outcome period samples and were then divided into deciles to account for non-linearities. All Models include province fixed effects and controls for whether students were retained at least once in their academic career. Standard errors are clustered at the classroom level.

Source: INVALSI waves of 2015–2016, 2017–2018, 2018–2019 and 2020–2021 (grades 2, 5, 8), own calculations.

Table 4.

Predicting secondary students' standardized INVALSI scores by cohort, gender and SES.

	Math			Reading
	M1	M2	M3	M1	M2	M3
COVID cohort	-0.170***	-0.170***	-0.187***	-0.081***	-0.077***	-0.065***
	(0.003)	(0.005)	(0.003)	(0.002)	(0.004)	(0.003)
Female student	-0.019***	-0.019***	-0.036***	0.130***	0.130***	0.146***
	(0.002)	(0.002)	(0.003)	(0.002)	(0.002)	(0.002)
ESCS quartile (ref: Bottom)
2nd	0.181***	0.182***	0.181***	0.168***	0.171***	0.168***
	(0.003)	(0.004)	(0.003)	(0.002)	(0.004)	(0.002)
3rd	0.286***	0.285***	0.286***	0.261***	0.262***	0.261***
	(0.003)	(0.004)	(0.003)	(0.003)	(0.004)	(0.003)
Top	0.443***	0.445***	0.443***	0.409***	0.413***	0.409***
	(0.003)	(0.004)	(0.003)	(0.003)	(0.004)	(0.003)
COVID cohort * ESCS (ref: bottom)
COVID cohort * 2nd		-0.001			-0.006
		(0.005)			(0.005)
COVID cohort * 3rd		0.004			-0.002
		(0.005)			(0.005)
COVID cohort * Top		-0.004			-0.006
		(0.006)			(0.005)
COVID cohort * Female			0.033***			-0.031***
			(0.004)			(0.003)
Observations	881179	881179	881179	876398	876398	876398
R2	0.419	0.419	0.419	0.447	0.447	0.447
Prior scores control	Yes	Yes	Yes	Yes	Yes	Yes

Notes: The COVID cohort identifies the secondary school cohort of students who sat the grade 8 assessment at the end of the 2020–2021 academic year. They are compared to the cohort who sat the same assessment in 2018–2019. ESCS is the INVALSI index of Economic, Social and Cultural Status. Prior scores were measured using students' INVALSI scores in the assessment they sat three years before the outcome was measured. They were standardized for the full baseline period samples before matching them with the outcome period samples and were then divided into deciles to account for non-linearities. All Models include province fixed effects and controls for whether students were retained at least once in their academic career. Standard errors are clustered at the classroom level.

Source: INVALSI waves of 2015–2016, 2017–2018, 2018–2019 and 2020–2021 (grades 2, 5, 8), own calculations.

In terms of COVID achievement gaps across students' SES, we also find different results depending on whether students attended primary or secondary school and whether the focus was mathematics or reading. Estimates from Model (2) in Table 3 indicate that primary students in the bottom quartile of ESCS were less disadvantaged in the COVID cohort compared to the non-COVID cohort, since they experienced the largest gain in reading scores and the smallest loss in mathematics scores, conditional on their prior achievement. At the same time, results show that students in the second quartile of ESCS (the next most disadvantaged) were more disadvantaged in the COVID cohort, since they experienced the smallest gain in reading scores and the largest loss in mathematics scores, conditional on their prior achievement. The net change in overall SES inequalities between the two cohorts is ambiguous. Results from fitting Model (2) with the continuous version of the ESCS indicator presented in Table D1 in the Online Supplementary Material suggest that primary school inequalities in reading were the same in the COVID and non-COVID cohort, while they increased by a very small amount in mathematics (statistically significant only at the 10% level).

Estimates from Model (3) in Table 4 show that COVID achievement gaps in math and reading for secondary students were statistically the same across quartiles of ESCS, conditional on students’ prior achievement. This indicates that SES disparities remained stable between the two cohorts. Results obtained fitting Model (2) with the continuous version of the ESCS indicator presented in Table D2 in the Online Supplementary Material reveal small decreases in inequality in both mathematics and reading in the COVID cohort. Summarizing our results, we can conclude that SES disparities remained somewhat stable between the two cohorts or were slightly reduced in the COVID cohort. These results are broadly aligned with findings by Bazoli et al. (2022) who do not find statistically different differences in COVID-achievement gaps between students with an advantaged and a disadvantaged socio-economic background.

Estimates from Model (3) in Table 3, Table 4 indicate gender differences in COVID achievement gaps. Estimates from Table 3 show that in general, at the primary level, girls lost ground to boys in the COVID cohort. Overall, results presented in Model (3) suggest that at the primary level, boys experienced a 47% increase in the expected yearly learning gain in reading, while girls experienced a 15% increase in reading. By contrast, the negative COVID achievement gap in math scores was larger for girls. Since girls had lower achievement than boys, this led to an increase in the gender gap in mathematics.

Estimates from Model (3) in Table 4, show that gender gaps in math and reading decreased among secondary school students in the COVID cohort. This is because the negative COVID achievement gap was larger for boys in math scores and larger for girls in reading scores. At the secondary level, boys experienced a 90% reduction in their expected yearly learning progress in mathematics while among girls the reduction corresponded to 77% of the yearly learning progress. In reading boys experienced a 33% reduction of their expected learning gain, while among girls this reduction corresponded to 49% of their expected learning gain.

4.2. COVID achievement gaps by previous performance

Fig. 1 presents estimates on COVID achievement gaps by prior achievement obtained from Model (4). Results show that for all school levels and domains, conditional on SES and gender, previously high achievers had higher scores when they were in the COVID cohort than when they were part of the non-COVID cohort. For example, primary students who were in the highest decile of prior achievement scored about one fifth of a standard deviation higher in reading in the COVID cohort compared to the non-COVID cohort. By contrast, students who were in the middle part of the prior achievement distribution were generally penalized from being in the COVID cohort, conditional on their SES and gender. For example, secondary school students who had a prior reading achievement in the fifth decile scored about 15% of a standard deviation less in the COVID cohort compared to the non-COVID cohort.

Fig. 1.

The COVID achievement gap (COVID - non-COVID cohort), by prior achievement. Notes: Results from Models comparing the COVID cohorts and the non-COVID cohorts controlling for students’ gender, whether they were retained at least once, their decile of scores in the previous INVALSI test, their quartile of ESCS indicator, province fixed effects and the interaction between an indicator for being in the COVID cohort and standardized scores in the previous INVALSI test. The COVID cohorts identify the primary and secondary school cohorts of students who sat, respectively, the grade 5 and grade 8 assessment in 2020–2021. They are compared to the cohorts who sat the same assessments in 2018–2019. Prior achievment was measured using students' INVALSI scores in the assessment they sat three years before the outcome was measured. They were standardized for the full baseline period samples before matching them with the outcome period samples, and were then divided into deciles to account for non-linearities. Results are presented with 95% confidence intervals obtained by clustering students within classrooms they attended in the year where outcomes are measured.

Source: INVALSI waves of 2015–2016, 2017–2018, 2018–2019 and 2020–2021 (grades 2, 5, 8), own calculations.

Students who were previously in the lower part of the prior achievement distribution had different results depending on the academic domain and the level of schooling being considered. Previously low-achieving primary school students in the COVID cohort, scored marginally lower in math and significantly higher in reading compared to those in the non-COVID cohort. By contrast, secondary school students who were previously low-achieving had math scores that were about one fourth of a standard deviation lower in the COVID cohort compared to the non-COVID cohort, similarly to previously middle-achievers. This amounts to more than one year of secondary school expected learning gains (Avvisati & Givord, 2021). In reading, previously low achievers also scored lower in the COVID cohort than in the non-COVID cohort, but their penalty was smaller than the one experienced by middle-achievers. Results from models fit including school fixed effects presented in Figure C1 in the Online Supplementary Material are similar to those in Fig. 1, suggesting that effects are mostly driven by differences within schools.

Fig. 2 and Fig. 3 present estimated COVID achievement gaps by SES and prior achievement for primary and secondary school students respectively. Results suggest that, in all domains and levels of schooling, previously high achievers who experienced the largest score point gain from being in the COVID cohorts were those with a lower SES background. In particular, among students in the top 3–4 deciles of prior achievement, students in the lowest quartile of the ESCS indicator experienced the largest increase in scores from being in the COVID cohort. The differences were especially stark for secondary school students (Figure 3). For example, among secondary students in the top decile of prior math achievement, those from the lowest SES quartile had a math score that was about 0.5 standard deviations higher in the COVID, while those in the top SES quartile had a performance that was only 0.1 standard deviations higher in the COVID cohort. In the rest of the performance distribution, SES differences were marginal, with the exception of reading scores in secondary education, where we identify evidence of compensation. At the secondary school level, among previously low achievers in reading (those in the first three deciles of prior performance), high-SES students in the COVID cohort had similar scores as those in the non-COVID cohort, while those with disadvantaged background had lower scores in the COVID cohort compared to those in the non-COVID cohort. Our estimates are robust to the use of alternative indicators of socio-economic status, namely parental educational attainment (results are reported in Online Supplementary Material E).

Fig. 2.

The primary school COVID achievement gap (COVID - non-COVID cohort), by SES and prior achievement. Notes: Results from Models comparing the COVID cohorts and the non-COVID cohorts controlling for students’ gender, whether they were retained at least once, their decile of scores in the previous INVALSI test, their quartile of ESCS indicator, province fixed effects, as well as interactions between an indicator for being in the COVID cohort, standardized scores in the previous INVALSI test and the ESCS indicator. The COVID cohorts identify the primary and secondary school cohorts of students who sat, respectively, the grade 5 and grade 8 assessment in 2020–2021. They are compared to the cohorts who sat the same assessments in 2018–2019. Prior achievement was measured using students' INVALSI scores in the assessment they sat three years before the outcome was measured. They were standardized for the full baseline period samples before matching them with the outcome period samples, and were then divided into deciles to account for non-linearities. Results are presented with 95% confidence intervals obtained by clustering students within classrooms they attended in the year where outcomes are measured.

Source: INVALSI waves of 2015–2016, 2017–2018, 2018–2019 and 2020–2021 (grades 2, 5, 8), own calculations.

Fig. 3.

The secondary school COVID achievement gap (COVID - non-COVID cohort), by SES and prior achievement. Notes: Results from Models comparing the COVID cohorts and the non-COVID cohorts controlling for students’ gender, whether they were retained at least once, their decile of scores in the previous INVALSI test, their quartile of ESCS indicator, as well as interactions between an indicator for being in the COVID cohort, standardized scores in the previous INVALSI test and the ESCS indicator. The COVID cohorts identify the primary and secondary school cohorts of students who sat, respectively, the grade 5 and grade 8 assessment in 2020–2021. They are compared to the cohorts who sat the same assessments in 2018–2019. Prior achievement was measured using students' INVALSI scores in the assessment they sat three years before the outcome was measured. They were standardized for the full baseline period samples before matching them with the outcome period samples, and were then divided into deciles to account for non-linearities. Results are presented with 95% confidence intervals obtained by clustering students within classrooms they attended in the year where outcomes are measured.

Source: INVALSI waves of 2015–2016, 2017–2018, 2018–2019 and 2020–2021 (grades 2, 5, 8), own calculations.

Fig. 4 presents the estimated COVID achievement gaps by gender and prior achievement. Results suggest that the male gain in primary school reading scores that is associated with being in the COVID cohort was greatest for previously low and middle achievers. The opposite was true for the male gain in secondary school reading scores associated with being in the COVID cohort, which was greater for the highest achievers. The advantage of secondary school female students in mathematics in the COVID cohort was recorded for all students except previously lowest achievers.

Fig. 4.

The COVID achievement gap (COVID - non-COVID cohort), by gender and prior achievement. Notes: Results from Models comparing the COVID cohorts and the non-COVID cohorts controlling for students’ gender, whether they were retained at least once, their decile of scores in the previous INVALSI test, their quartile of ESCS indicator, province fixed effects, as well as interactions between an indicator for being in the COVID cohort, standardized scores in the previous INVALSI test and students' gender. The COVID cohorts identify the primary and secondary school cohorts of students who sat, respectively, the grade 5 and grade 8 assessment in 2020–2021. They are compared to the cohorts who sat the same assessments in 2018–2019. Results are presented with 95% confidence intervals obtained by clustering students within classrooms they attended in the year where outcomes are measured.

Source: INVALSI waves of 2015–2016, 2017–2018, 2018–2019 and 2020–2021 (grades 2, 5, 8), own calculations.

5. Discussion

The present study provides a comprehensive assessment of the medium-term impacts of pandemic-related disruptions on the mathematics and reading achievement of primary and lower secondary school students. Estimated results should be interpreted to reflect the change in achievement following generalised and localised school closures, as well as the changes in the life of children and their families that resulted from the range of measures that were adopted to limit the spread of the Sars-Cov-2 virus.

Identifying the effects of the COVID-19 pandemic on learning is important for three main reasons. First, it allows us to identify any learning gaps that might have arisen among affected cohorts, as well as which groups suffered the most. As such, estimates can inform policymakers and educational practitioners in the design and allocation of remedial interventions with the aim of preventing long-term negative effects for affected students. This is crucial in Italy, since school dropout and NEET rates are among the highest in Europe (OECD, 2021b).

Second, identifying the likely impact of major educational disruptions can inform how education systems could prepare themselves to absorb the effects of future shocks. For example, climate change is increasing the frequency of extreme weather events and as a result educational disruptions are increasingly likely to occur in the future (Horvath & Borgonovi, 2022). Information on the learning deficits resulting from the pandemic on learning could also be used by decision makers to evaluate the impact of school closures and thus the opportunity of adopting such measures in the future. Evidence from natural disasters was used by policymakers to make decisions on whether to keep schools closed or not after the first phase of COVID-19 (Harmey & Moss, 2021).

Third, estimates on the learning losses experienced by different groups of students indirectly reveal the effectiveness with which schools promoted their learning before the pandemic since, in the absence of major and large-scale remedial policies, one would expect school closures to be associated with poorer learning outcomes. In a similar vein, research investigating how socio-economic disparities in achievement evolve as a result of school closures during long summer breaks has been used to estimate the value of schooling for children with different home backgrounds (Entwisle and Alexander, 1992, Quinn et al., 2016).

In our analyses, we identify a reduction in mathematics and reading achievement for secondary school students in the COVID cohort compared to similar students in the non-COVID cohort. In line with other studies, we find that the COVID achievement gap was most severe in mathematics. Contrary to evidence from most other countries in which there were no differences in effects across age groups (Betthäuser et al., 2022), for primary students, we identified a smaller reduction in math achievement and an increase in reading achievement associated with being in the COVID cohort. Our estimates are comparable to those found in another study estimating the effects of the pandemic on achievement in Italy which was conducted using a different estimation method and focused exclusively on the samples of students who sat INVALSI tests under external monitoring rather than the full population data (Bazoli et al., 2022).

The differences between primary and lower secondary school children could be explained by the fact that between Spring 2020 and the end of academic year 2020–21 primary school children benefited from fewer days of closed schools than lower secondary school children. The fact that the primary school COVID achievement gap in reading was actually positive could be explained by the significant amount of time young children spent with their parents on a one-to-one basis and the fact that they could practice reading skills and text comprehension skills effectively within the family. By contrast, since many families might have lacked skills to help their children progress in mathematics, a negative effect was observed in mathematics. The role of remedial behaviour on the part of teachers and parents of young children appears confirmed by the finding that among primary school students, contrary to expectations, the effect of the pandemic was positive among the lowest achievers in reading and less negative among the lowest achievers in mathematics. Low-achieving students are in fact those who were most likely to be the target of remedial support at school and close monitoring within the household.

Among older children we observe a similar pattern in reading, with previously lower achievers being less penalized than average performing students in the COVID cohort, while in mathematics we observe a similar penalty for both low and average achievers. This can be explained by the fact that remedying a learning deficit in mathematics may be more dependent on the intervention of teachers than household members compared to remedying a learning deficit in reading. Organising remedial interventions might have been more difficult in lower secondary schools since they faced longer closures than primary schools. Furthermore, the secondary school curriculum allows for fewer informal reallocations of time to fill gaps in specific academic domains.

In Italy, primary school students typically have a reference teacher who is responsible to deliver the majority of instruction. Such teacher could allocate more time to the material in which children had accumulated gaps during school closures. By contrast, in secondary school different teachers are responsible for different subjects and each teacher has a fixed time allocation. It was therefore more difficult for teachers in secondary school to deviate from pre-arranged time allocation across different subjects, irrespective of students’ progress during school closures.

Interestingly we find that previously high achievers in the COVID cohorts had a higher score in the INVALSI math and reading test than previously high achievers in the non-COVID cohorts. Irrespective of subject domain and grade attended, we are faced with the paradox that high achievers benefited in terms of learning outcomes from the pandemic. This result could reflect the fact that talented students during the pandemic had greater freedom to explore their interests and learn at their (higher) pace and that technological innovations supported such learning and exploration. These students might have also benefited from the more frequent one-to-one interactions within the household that might have arisen because of teleworking arrangements and limitations to mobility.

Our results indicate that the group of students whose scores were lowest in the COVID cohort relative to the non-COVID cohort were previously middle achievers. Two factors can explain this finding. First, this is the group of students who, in normal times, benefits the most from schooling. In Italy, no ability grouping exists in primary and lower secondary schools (OECD, 2020) and, as a result, classes are often heterogeneous with respect to students’ ability (INVALSI, 2020). Furthermore, teachers in Italy have comparatively low levels of teacher pedagogical preparation and, in particular, do not routinely use approaches in their teaching that allow students with different levels of achievement to learn (OECD, 2018, OECD, 2019). As a result, teachers in this context tend to tailor instruction to the needs of average performers, reducing the opportunities of low achievers to overcome their gaps and of high achievers to flourish. In this light, middle achievers have more to lose when schooling was disrupted, because they were the group who was most supported by schooling in normal times. Second, middle achievers are the students who were least likely to benefit from remedial support since their achievement did not fall below minimum standards. At the same time, they did not benefit from the wealth of opportunities that arose because of the new online learning communities that were created during school closures, and by the variety of learning material and resources that afforded high achievers the possibility to engage in high quality online learning.

Contrary to evidence from most other countries (Betthäuser et al., 2022), our results suggest that socio-economic disparities did not increase in the COVID cohort as expected. In fact, we find that among primary school students, SES disparities were somewhat lower, since the most disadvantaged students had the largest positive COVID achievement gap in reading and the smallest negative COVID achievement gap in mathematics. For secondary school students, we find that the penalty associated with being in the COVID cohort was similar across levels of SES. These results are also broadly in line with the work of (Bazoli et al., 2022).

We find that differences in SES-disparities in the COVID and non-COVID cohort varied across levels of previous achievement. Generally, for both primary and lower secondary students, SES disparities were reduced among high-achieving students in the COVID cohort. This could reflect the unobserved skills and motivation that high-achieving but socio-economically disadvantaged children possess, which could have shielded them from the negative consequences of pandemic-related disruptions. It could also reflect the fact that distance learning might have reduced some of the hidden stereotypes and biases that may guide teachers’ behaviours during regular face to face instruction (Batruch et al., 2017). Finally, distance learning might have also reduced negative peer influences that socio-economically disadvantaged but high-achieving students might experience during regular schooling (including being the victims of bullying or simply low levels of educational expectations). These results are not aligned with the simulated effects of COVID on skills acquisition (Agostinelli et al., 2022) and therefore warrant further analyses and investigation.

We found evidence that, in the COVID cohort, gender gaps in both reading and math were lower for secondary school students, as well as in reading for primary school students. The largest reductions in gender gaps were for low-achieving primary school students in reading, among which boys recorded the largest positive COVID achievement gap. These results suggest that the pandemic might have disrupted the influence of any stereotypes teachers and peers might have held on the math ability of girls and the reading ability of boys, thus leveling the playing field for both genders, and that young boys benefited from one-to-one interactions within the home.

In the current study we report average results for the full Italian territory. However, as noted, Italy is a country with large regional variation in levels of economic development, quality of schooling and of the health care system, which led to major differences in the toll the pandemic had on the health, economic, and social life of individuals (Possenti et al., 2021). For example, whereas the pandemic first hit Northern regions which led them to close schools earlier, some Southern regions implemented stricter stay-at-home orders because of fears that local health care capacity would be overwhelmed. Therefore, it is possible that the average results that we report could mask differences across geographical areas. In Supplementary Online Materials Figures F1 and F2 we illustrate the variation across Italian provinces in the difference between the COVID and the non-COVID cohorts in the reading and math achievement of primary and secondary students. Results suggest a large degree of variation not only between Northern and Southern regions but also within regions, especially among primary school children. A more in-depth examination of geographical differences and heterogeneities by ESCS, gender and level of prior achievement is beyond the scope of this work but should be pursued in future research. Such work could also compare geographical differences in achievement differentials with geographical differences in the length of school closures or the severity of COVID-related disruptions.

The current study is descriptive: as such it can be used to identify the difference in achievement between the cohorts who sat INVALSI tests just before the pandemic hit and those who sat the tests in the Spring of 2021. Estimated differences would reflect the causal effect of the pandemic on academic achievement assuming the absence of underlying trends in achievement and the lack of compositional differences in the two cohorts not accounted for by observable characteristics. Although the data at hand do not allow to fully verify these assumptions, we provided suggestive evidence that they plausibly hold.

In the Online Supplementary Material B, we reported trends in the math and reading achievement of Italian students using PISA, PIRLS and TIMSS data. These trends are generally in contrast with our estimates of COVID achievement gaps. For example, in primary and secondary school mathematics, trends in achievement estimated using data from TIMSS and PISA were generally stable or changing marginally (less than 2% of a SD yearly), while we measured drops in achievement of 13% and 17% of a SD between the COVID and non-COVID cohorts. In terms of primary school reading, trends based on PIRLS data signal a 1.5% of a SD yearly increase in the last period, while we recorded a COVID achievement gap of 10% of a SD. In secondary school reading, data from PISA reveal a yearly decline of 3% of a SD whereas we identify an overall decline of almost 8% of a SD. However, whereas in PISA the decline in reading at the secondary level appears most pronounced among boys (4% of a SD against 1% of a SD among girls), our results indicate that the decline corresponded to 6% of a SD among boys but over 10% of a SD among girls. Overall, we are confident that our results are not driven by structural trends in achievement.

In terms of compositional differences between the COVID and non-COVID cohorts, we have shown that most observable characteristics were comparable in the two cohorts. Naturally, there could be some changes in unobservable characteristics that could be affecting our results, but this is unlikely given the short time distance between the two cohorts. By focusing on students whose performance could be traced back in time, we were also able to tackle the issue of differential retention across the two cohorts caused by the policy change during the pandemic. However, this decision implies we excluded some of the most disadvantaged students from our analyses. In this light, our estimates of COVID achievement gaps could represent an upper bound, especially for secondary school students, among whom retention is most diffused. We are confident that our results on SES-differences are not driven by this choice, since differences are mostly for high achievers, who are the least likely to be retained.

A final potential difference between the COVID and non-COVID cohorts that could invalidate our estimates is if they and their teachers adopted a different behavior during the INVALSI test session, i.e. if the incidence of cheating differed before and after COVID. Although the INVALSI test has no consequences for individual students, it has been documented that some students cheat during the test, aided by their teachers who do not monitor the test session effectively (Lucifora & Tonello, 2015). Computer delivery is very effective in eliminating cheating and the INVALSI test was administered on a computer to grade 8 students both before and after COVID. However, grade 5 students completed a paper-based version of the test both before and after COVID. It is possible that some teachers might have allowed more cheating in the COVID cohort, recognizing the difficulties encountered by their students during the pandemic. A good indicator of the pervasiveness of cheating is to compare population level scores with the scores of the subsample of students who sat the INVALSI test under the supervision of an external monitor rather than their teacher (Esteban, 2013). We ran our main specifications on the subsample of students who sat the test under external monitoring and estimates that we obtained using population level data – to which INVALSI routinely applies a correction factor to account for cheating. Results are presented in Tables G1 and G2 in the Online Supplementary Material and are aligned with our results from the population data. This suggests that either cheating behaviour did not differ across the two cohorts or that the correction factor implemented by INVALSI effectively accounts for differential behaviour before and after COVID in the sample of students who sat the test with their teachers rather than external monitoring.

An important limitation of our study is that we were able to estimate the difference in achievement between the two cohorts but, assuming this difference was causally determined by the pandemic, we cannot determine why. For example, we do not have information on how children spent their time before the pandemic hit, when schools were closed and when mobility was restricted by law. Nor do we have information on the level of engagement of their parents, what activities teachers and school principals put in place to guarantee the provision of educational services during the pandemic and the peer environment students experienced before and during the pandemic. The pandemic not only disrupted how teaching and learning were organized but first and foremost the social environments in which students operated. As such, it might have modified the social norms and expectations students held for themselves but also the social norms they were confronted with from peers, teachers and parents. Further research is needed to better identify the mechanisms responsible for observed effects and provide input to education policy-makers, teachers and families on how best to address the consequences on learning arising from educational disruptions.

Appendix A. Supplementary material

Supplementary material

.