Education data needs and challenges for building back from COVID-19

Abstract

The COVID-19 pandemic has led governments worldwide to impose extensive restrictions on citizens, some of which may have long-term impact after their removal. Education is arguably the policy domain where closure policies are anticipated to lead to greatest lasting loss, in this case learning loss. Currently, limited data exists from which researchers and practitioners can draw insightful conclusions about how to remedy the problem. In this paper, we outline the global pattern in pandemic school-closure periods and illustrate data needs through the examples drawn from Brazil and India, two large countries which experienced prolonged periods of school closures during the pandemic. We conclude with a series of recommendations for building an improved data environment at government, school and household levels, to serve the building back agenda in education, and to provide better opportunities for evidence-based policymaking thereafter.

1. Introduction

Much has been written about the unprecedented scope and length of government mandated closure policies around the world in response to the COVID-19 pandemic. In 2020, before vaccines were available, the most important research questions in service of evidence-based policymaking pertained to the effectiveness of closure policies to encourage the observance of protective behaviours (Petherick et al., 2021; Petherick et al., 2020a, Petherick et al., 2020b; Bavel et al., 2020) and, ultimately, to reduce infection rates (Islam et al., 2020). Closure policies ranging from the cessation of public transport services to stay-at-home orders had various impacts on populations that observed them; for example, many people experienced mental health difficulties (Aknin et al., 2022, Robinson et al., 2022) and/or loss of income (von Wachter, 2021), often to the point of facing food insecurity (Dasgupta and Robinson, 2022). In 2021, as health authorities rolled out vaccination programmes, and while large swathes of populations awaited their turn, closures remained the primary policy tools available in most countries. At time of writing, in 2022, and likely for several years ahead, the most urgent research questions have shifted. Given the profound social and economic costs of some COVID-19 response policies, governments now need to know how to build back. There is a pressing need for data to describe the scale and nature of what is to be remedied, as well as data about the relative effectiveness of the policy options for recovery.

Among all public policy domains, education may be where the most profound impacts and building back challenges lie (Reuge et al., 2021). In almost all countries, governments closed schools quickly following the WHO’s announcement that a novel coronavirus had spread to pandemic proportions. They did this as part of a raft of closures across various policy domains (Hale et al., 2021, Cheng et al., 2020). Then, as the months passed and concern about the long-term effects of school closures rose, intense discussion ensued about whether children suffered badly from COVID-19 relative to adults (Zimmermann and Curtis, 2021), or whether some might, in fact, be at particular risk from certain medical conditions brought on by infection (Consiglio et al., 2020). For months it remained unclear how much school closures contributed to the spread of COVID-19 in the general population (Auger et al., 2020; Flasche and Edmunds, 2021). Consequently, policymakers debated heatedly whether school closure policies should be loosened before some other forms of restrictive policies, and, if so, for which age groups (Hawk et al., 2021). Throughout this period, limitations in the availability, reliability, and external validity of data-based insights contributed to intensity of discussion.

It is estimated that more than 90% of schools across the world faced government-required closures during 2020 (Lennox et al., 2021). Students were kept at home while parents and caretakers were also taking part in distance work schemes or were perhaps unemployed. To attempt to reduce the negative impacts of school closures, various strategies emerged from different parts of education systems, from innovative individual teachers to ministries. These strategies largely attempted to deliver teaching using online modes of communication, TV and radio programmes, and printed materials to support at-home lessons (Lennox et al., 2021). Again, much of the time, teachers, schools, and policymakers were relying on intuition, rather than evidence, to guide their decisions about what to do. At this point, faced with the challenge of variance in learning loss, there is a pressing need to reduce the extent to which education ministries, schools, teachers, and parents are “flying blind”.

In this paper we discuss the forms of data that would be helpful to inform the building back from COVID-19 agenda in education and thereafter provide more robust data systems in this area. As we review briefly in the next section, the literature anticipates learning losses from the extensive periods of school closures to be profound, and, without appropriate interventions, these losses are likely to worsen after students return to in-person classes (Kaffenberger, 2021). While large policy-tracking datasets have enumerated the length of school closure policies for virtually every country in the world, and for many subnational jurisdictions of large countries (Hale et al., 2021, Cheng et al., 2020), such systematic data is lacking for the types of education provision furnished on students during school closures, as well as on students’ ability to access and their actual use of these resources. After outlining global patterns in the duration of school closures, we discuss education data and its use in the context of the parallel school management1 system exemplified by the Brazilian municipality of Belo Horizonte, and some initial insights into learning loss from household survey data collected in three states in India. We round off our discussion with a series of recommendations for data collection, organisation and sharing at the government, school and household levels, noting that interventions or improvements that focus on any single level without taking into account weaknesses that exist at other levels will lead to suboptimal outcomes. Since there is no widely accepted theoretical framework for how data systems in education can be optimised, the suggestions that we highlight are those that consistently appear to work across contexts.

2. The global perspective

A starting point for making sense of the worldwide impact of school closures during the COVID-19 pandemic is the huge global variation in their duration. Fig. 1 illustrates how, for some countries, schools have been shut for the bulk of a two-year period, while in others, closures were short lived. Brief periods of closure were the norm among African countries, for example, while in the Americas, schools were shut for much longer.

Fig. 1.

Number of school closure days Jan 2020 - Dec 2021. This figure depicts the total number of partial (some levels) or full (all levels) school closure days over the whole territory or in at least one subnational region of a country.

Source: Oxford COVID-19 Government Response Tracker (available at https://github.com/OxCGRT/covid-policy-tracker).

A number of attempts have already been made by scholars to understand the impact of COVID-19 school closures (Alban Conto et al., 2021). Estimation of learning loss is difficult using merely school closure periods because it relies on many untested assumptions. However, a few scholars have attempted this using the best data they had available. Engzell, Frey and Verhagen (2021), taking advantage of student tests that were administered before and after school closures in the Netherlands, estimated that on average students lost about one-fifth of a school year, and students in less advantageous backgrounds had losses up to 60% above the mean. It is important to highlight that the Netherlands had a relatively short period of school closures (8 weeks). Ardington et al. (2021) also find evidence of learning losses in the reading scores of students in South Africa, ranging from 57% to 81% of a year of learning. Finally, according to a meta-analysis that drew together data from 18 studies spread over 3 continents, conducted by König and Frey (2022), the negative impact of COVID-19 is on par with summer learning losses on per week basis —implying that Fig. 1 is a decent guide to where losses have been greatest around the world.

Given the data constraints, and how recent COVID-19 effects are, analyses of other sudden school closures, with different causes, offer further guidance as to possible learning impacts of the COVID-19 closures. For example, the 2004 tsunami that hit Aceh province in Indonesia especially hard was associated with reduced educational aspirations among children, even though school enrollment recovered (Frankenberg et al., 2013). During the West African Ebola outbreak, declared as a “major epidemic” by the WHO in 2014, schools in Guinea, Liberia and Sierra Leone were closed for 5–9 months, interrupting education for about 5 million children (Yao et al., 2021). Ebola had a range of educational impacts, with some evidence that a short-term shock turned into a more permanent one, and with one study reporting that 17% of girls never re-enrolled when schools reopened in part due to a spike in teenage pregnancies (Bandiera et al., 2019). Overall school attendance numbers recovered three or four years later (Yao et al., 2021), however ethnographic research found substantial learning loss (Kostelny et al., 2016).

Quantitative evidence of long-term learning loss due to sudden school closures can be drawn from a study of the impact of a large earthquake that hit northern Pakistan in 2005 (Andrabi et al., 2020). Even though there was no relationship between distance from the earthquake’s epicentre and the state of public infrastructure several years later (due to recovery investment), and similarly no drop-off in school enrollment closer to the area that was hit – the nearer that children aged 3–15 lived to the fault line, the worse they scored on academic tests over time. Learning loss became exacerbated after returning to the classroom, with those children who experienced the forced school closures facing estimated learning losses equivalent to 1.5 fewer years of schooling, even though schools in the earthquake-affected zone were closed for an average of just 14 weeks. One insight from this study is that the education level of mothers played a substantial role in lessening learning loss.

Two large developing countries that experienced prolonged school closures during the COVID-19 pandemic are Brazil and India. Brazil had the second longest period of school closures (Saudi Arabia had the longest by 3 days), with some types of schools required to close in at least part of the country for 660 days between January 2020 and December 2021 (almost double of the global mean of 359 days) (Hale et al., 2021). India ranks 18th over the same period, with 617 days. These case studies underscore some of the complexities of evaluating learning loss during the COVID-19 pandemic. They also offer some ideas for how to fill the data void, which we pick up in the subsequent section, outlining worldwide data needs and recommendations.

3. Subnational variation in Brazil

Differences in policy content by subnational governments in Brazil, as well as variation in the implementation of policies, suggest that the COVID-19 pandemic will widen learning gaps in various student populations in the country. An analysis of learning loss in São Paulo state, for example, found that early in the pandemic, students studying remotely learned 27.5% on average of what they were expected to, using in-person learning as a reference (Lichand et al., 2022). Because Brazil is one of the countries for which there is detailed, publicly available subnational data about the length of school closure policies, it is possible to observe that different levels of government (state and municipality) often took different approaches to closing schools during the pandemic, and these differences were even more pronounced when their supplemental policies to support learning are considered.

Brazil’s federalist system creates the necessity for all levels of government to cooperate in the provision of public education. This happens because there are two parallel public educational systems within the same territory, a state-wide system that is run by the state government, and various municipal systems that are run by each municipality within each state. In other words, within any given municipality, there are public schools that are managed by the municipal government but also public schools that are managed by the state government. The latter share policies with all other state schools within the same state, regardless of the municipality they are in. The schools that are managed by the municipality are only subject to the policies coming from the municipal governments (because the Brazilian constitution does not privilege state-level policies over municipal-level policies). Hence, in a crisis, this complex management scenario can potentially hinder student learning if states and municipalities are not aligned in the responses that they employ. In this context, the federal government should provide overall guidelines and parameters that would help state and municipal administrators to make decisions, aligning policies across all levels of government. However, during the COVID-19 pandemic, Brazil’s federal government was largely inactive, leaving lower levels of government to develop and implement response policies in many areas (Petherick et al., 2020a, Petherick et al., 2020b).

Analysing data on school closure policies from state and municipal governments in Brazil, coded by the Oxford COVID-19 Government Response Tracker (OxCGRT) project, reveals that the percentage of days when state and municipal administrations in the same state adopted different policies towards school closures ranged from 30% to 87% (between 1 January 2020–15 February 2022), with an average of 46.4% days, across all states of the country (see Fig. 2). The lack of guidance from the federal government can be seen as one of the factors leading to this variation in policy strategies. The consequence of the discrepancies in school closure policies is likely to impact on learning not just due to lower learning levels while students were at home, but because as students move up the school years, for example to primary to secondary school some will change from a municipal-managed school to a state-managed school, while some will remain in the state-manage system. Hence teachers will be faced with unusual levels of variance in student ability in the classroom. Present-day differences in educational outcomes could, looking ahead, lead to widening learning gaps if effective remedial strategies are not adopted.

Fig. 2.

: Proportion of days when state and municipal administrations had different school closure policies between 1 January 2020–15 February 2022. The difference in school closure policy is defined as state and municipal administrations having different levels of school closure restrictions according to the OxCGRT C1 (school closure policy) ordinal indicator.

The example of the state of Minas Gerais, and its capital Belo Horizonte, provides insightful nuance into the struggles of coordination between different administrations even in a fairly well off and politically aligned context. Both of these levels of governments acted fast when deciding to interrupt the 2020 academic year: in mid-March 2020 both state and municipal administrations determined that schools under their jurisdiction had to be closed until further notice.2 These decisions were taken less than a month after the first COVID-19 case was detected in Belo Horizonte and when only the fourth case had been detected in the state. However, once schools were closed, the rollout of policies for remote learning was very different.

In early April 2020,3 the state government instituted remote work for teachers and other school professionals. About ten days later, it released a set of parameters and references for remote schooling activities.4 Using the state infrastructure for TV and radio broadcasts, the state department of education created programming to provide academic content for students (this was known as “Se Liga na Educação”).5 Additionally, it offered online content with free mobile data plans for students through a state-developed app (Digital Conexão Escola), and released printed material for students who did not have access to either TV and radio or the internet. Relying on the distributional capabilities of schools located across the state and the 47 regional offices of education (which are subdivisions of the state department of education), state officials developed an online system that enabled school principals to track students’ access to the materials. Based on these data, the state central administration was then able to pinpoint students who did not have access to online or broadcast resources, making it easier to gauge the need for printed materials and further at-home support (a programme called Busca Ativa).

By July 2020, a second set of orientation documents (Planos de Estudos Tutorados) were made available for state schools. With these policies, and by altering the scheduling of recess days so as to cover the period that students spent at home without remote learning support, the state tried to reduce learning loss. Already in September 2020, the state administration started to roll out the return of in-school activities in specific geographical locations within the state that had reported low rates of infection.6

However, the extra support for remote teaching and learning in municipal schools in Belo Horizonte was carried out at a much slower pace. Part of the reason for this is related to the fact that municipal schools tend to provide early childhood education and education for grades 1–5, with the state schools taking responsibility for grades 6–9, and because generally it is harder for students in early grades to take advantage of online materials and support, given the nature of the content and the care that they require. Nonetheless, there is still overlap in the grade range for which municipality- and state-run schools cater, and constitutionally, responsibility is formally shared by both municipal and state governments for all grades 1–9.

In effect, it was only on 16 July 2020 that the municipal administration defined the conditions for remote work for teachers and other school professionals.7 Two months later, in September 2020, it established parameters for activities focusing on students in specific grades, such as those that were likely to transition to another school system by the end of the academic year.8 And only on 18 November 2020 did the municipal administration create special committees to support all students' return to in-person schooling activities. In January 2021, Belo Horizonte’s administration decided to redefine the academic calendars of 2020 and 2021, so that they ran together. This was done to create sufficient learning hours overall, given the delays to the municipal government’s rollout of remote learning.

The misalignment that emerged in the education policies in Belo Horizonte’s municipal and state-run schools highlights the potential weaknesses created by the absence of guidance from the federal government in such a system. This misalignment has created two groups of students within the same territory with very different learning experiences during the crisis. Concerningly, understanding the impact of the differences in state and municipal strategies is almost impossible as there are no data available on the effectiveness of either. To properly evaluate the policies carried out and the potential widening of learning gaps, one would ideally have data on student outcomes before, during and after the implementation of the policies, as well as specific details about how these policies reached students and education professionals.

4. Subnational variation in India

In India, as in Brazil, education policy is shared between different levels of government; both the central government and the states are able to enforce decisions affecting education policy (Tilak, 2017). During the pandemic, the National Education Policy (National Education Policy, 2020) launched by the central government took precedence over state policy, though it empowered states to take ownership of proposed reforms, including those that ensure the right to free and compulsory education for all children.

Evaluating the impact of India’s school closures overall is challenging. But reliable data does exist with which to directly assess learning loss during the pandemic for a majority of districts in 3 states. Using the Annual Status of Education Report (ASER) – a large-scale household-based assessment conducted in India since 2005 – it is possible to evaluate learning loss in basic reading and arithmetic in Chhattisgarh, Karnataka, and West Bengal.

Fig. 3 uses OxCGRT data to show the timeline of compulsory school closures in each of these 3 states, with 3 of the vertical dashed lines representing the timing of ASER household surveys. India’s federal government imposed school closures on the whole country from March until October 2020, after which the states brought in locally specific policies. Karnataka began reopening some colleges and universities in October 2020, with some classes resuming from January 2021. It eventually opened up for teaching of all grades with some social distancing restrictions in February 2021. Then, in April 2021, when infection levels rose during India’s second wave, Karnataka re-imposed school closure requirements, reopening gradually once again between August and October 2021. The other two states, by comparison, initially adopted a more conservative stance. In Chhattisgarh and West Bengal, only some grades were allowed to reopen before March 2021, and then schools were required to fully close again amid restrictions imposed to counter the second wave. Chhattisgarh, similarly to Karnataka, started reopening schools after the second wave in July 2021, and reached its lowest levels of school closure stringency in October 2021. West Bengal, however, only reopened some schools from November 2021. Despite these differences, there was relatively little disparity in the cumulative period of school closures in the 3 states: the total period of government-mandated school closures for all schools was 605 days for Chhattisgarh (with 54 days of required closures for some schools only), 541 days for all schools in Karnataka (with 122 days of closure requirements for some), and 656 days for all schools West Bengal (where there were no partial closure requirements).

Fig. 3.

Timeline of mandated and recommended school closures9 in Indian states. During the pandemic, ASER was able to conduct household surveys during brief windows when infection rates and policies permitted in those 3 states (as indicated by 3 of the vertical dashed lines).

The sheer scale and consistency with which household-level data on learning changes have been collected in these 3 states is globally unusual during the pandemic. Prior to the pandemic, ASER India was the world’s largest annual household survey, aiming to provide estimates of children’s schooling status and basic learning attainment for every state and rural district in India.10 It manages this because it assesses all children irrespective of their schooling status, and because it records data on children from the age of 3, individually assessing those aged 5–16 on arithmetic and reading competencies.11

The simplicity of ASER’s household survey tools makes it possible for volunteers who are ordinary citizens, rather than trained researchers, to go house to house, implementing surveys. This is a critical feature of the project that leads to high levels of retention across survey years and enables large-scale data collection in India’s many languages. Prior to the pandemic, ASER’s process of citizen-enumerated surveys had been shown to generate widespread engagement of critical stakeholders including parents, teachers, and community members; it had also been found to promote new mechanisms for accountability and evidence-based policymaking (Aslam et al., 2019). Having volunteers sit with each child to assess them in their own homes, with a parent present, has also been shown to lead to greater parental engagement in their child’s education. For many illiterate parents, an ASER assessment is the first time they are exposed to what their child knows (Banerji, 2021 p.183).

Data from ASER’s mid-pandemic arithmetic and reading assessments in Chhattisgarh, Karnataka, and West Bengal are depicted in Fig. 4 and Fig. 5 respectively. The data is split by parental education levels. These graphs show that the share of grade 1 students at “beginner” level i.e. those that are unable to identify letters or numbers, increased in the survey conducted during the pandemic compared to previous years across the 3 states (see Appendix Fig. A3, Fig. A4, Fig. A5, Fig. A6 for data on grade 5 and grade 8 students). This descriptive change is especially notable given that, prior to the pandemic, ASER household surveys were conducted every two years (2014, 2016, 2018), with no survey possible due to closure policies at the next time point in the sequence (2020). Hence the period of time between the 2 right-hand columns of each of the graphs extends over approximately 3 years rather than 2—and during the 2018–2021 period schools were open for the majority of the time (i.e. through to March 2020).

Fig. 4.

ASER Arithmetic Assessment scores for grade 1 students, by parental educational attainment. Note that 'Low' parental education is defined as both parents having completed Std 5 or below and 'high' parental education as both parents having completed at least Std 9. See appendix Fig. A3, Fig. A4, for grades 5 and 8.

Fig. 5.

ASER Reading Assessment scores for grade 1 students, by parental educational attainment. Note that 'Low' parental education is defined as both parents having completed Std 5 or below and 'high' parental education as both parents having completed at least Std 9. See appendix Fig. A5, Fig. A6, for grades 5 and 8.

Fig. A3.

ASER Arithmetic Assessment scores for Grade 5 students, by parental educational attainment. Note that 'Low' parental education is defined as both parents having completed Std 5 or below and 'high' parental education as both parents having completed at least Std 9.

Fig. A4.

ASER Arithmetic Assessment scores for Grade 8 students, by parental educational attainment. Note that 'Low' parental education is defined as both parents having completed Std 5 or below and 'high' parental education as both parents having completed at least Std 9.

Fig. A5.

ASER Reading Assessment scores for Grade 5 students, by parental educational attainment. Note that 'Low' parental education is defined as both parents having completed Std 5 or below and 'high' parental education as both parents having completed at least Std 9.

Fig. A6.

ASER Reading Assessment scores for Grade 8 students, by parental educational attainment. Note that 'Low' parental education is defined as both parents having completed Std 5 or below and 'high' parental education as both parents having completed at least Std 9.

Learning differences are especially apparent at the lower end of foundational skill development. Overall, the percentage of grade 1 students performing at the poorest level in arithmetic – unable to identify integers – went up from 37.19% in 2018 to 50.36% in 2021 in Chhattisgarh, from 28.11% to 42.08% in Karnataka, and from 20.44% to 29.11% in West Bengal. Similarly, the share of students unable to identify letters went up from 44.72% to 58.04% from 2018 to 2021 in Chhattisgarh; from 38.34% to 56.65% in Karnataka; and from 24.57% to 30.84% in West Bengal. This suggests that school closures in 2020–2021 consistently led to learning losses among primary-age students.

It is also worth noting that on average grade 1 students whose parents were both educated up to grade 5 had consistently worse educational outcomes than those whose parents were both educated to grade 9 or above (see Fig. 3, Fig. 4, and Appendix Fig. A3, Fig. A4, Fig. A5, Fig. A6 for equivalent figures for grade 5 and grade 8 students). In other words, the learning loss due to the pandemic appears to be larger for more disadvantaged children, though still apparent in this group of more advantaged children. In West Bengal, for instance, the proportion of grade 1 students at beginner level increased by 20% points from 38.68% to 58.27% among students whose parents have low educational attainment, compared to 3% points (7.96% in 2018 to 10.9% in 2021) for students with more educated parents (both achieving grade 9). While this data does not speak to changes in classroom variance in learning levels due to the pandemic, it suggests that this is a strong concern, and a challenge with which teachers may have to grapple. The data also suggest that students in economically disadvantaged states, such as Chhattisgarh, who were already worse off than students in states with higher average learning levels prior to the pandemic,12 have fallen even further behind relative to these peers. Thus, at both the household level (by level of parental education) as well as at the system level (in terms of effectiveness of education delivery), differences in learning equity in India may have diverged further – the education of students who were worse off appears to have suffered most in the pandemic.

5. Data system recommendations

In the paragraphs below we lay out some initial suggestions towards creating an improved data ecosystem that would assist educators and education systems in building back from the disruptions of the COVID-19 pandemic, and lay the ground for broader evidence-based policymaking in education beyond this particular challenge. A policy agenda that seeks to recoup the educational trajectories of students who struggled to make progress during the pandemic, and that seeks to avoid exacerbating learning loss beyond the initial return to in-person teaching, must be informed by evidence about where efforts need to be focused and what works. It therefore needs to place the collection, use and sharing of data at its core. Often this will require shifting organisational cultures to embrace empiricism and cooperation. New mechanisms and routines need to be established for data to flow from schools up to ministries and back down again. It will also require supporting capacity-building efforts to enable evidence collection and interpretation, especially in local governments and in schools.13

5.1. Government level

The prominence of the issue of urgent post-pandemic data needs in education could, for many governments, helpfully evolve the building, or the improvement of such systems for the long run, and contribute to a new normal of continuous data-driven improvements in education. To this end, political support is advantageous, yet political will driving educational data production is generally insufficient without additional, practical support for developing associated competencies and incentives to use that data.

Prior to the pandemic, there was plenty of room for improvement in the extent to which and in the various ways that governments measured students’ educational progress and how well education systems were functioning. For example, standardised, regular and reliable assessment data, and teacher absenteeism data, is critical for national education authorities to make decisions on teacher deployment and other resource allocations.14 Too often government data systems have only included inputs such as the number and characteristics of teachers and students, textbooks and infrastructure, leaving out learning outcomes and teacher attendance. Moreover, external provision of technical support has often left government officials without the skills to conduct data analysis, and data management is often confined to a few individuals within ministries who control access, also contributing to limited data use.

The World Bank’s Systems Approach for Better Education Results (SABER) tool on education management information systems (EMIS) measures policies and legislation, as well as standards and strategies. An EMIS is the primary mechanism by which governments systematically monitor progress towards learning goals and can foster accountability for reaching them. In some countries, such as Somalia, these information systems have only recently been developed, whilst in others, such as Brazil, there are established data systems, though even among countries with strong systems, challenges with data accessibility and sharing persist. When EMIS data is difficult to access, alternative data systems proliferate creating fragmentation and confusion in the system.

As the examples from Brazil and India made clear, many governments currently lack information of several types. For example, there are varying levels of data on the forms of teaching made available to students who were studying at home, on how able students were to come by these, and on how effective such materials have been at delivering learning (Hossain, 2021). To be instructive to the building back agenda, therefore, an EMIS could be minimally adapted to incorporate the extent of school closures in different parts of the country, alongside whatever information can be collected on the materials that were made available to students during closures and the means of distribution – as well as data on variation in local access to the common means of receiving these materials, such as radio signal coverage, TV ownership and internet connectivity. As König and Frey (2022)’s meta-analysis showed, even just school closure data usefully hints at how much learning loss has occurred; additional data types are likely helpful in fine-tuning estimates.

Hence, at this moment, there is tremendous value within education systems in striving for all data relevant to the building back agenda to be made accessible through a central platform. This would allow easier access and use of common data by all stakeholders for management and decision making. For accountability purposes it would also allow triangulation of data from citizen-led surveys that directly assess students’ educational attainment with the official EMIS data on learning levels. In-school formative assessments provide data on learning levels within classrooms and on how well, given concerns about widening variance of abilities in the classroom, teachers are managing to teach at the right level. (Teaching at the right level involves teaching to students’ actual learning level instead of following standardised curricula that are likely to be over-ambitious for many students returning to in-person learning.) Gathering together these different forms of data in one place would help education policymakers make decisions according to the specific shape of current needs, during the building back period and beyond, by, for example, more precisely deploying limited resources for teacher training to support teaching at the right level so that they reach the particular geographical regions and class types where variance has grown widest.

5.2. School level

At the school level, two forms of data are needed. The first is standardised summative assessments of students’ learning so that teachers get a sense of individual students’ levels (as well as to inform educational authorities’ investments and policy decisions). The second is formative assessments for teachers to receive feedback on how effectively they are addressing the range of needs. Increasing teacher monitoring may be a sensitive issue, and so should be done in discussion with, and ideally in collaboration with, teachers’ groups. The pandemic thoroughly modified the conditions in which teachers performed their work. They have been forced to adapt to new situations, and many are exhausted. Moreover, they are now faced with the heightened challenge of helping students who, during the period when schools were closed, have studied very differently or not at all.

This context is the ideal time for education authorities to respond by providing tools and training that enable teachers to integrate formative assessments as a central pillar of education delivery, whilst empowering them to innovate with evidence-based approaches to boost learning and enhance classroom management practices. There is ample evidence that ‘teaching at the right level’ can lead to large foundational learning gains (Teaching at the Right Level, 2022). Indeed, some recent modelling work suggests that rolling out teaching at the right level alongside supportive remedial adaptations, such as providing training to assist teachers to adjust their pedagogical practices accordingly (Banerjee et al., 2016), and to prioritise students’ acquiring essential skills, could even recoup pandemic-period learning losses (Kaffenberger, 2021).

The research community continues to contribute to efforts that furnish teachers with better data on how to improve in-person classroom practice, and implement remedial strategies. Yet more could be done to serve education delivery. Currently, the widely accepted tools to assess classroom practice have been designed with academics in mind as their primary users, rather than to generate data for teachers and supervisors to use in peer-to-peer feedback. There is a huge opportunity to develop such tools, to test them in the field, and to provide the means to distribute more widely the teaching insights that they generate. It may aid students’ learning outcomes over time to link teachers’ career progression to performance, although this is a politicised matter in some countries.

5.3. Household level

Encouragingly, large-scale household-based assessments have expanded across the globe in the past decade. For example, they are now conducted in Pakistan and across many sub-Saharan African and Latin American countries (see PAL Network for details). But this dissemination could go further, and needs to be urgently redeployed post-pandemic school closures. Large-scale household surveys that collect data to assess students’ learning, such as ASER, may provide less elaborate data on learning attainment than in-school summative assessments, but are valuable because they tend to be more reliable. As such they are a helpful comparison with school data, and should be incorporated into government decision-making.

A common weakness of household surveys is that there are few widely adopted ‘standard’ survey items (questions), which means that comparison across surveys is more imperfect than could be the case. This is especially problematic when surveys within a country tend to be local or patchy in coverage. There are efforts to remedy this, however. The pandemic led to a research consortium called the EdTech Hub partnering with Oxford University and creating a working group of nine leading academics across eight institutions, known as the Building EdTech Evidence and Research (BETER) group. One strand of this group’s work examined education-related surveys and collated them to identify data gaps. Most significantly, BETER members identified survey items relevant for inclusion in household (rather than school-based) surveys across countries. Items were developed in the research domains of: learning support, challenges during COVID-19, school re-opening, household experiences and government response. Based on these efforts, various developers of generalist surveys (i.e. surveys that are not specific to gathering data about education) have recognised the potential to integrate specific education items beyond the pandemic, in order to increase the quality and quantity of data available to education policymakers.

6. Conclusion

Even before school closures caused by COVID-19, educational inequality has been a norm across and within most countries around the world (Lucas, 2001; Darling-Hammond, 1995, Heckman, 2011, Jencks et al., 2014). Differences in access to opportunities to learn directly affect children’s learning outcomes. Without diplomas and appropriate skills, students’ employment opportunities will be severely limited. All indications suggest that COVID-19 has exacerbated this, and without evidence-based remediation, will continue to impact the learning of current students for years to come.

This paper has outlined the pressing need for an improved data ecosystem in education that has been made more urgent by the widespread learning loss brought on by school closures during the COVID-19 pandemic. Using the available data, researchers have sought to contribute to the building back from COVID-19 agenda on many fronts. From the educational policy perspective, they have provided estimates of the impact of the COVID-19 pandemic on learning (Selvaraj et al., 2021; Alban Conto et al., 2021; Angrist et al., 2021) and the potential widening of learning gaps between different student populations (Andrew et al., 2020, Bonal and González, 2020, Engzell et al., 2021; Haelermans et al., 2022). Moreover, they have sought to estimate the effectiveness of interventions adopted during the crisis (Angrist et al., 2020), and have moved to study the recovery policies that work to mitigate its impact on students (Kaffenberger, 2021). However, taken together, these efforts seem sporadic and limited in geographical coverage, in a context when a new level of global effort and commitment is needed. The limitations are often due to available data. Indeed, the paucity of crucial types of data discussed in this paper make more detailed studies and the potential policy use of their findings challenging. Both to inform the building back agenda, and to provide future opportunities for evidence-based policymaking in education, an improved data ecosystem is required.

Our recommendations focus on how to improve knowledge of precisely where the learning gaps are and how severe they are. These were already priorities among education policymakers, but are even more pressing due to school closures prompted by the COVID-19 pandemic. Further, we have argued that the impact of educational interventions during the pandemic needs more careful assessment – with particular attention paid to the conditions under which common interventions have been more or less effective. As noted in our earlier discussion, in-class learning has been interrupted over the years prior to the COVID-19 pandemic by various natural and political emergencies. Appropriate evaluation of interventions during the pandemic could thus contribute to creating improved infrastructure (e.g. EMIS) and competencies (e.g. teachers’ professional skills) that build general robustness for potential future interruptions to in-class learning, whatever their cause.

Considering planning and system management at the governmental level, strengthening data collection that can inform decision making is crucial. Educational systems would broadly benefit from developing a culture that relies on reliable information coming from the ground up to better understand the needs and vulnerabilities of students, teachers, and school professionals. Improving EMIS is paramount for this.

At the school level, data collection and provision should be geared towards supporting teachers and other school professionals. In the challenging context of students returning to in-person classes following long periods of school closures, schools need to be able to access variance in learning levels and respond to it. Initiatives to provide teachers with the necessary skills and information to evaluate and teach in such conditions are central to alleviating pandemic learning losses, and more generally to reducing the widespread problem of struggling students being left behind. Alternative forms of assessment, in particular household surveys that incorporate standard survey items, can be helpfully drawn upon in parallel to classroom assessments.

Many of our data collection recommendations could be taken forward by the research community, which has stepped forward in countless ways during the pandemic to provide policymakers with evidence of how to reduce the spread and health effects of COVID-19. There is ample opportunity for academics to continue to contribute to an improved data ecosystem in education. In the years to come, assisting policymakers in the mitigation of learning losses incurred during school closures and making inroads into the wider learning crisis is a prominent global need.

CRediT authorship contribution statement

Filipe Recch: Conceptualization, Investigation, Formal analysis, Writing – original draft, Writing – review & editing. Anna Petherick: Conceptualization, Investigation, Formal analysis, Writing – original draft, Writing – review & editing, Supervision. Rachel Hinton: Conceptualization, Investigation, Formal analysis, Writing – original draft. Radhika Nagesh: Writing – original draft, Writing – review & editing. Rodrigo Furst: Data curation, Visualization. Rafael Goldszmidt: Validation, Conceptualization, Writing – review & editing.

Appendix

See in Fig. A1, Fig. A2, Fig. A3, Fig. A4, Fig. A5, Fig. A6.

Fig. A1.

Number of partial school closure days Jan 2020 - Dec 2021. This figure depicts the total number of partial (some levels) schools closure days in the whole country or in at least in one subnational region from January 2020 to December 2021.

Source: Oxford COVID-19 Government Response Tracker (available at https://github.com/OxCGRT/covid-policy-tracker).

Fig. A2.

Number of full school closure days Jan 2020 - Dec 2021. This figure depicts the total number of full (all levels) schools closure days in the whole country or in at least in one subnational region from January 2020 to December 2021.

Source: Oxford COVID-19 Government Response Tracker (available at https://github.com/OxCGRT/covid-policy-tracker).

Data Availability

Data will be made available on request.