In-Person Instruction and Educational Outcomes of K–8 Students During the COVID-19 Pandemic

Abstract

OBJECTIVES

Quantify the relationship between district policy permitting in-person instruction and educational outcomes during the 2020 to 2021 academic year for kindergarten through eighth grade students.

METHODS

An ecological, repeated cross-sectional analysis of grade-level proficiency of students enrolled in public school districts in North Carolina (n = 115 school districts) was conducted. Univariate and multivariate analyses were performed to evaluate the association between the proportion of the school year a district spent in-person and 2020 to 2021 end-of-year student proficiency in the district. We then fit a multivariable linear regression model, weighted by district size, and adjusted for district-level 2018 to 2019 proficiency and district-level factors (rural or urban, area deprivation).

RESULTS

Compared to 2018 to 2019, there was a 12.1% decrease (95% confidence interval [CI]: 16.8–19.3) in mathematics and an 18.1% decrease (95% CI: 10.8–13.4) in reading proficiency across the state at the end of 2020 to 2021. Compared to a district that remained entirely remote for the 2020 to 2021 school year, a district offering full in-person instruction had 12% (95% CI: 11%–12.9%) and 4.1% (95% CI: 3.5%–4.8%) more students achieve grade-level proficiency in mathematics and reading, respectively. In-person instruction was associated with greater increases in mathematics proficiency than reading, and greater increases in elementary-level students’ proficiency than middle school–level.

CONCLUSIONS

The proportion of students achieving grade-level proficiency in 2020 to 2021 fell below prepandemic levels at each evaluated time point in the academic year. Increased time spent in-person by a school district was associated with an increased proportion of students achieving grade-level end-of-grade proficiency in both mathematics and reading.

At the start of the coronavirus disease 2019 (COVID-19) pandemic in March 2020, many school buildings within the United States closed, with most transitioning to remote instruction, in hopes of preemptively limiting disease transmission. In the following 2020 to 2021 academic year, school building reopening plans and access to in-person instruction varied greatly across districts in the United States,¹ despite evolving evidence that in-person instruction could occur safely.^2,3 Prolonged school building closures and reduced in-person instruction were often associated with increasing community socioeconomic disadvantage⁴ rather than higher reported community COVID-19 rates.⁵ Furthermore, only 60% of students regularly attended educational sessions⁶ when schools transitioned to virtual classrooms, with students from historically disadvantaged demographics disproportionately affected, thereby further compounding academic inequities.^7,8 Early national data suggested that by the end of the 2020 to 2021 academic year, kindergarten through 12th grade (K–12) students were ∼5 months behind in mathematics and 4 months behind in reading compared to previous years.⁸

Even before the COVID-19 pandemic, the importance of regular school attendance for academic achievement is well documented. Students who miss <10% of a school year have a greater chance of academic success and increased probability of high school graduation as compared to students who missed >10% of a school year.⁹ Inability to achieve mastery of grade-level skills results in difficulties advancing through future grade levels and places students at higher risk of dropping out of school. Long-term, students who drop out of school have reduced lifetime earnings,¹⁰ are more likely to rate their health as poor,^11,12 are more likely to interact with the criminal justice system,^11,13 and have decreased life expectancy compared to those who complete high school.¹⁴

To date, much of the research around COVID-19-related K–12 school closures focused on risk and mitigation of COVID-19 disease. Nevertheless, with proven mitigation strategies and vaccinations available for school-aged children, the greatest public health risks actually lie in the consequences of school shutdowns, including preventable education loss. To date, there are limited evaluations quantifying the impact of in-person instruction secondary to specific school district policies, yet understanding this impact is critical to inform prioritization of anticipated increases in federal funding for public schools⁸ and school closure policies during future pandemics. To address this need, we used educational data from kindergarten through eighth grade (K–8) students in North Carolina public schools to evaluate the association between in-person education time during the 2020 to 2021 academic year and educational outcomes.

Methods

We first conducted a statewide, district-level, time-trend, ecological analysis comparing grade-level proficiency measured by 2 state-administered standardized end-of-grade tests given before and after the onset of the COVID-19 pandemic. The statewide analysis included all elementary and middle schools (third through eighth grade [grades 3–8]) in NC public school districts that completed state-issued standardized assessments during the 2018 to 2019 and 2020 to 2021 academic years. Separately, we invited 13 districts participating in the ABC Science Collaborative¹⁵ to provide individual-level, deidentified, district-administered, standardized testing data for K–8 students during the 2018 to 2019, 2019 to 2020, 2020 to 2021 academic years. Three NC school districts provided data for the study. Each participating district was urban and offered either hybrid or fully in-person instruction for the majority of the 2020 to 2021 academic year (range: 75%–100%). Student enrollment among the districts ranged from 5000 to 20 000. We then conducted a cross-sectional analysis of grade-level proficiency on beginning-of-year (BOY) and middle-of-year (MOY) standardized tests, comparing grade-level proficiency across these districts. Supplemental Table 5 contrasts the study characteristics of the statewide and district-wide analyses.

Statewide Analysis

We obtained publicly available standardized testing data for the 2018 to 2019 and 2020 to 2021 academic years on state-issued, end-of-grade mathematics and reading examinations from the NC Department of Public Instruction (DPI) “School Assessment and Other Indicator Data” reports.^16,17 For these reports, the proportion of students achieving grade-level proficiency (a score of level 3 or higher) is available at the level of the state, district, and school; at each level, the proportion of students achieving grade-level proficiency is stratified according to student grade and subject area. Additionally, aggregate student scores are available by demographic subgroups. Next, we used school district policy updates available via the NC School Board Association¹⁸ to determine the total number of weeks of the 2020 to 2021 academic year that each grade within each school district offered fully in-person, hybrid, or remote instruction.

Definitions

We included the following factors as potential confounders of student academic achievement: rural or urban status and median area deprivation index (ADI) rank score corresponding to each district’s surrounding county. We characterized a school district as rural if the National Center for Health Statistics Rural-Urban Classification Scheme assigned the surrounding county a classification of 5 or 6 (micropolitan and noncore), and urban if the county was assigned 1 to 4 (large central metropolitan, large fringe metropolitan, medium metropolitan, small metropolitan).¹⁹ We calculated district-level ADI rank scores as a measure of socioeconomic disadvantage accounting for income, education, employment, and availability of quality housing within the district. We used the University of Wisconsin School of Medicine and Public Health’s Neighborhood Atlas²⁰ to assign each school district the median national ADI rank across the census block groups that comprised each corresponding county. We then dichotomized the median ADI rank as higher ADI (national rank >50%), indicating greater socioeconomic disadvantage, or lower ADI (national rank ≤50%).²¹ Of the total 115 school districts in NC, 15 are city school districts; for these, we applied the values for the rural or urban status and ADI rank score that corresponded to the surrounding county. Additionally, we examined historic district proficiency defined as the proportion of students achieving grade-level or higher proficiency in 2018 to 2019 as reported by the NC DPI. Analyses were weighted by school district size; this was determined by the number of students enrolled in the district during the 2020 to 2021 academic year (publicly available via NC DPI). Finally, for descriptive analyses, we defined in-person time as the proportion of the year that districts offered hybrid or full in-person education, categorized as <33%, 33% to 66%, or >66% of the year.

The student-level demographic subgroups included race or ethnicity, student economic status, disability status, English learner (includes English as a second language and English language learners), and migrant status. In accordance with the Every Student Succeeds Act, students were considered economically disadvantaged if living in a household earning no more than 185% of the federal poverty threshold (i.e., for a family of 4 with 2 children: $25 100 in 2018, $26 200 in 2020).^22,23 Student race and ethnicity were categorized in accordance with reporting via parent or caregiver as a single mutually exclusive variable including the following categories: American Indian, Asian, Black, Hispanic, White, or 2 or more races. Student disability status was determined according to the Individuals with Disabilities Education Act²⁴; status as an English learner or migrant was defined according to Title III definitions.²⁵

Statistical Analyses

We conducted descriptive analyses of the 115 districts. We performed univariate analyses of the change in student proficiency in 2020 to 2021 relative to prepandemic 2018 to 2019 grade-level proficiency by relevant demographic factors and the categorized in-person time variable, with associated 95% confidence intervals (CIs). To evaluate the association between district time spent in-person and 2020 to 2021 student proficiency in the district, we fit a multivariable linear regression model adjusted for district-level 2018 to 2019 proficiency and district-level factors (rural or urban, ADI). District level proficiency was weighted by district size (by using 2020–2021 enrollment). The exposure variable (district time spent in person) was a modeled as a continuous variable; to understand the overall impact of school closures, we calculated the change in proficiency associated with a 10% increase in the proportion of the school year spent via in-person instruction. To estimate how time spent in-person impacted proficiency for different demographic subgroups, we repeated the multivariable model by student demographic groups.

District policies regarding in-person instruction varied by grade level, with elementary school students often returning to in-person instruction sooner than higher grade levels. Therefore, we conducted a secondary analysis to evaluate 2020 to 2021 proficiency by school level (traditional elementary [grades kindergarten through fifth grade, K–5] versus middle school [6th through eighth grade, 6–8] levels) using the time spent in-person by school level. Finally, we conducted a sensitivity analysis where the proportion of time spent in-person was represented by policies that allowed full in-person instruction at all grade levels (hybrid format recategorized as not fully in-person). Statistical analysis was conducted by using R Statistical Software (v3.6.1; R Core Team 2019).

District-Wide Analysis

Three districts provided district-administered (but not publicly reported) BOY and MOY mathematics and reading test scores for the 2018 to 2019, 2019 to 2020, and 2020 to 2021 academic years. All districts used state-approved platforms,²⁶ including i-Ready testing (Curriculum Associates, North Billerica, MA)²⁷ by 2 districts, and Star Assessments (Renaissance Learning, Wisconsin Rapids, WI)²⁸ by 1 district. Student proficiency on BOY and MOY assessments was reported as below, at, or above grade-level. Districts also provided demographics including number of students enrolled and race or ethnicity composition aggregated by school.

Statistical Analysis

We performed descriptive analyses to determine and compare the proportion of students achieving grade-level BOY or MOY proficiency, stratified by subject-area during the “prepandemic” (2018–2019, 2019–2020) and “pandemic” (2020–2021) school years. Using difference in proportions calculations, we compared the proportion of students achieving BOY or MOY proficiency during prepandemic and pandemic years for each district.

Results

Statewide Analysis

Across the 115 NC districts, the median proportion of the year with hybrid or full in-person instruction allowed was 79.8% (quarter 1 46.7, quarter 3 97.3). Sixteen districts allowed in-person instruction for <33% of the year, 22 districts for 33% to 66%, and 77 districts for >66% of the year. Most districts were classified as rural (N = 60); 55 were classified as urban. District median ADI rank was 69.1 (quarter 1 58.7, quarter 3 76.6). Table 1 summarizes students tested in 2020 to 2021, schools represented, and student-level demographics. Supplemental Table 6 displays the district-level demographics and policies, and their associated levels of student proficiency.

TABLE 1.

Statewide Analysis: Demographics of Schools and Students Represented, by Subject and Year

Student Demographic	Mathematics				Reading
	2018–2019a	2020–2021			2018–2019a	2020–2021
	Schools Represented	Students Expected	Students Tested, %	Schools Represented	Schools Represented	Expected Students	Students Tested, %	Schools Represented
All students	1878	704 929	93	1838	1892	708 454	93	1855
Economic statusb
Economically disadvantaged	1869	286 179	92	1793	1883	286 554	92	1816
Not economically disadvantaged	1849	418 750	94	1830	1859	421 900	94	1815
Race or ethnicityc
American Indian	127	7715	>95	114	128	7721	>95	104
Asian	471	26 108	91	503	531	26 808	91	438
Black	1580	179 856	90	1458	1590	180 189	90	1453
Hispanic	1697	141 642	94	1667	1702	141 982	94	1574
White	1748	312 689	95	1718	1754	314 683	95	1681
Two or more	1203	35 896	93	1188	1216	36 044	93	922
Disability statusd
Student with a disability	1732	91 775	91	1497	1729	91 806	91	1416
Not a student with a disability	1849	613 154	94	1823	1872	616 648	94	1834
English learnere	1328	84 089	93	1159	1261	84 119	95	942
Migrantf	100	N/A	N/A	36	94	N/A	N/A	30

Values are n unless noted otherwise.

^a There is no value for expected students and students tested in 2018–2019. Historically, before the COVID-19 pandemic, 95% of a demographic subgroup was required to test for public reporting of the subgroup.

^b Economic status is defined in accordance with the Every Students Succeeds Act; a categorization of economic disadvantage is applied to individuals living in a household earning no more than 185% of the federal poverty threshold).²²

^c Race or ethnicity is categorized in accordance with North Carolina Department of Public Instruction (NC DPI) reporting via parent or caregiver as a single, mutually exclusive variable including the following categories: American Indian, Asian, Black, Hispanic, White, or 2 or more races.

^d Disability status is determined according to the Individuals with Disabilities Education Act.²³

^e English Learner status was defined according to Title III definitions.²⁴

^f Migrant status was defined according to Title III definitions.²⁴ Not reported.

Statewide, there was a 12.1 percentage point decrease (95% CI: 16.8–19.3) in mathematics proficiency and an 18.1 percentage point decrease (95% CI: 10.8–13.4) in reading proficiency at the end of the 2020 to 2021 academic year compared to 2018 to 2019. Table 2 includes the univariate analysis with relative change in proficiency across all relevant demographic subgroups.

TABLE 2.

Proportion of Students (Grades 3–8) Achieving Grade-Level End-of-Grade Proficiency, by Student Demographic Subgroup

	Mathematics			Reading
Student Demographic	2018–2019 (% Proficient)	2020–2021 (% Proficient)	Difference in Proficiencya (95% CI)	2018–2019 Academic Year (% Proficient)	2020–2021 Academic Year (% Proficient)	Difference in Proficiencya (95% CI)
All students	56.7	38.6	−18.1 (−19.3 to −16.8)	54.1	42.0	−12.1 (−13.4 to −10.8)
Economic statusb
Economically disadvantaged	46.7	26.4	−20.3 (−21.5 to −19.1)	43.6	30.2	−13.4 (−14.6 to −12.2)
Not economically disadvantaged	67.1	46.9	−20.2 (−21.4 to −18.9)	65.4	50.6	−14.9 (−16.1 to −13.6)
Race and ethnicityc
American Indian	48.6	23.9	−24.7 (−29.6 to −19.7)	46.2	30.5	−15.7 (−20.9 to −10.5)
Asian	75.6	62.8	−12.8 (−15.1 to −10.4)	68.3	61.8	−6.5 (−9.0 to −4.0)
Black	41.2	22.3	−18.9 (−20.2 to −17.6)	40.5	28.4	−12.1 (−13.4 to −10.8)
Hispanic	53.2	33.0	−20.2 (−21.5 to −18.9)	46.6	34.2	−12.3 (−13.7 to −11.0)
White	66.4	49.5	−16.9 (−18.1 to −15.5)	65.6	53.9	−11.7 (−13.0 to −10.4)
Two or more	56.4	37.5	−18.9 (−20.5 to −17.3)	57.8	44.1	−13.7 (−15.4 to −12.0)
Disability statusd
Student with a disability	23.4	16.7	−6.7 (−7.8 to −5.6)	20.9	15.0	−5.9 (−6.9 to −4.8)
Not a student with a disability	62.0	42.4	−19.6 (−20.8 to −18.3)	59.7	46.7	−13.0 (−14.3 to −11.7)
English learnere	37.2	21.1	−16.1 (−17.6 to −14.5)	23.3	14.2	−9.1 (−10.4 to −7.8)
Migrantf	40.3	19.6	−20.7 (−27.5 to −13.8)	33.8	19.6	−14.3 (−21.6 to −6.9)

Values reported as % (N), where N = number of schools contributing data

^a Difference in proportions

^b Economic status is defined in accordance with the Every Students Succeeds Act; a categorization of economic disadvantage is applied to individuals living in a household earning no more than 185% of the federal poverty threshold).²³

^c Race and ethnicity is categorized in accordance with North Carolina Department of Public Instruction (NC DPI) reporting via parent or caregiver as a single, mutually exclusive variable including the following categories: American Indian, Asian, Black, Hispanic, White, or 2 or more races.

^d Disability status is determined according to the Individuals with Disabilities Education Act.²⁴

^e English learner status was defined according to Title III definitions.²⁵

^f Migrant status was defined according to Title III definitions.²⁵

On multivariate linear regression, more time in person was independently associated with increased mathematics and reading proficiency during the 2020 to 2021 academic year at the district-level for all students and across the majority of studied demographic student groups (Table 3). Among all students, every 10% increase in proportion of the year allowing in-person instruction was associated with a 1.20 percentage point increase in students achieving grade-level proficiency in mathematics and a 0.41 percentage point increase in the proportion of students achieving grade-level proficiency in reading. Additionally, for nearly all studied student subgroups, increasing proportion of the year in-person was associated with a greater increase in mathematics than reading proficiency. On the secondary multivariate analysis of proficiency by school grade level (elementary [grades 3–5] versus middle [grades 6–8] school), in-person instruction remained independently associated with increased proficiency in 2020 to 2021; however, the impact of in-person instruction on elementary mathematics and reading proficiency was twice that of the impact on middle school (Supplemental Table 7). For each 10% increase in proportion of time in person, elementary students (grades 3–5) had a 1.40 percentage point increase in mathematics and a 0.46 percentage point increase in reading proficiency; whereas middle school students (grades 6–8) had a 0.84 percentage point increase in mathematics and a 0.29 percentage point increase in reading proficiency.

TABLE 3.

Multivariable Analysis of District-level End-of-Grade (2020–2021) Student Proficiency by a 10% Increase in District Time In-Person^a

Student Demographic	Change in % Students Proficient in Mathematics	Change in % Students Proficient in Reading
All students	1.20 (1.10–1.29)	0.41 (0.35–0.48)
Economic status
Economically disadvantaged	1.38 (1.29–1.46)	0.48 (0.40–0.55)
Not economically disadvantaged	1.20 (1.10–1.31)	0.48 (0.39–0.56)
Race and ethnicity
American Indian	0.42 (0.09–0.74)	1.36 (0.98–1.75)
Asian	0.64 (0.24–1.04)	0.42 (0.07–0.77)
Black	1.04 (0.93–1.15)	0.65 (0.55–0.75)
Hispanic	1.54 (1.43–1.65)	0.52 (0.42–0.62)
White	0.96 (0.86–1.07)	0.20 (0.10–0.29)
Two or more races	0.87 (0.69–1.04)	0.67 (0.51–0.83)
Disability status
Student with a disability	0.38 (0.29–0.48)	0.02 (−0.07 to 0.11)
Not a student with a disability	1.30 (1.21–1.40)	0.47 (0.41–0.54)
English learner	0.88 (0.74–1.03)	0.14 (0.02–0.27)
Migrant	1.37 (0.52–2.22)	0.96 (0.14–1.78)

^aModel estimates reported as β-coefficient (95% CI). Multivariate linear regression of change in the percentage point of students proficient modeled for a 10% increase in time the district allowed in-person instruction, adjusting for 2018–2019 proficiency, and controlling for district rural or urban status and area deprivation index rank score.

Finally, in a sensitivity analysis of districts allowing in-person instruction with minimal distancing (hybrid instruction categorized as not fully in-person), in-person time remained independently associated with increased district-level proficiency during the 2020–2021 academic year for all students. Here, every 10% increase in district in-person time was associated with a 3.89 percentage point increase in student grade-level proficiency in mathematics and a 1.15 percentage point increase in student grade-level proficiency in reading (Supplemental Table 7).

In addition to the above analysis that used 2020 to 2021 district-level proficiency data weighted by district size, model-based estimates were additionally calculated using unweighted 2020 to 2021 district level proficiency data. The results were consistent with the weighted model estimates and are reported in Supplemental Tables 8 and 9 (unweighted primary analysis and sensitivity analyses, respectively).

District-Wide Analysis of Beginning and Middle of Year Proficiency

The three districts that reported school-level data including district-administered BOY and MOY test scores offered either hybrid or fully in-person instruction for the majority of the 2020 to 2021 academic year (range: 75%–100%). Across the 3 districts, 3.2% fewer had BOY reading proficiency (95% CI: 2.3–4.1) and 3.7% fewer had MOY reading proficiency (95% CI: 2.8, 4.6) in 2020 to 2021 compared to the average proficiency in prepandemic years (2018–2019 and 2019–2020). For mathematics, BOY proficiency was 7.2 (95% CI: 6.3–8.1) percentage points lower and MOY proficiency was 9.4 (95% CI: 8.4–10.4) percentage points lower in 2020–2021 (Table 4). Supplemental Table 10 contains a breakdown of the proportion of students (grades K–8) achieving grade-level BOY and MOY proficiency by district and grade.

TABLE 4.

Proportion of Grade-level Proficient Students (Grades K-8) From 3 North Carolina on Beginning- and Middle- of Grade Testing

Time of y	Mathematics			Reading
	Prepandemic Average	2020–2021 Academic y	Difference, (95% CI), P	Prepandemic Average	2020–2021 Academic y	Difference, (95% CI), P
BOYa	33.7	26.5	−7.2 (−8.1 to −6.3)	37.3	34.1	−3.2 (−4.1 to −2.3)
MOYa	48.6	39.3	−9.4 (−10.4 to −8.4)	46.8	43.1	−3.7 (−4.6 to −2.8)

^aAggregate (n = 3 districts)

Discussion

This study quantifies the impact of in-person instruction during 2020 to 2021 on K–8 grade level proficiency in North Carolina. To add to existing alarming data on the impact of the COVID-19 pandemic on educational outcomes, we modeled the exposure variable of district time spent in person as a continuous variable to allow for quantification of the association between district policies and student proficiency. We found that districts that permitted greater access to in-person education had a significantly higher proportion of students achieving grade-level proficiency across almost all student demographic factors. These impacts were more pronounced for students in elementary grade levels, thereby highlighting the importance of in-person instruction during these formative years.

We then examined school-level outcomes within three districts where in-person instruction was offered for >75% of the school year. In this focused analysis, the student performance on standardized tests remained substantially impacted, perhaps because of the widespread school closings in March through June 2020, as well as the requirement for hybrid instruction for much of the 2020 to 2021 academic year. Students within these three participating districts began the year behind typical starting points (after March–June 2020 closure) and were unable to progress to prepandemic performance levels by the middle of the academic year (after required hybrid instruction). Performance decline was especially profound in mathematics proficiency.

Overall, the patterns in student performance seen here comport with publicly reported data elsewhere^29,30 and are consistent with previous summaries estimating the impact of learning loss on student performance after COVID-19–related school closures.^8,31,32 Even in best-case scenarios with brief school closures outside of the United States, minimal progress was made by students learning from home early in the COVID-19 pandemic, and disadvantaged children were most impacted.³³ In particular, our results support findings that student performance in mathematics was more impacted by school closures than reading (although performance in both subjects was negatively impacted),^8,29,34 and increased time spent learning remotely correlated with decreased student proficiency.^31,32,35 Several preexisting educational gaps have been well described: economically disadvantaged students have historically achieved lower proficiency than their economically advantaged counterparts, and disparities in access to quality education and ongoing structural racism contribute to racial disparities in the academic proficiency of Black and Hispanic students.³⁶ Our stratified analyses across economic and racial and ethnic subgroups indicated that after adjusting for historical proficiency, both socioeconomically advantaged and disadvantaged groups, as well as all racial or ethnic groups, benefited from increased in-person instruction except for students of Asian race.

Given the exposure-dependent relationship between increased access to in-person instruction and overall student proficiency in 2020 to 2021, policy measures are urgently needed to: (1) remedy existing achievement gaps for all students, especially those subject to uneven implementation of in-person instruction; and (2) strengthen the infrastructure needed to prevent school-building closure in the face of future COVID-19 surges or other infectious threats. With the distribution of education relief funding planned,⁸ urgent actions are needed. Without intervention, the thousands of students who became disengaged from school during the pandemic are at risk for dropping out altogether.⁸ Additionally, lifetime earnings losses for this cohort of K–12 students could continue to trend upwards and exceed the current estimate of $49 000 per person.⁸ To prevent extended learning loss and act upon lessons learned, pandemic relief policies like the Coronavirus Aid, Relief, and Economic Security Act³⁷ and pandemic preparedness policies like the drafted Prepare for and Respond to Existing Viruses, Emerging New Threats, and Pandemics (PREVENT) Act³⁸ must address strategies to tackle the widening educational gaps left in the wake of this current pandemic and support educational infrastructure in the setting of future infectious disease threats. However, failure to allocate specific funds for addressing ongoing educational gaps within the Coronavirus Aid, Relief, and Economic Security Act³⁹ and the lack of mention of K–12 education and schools within the developing Prepare for and Respond to Existing Viruses, Emerging New Threats, and Pandemics Act (PREVENT) bill⁴⁰ are cause for concern. Within NC alone, public school enrollment dropped considerably in the fall of 2020 with 25 000 fewer students enrolled statewide than the year prior. Therefore, the results reported here may underestimate district-level learning losses given that standardized testing reports are unable to capture students who did not return to the classroom.

Our study has several strengths. Using primarily publicly available data, we have synthesized a timely and in-depth statewide educational outcome summary detailing the association between district-level variation in in-person instruction offered and student achievement of grade-level proficiency. Furthermore, our findings can serve as a baseline to compare against the effectiveness of future strategies to remedy pandemic-related learning loss. Given ongoing disruptions to education because of quarantine after severe acute respiratory syndrome coronavirus 2 exposures,⁴³ our findings can be used to support strategies that limit disruptions to in-person education, such as test-to-stay or mask-to-stay strategies.

Our study also has several limitations. First, standardized testing data were used as a proxy for academic achievement; however, test score data are imperfect, providing just a cross-sectional data point and, therefore, should not be considered a sufficient representation of academic ability or progress. Second, the generalizability of our results is limited given the ecological study design used. Individual-level exposures were not available, so district-level exposures were applied to all individuals in the district, regardless of individual-level experiences. District policy does not equate to individual student return to in-person instruction, and attendance data summarizing the proportion of students returning to in-person learning once district policy allowed students to do so was unavailable. Third, on statewide analyses, individual data were aggregated into relevant demographic subgroups, thereby preventing analysis of the combined effect of multiple student-level demographic predictors within one predictive model for end-of-grade proficiency. Finally, in statewide analysis, prepandemic performance was represented by a single academic year (2018–2019) and may not account for year-to-year variability in district proficiency.

Conclusions

During the 2020 to 2021 academic year, the proportion of students achieving grade-level proficiency fell below prepandemic levels at each evaluated time point in the academic year. Increased time in-person was associated with increased grade-level end-of-grade proficiency in both mathematics and reading, with greater impacts on mathematics than reading proficiency, and in elementary students compared to middle-school students. Ongoing educational and policy efforts should prioritize in-person instruction for students and target interventions toward students in schools that had prolonged school closures during the COVID-19 pandemic.

Glossary

ADI

area deprivation index

BOY

beginning-of-year

CI

confidence interval

COVID-19

coronavirus disease 2019

DPI

Department of Public Instruction

K–8

kindergarten through 8th grade

K–12

kindergarten through 12^th grade

MOY

middle-of-year