Impacts of the Chicago School Readiness Project on Measures of Achievement, Cognitive Functioning, and Behavioral Regulation in Late Adolescence

Abstract

The current paper reports long-term impacts of the Chicago School Readiness Project (CSRP) on measures of achievement, cognitive functioning, and behavioral regulation taken toward the end of students’ high school careers. The CSRP was a self-regulation-focused early childhood intervention implemented in Head Start centers serving high-poverty neighborhoods in Chicago. The intervention was evaluated through a cluster randomized control trial, providing us with rare longitudinal evidence from an experimental study. However, the study was limited by issues with low power and baseline differences between experimental groups. Here, we report on follow-up data taken approximately 11 to 14 years after program completion, including measures of participants’ (n = 430) academic achievement, executive functioning, emotional regulation, and behavioral problems, and we provide a range of analytic estimates to address the study’s methodological concerns. Across our estimates, we found little evidence that the program had lasting impacts on indicators of late-adolescent functioning. Main effects were estimated with some imprecision, but nearly all models produced null effects across the broad array of outcomes considered. We also observed few indications that effects were moderated by post-treatment high school quality or later assignment to a light-touch mindset intervention. Implications for developmental theory and early childhood policy are discussed.

Introduction

As larger shares of children enroll in publicly funded early childhood educational (ECE) programs (e.g., Friedman-Kraus et al., 2021), our need for causal evidence regarding the long-term effects of early intervention has never been greater. The findings from foundational studies comparing the outcomes of children who did and did not participate in ECE programming continue to drive expectations that high-quality investments in early educational experiences should lead to long-lasting changes in developmental trajectories (Elango et al., 2016), yet we sorely lack long-term evidence from randomized control trials of more recent programs (Watts et al., 2019).

The need for more long-term evidence regarding whether high-quality ECE benefits children also dovetails with recent interest in understanding how to optimize these benefits using specific curricular interventions and enhancements (Jenkins et al., 2018). For example, the apparent importance of self-regulation for supporting early school readiness skills (Blair & Raver, 2015) has helped fuel the development of interventions within ECE programs intended to boost children’s early executive function and behavioral regulation (see review in Nesbitt & Farran, 2021). Indeed, correlational research has shown that various domains of early self-regulation predict a broad set of measures indicating adult functioning (e.g., executive function: Ahmed et al., 2021; self-control: Moffitt et al., 2011), which raises the question of whether early interventions designed to support child self-regulation may also produce longer-term impacts.

The current study aims to shed light on these questions by reporting new estimates of the long-term impacts of the Chicago School Readiness Project (CSRP), a self-regulation focused intervention that attempted to improve children’s development and learning in Head Start classrooms serving high-poverty areas of Chicago. The intervention provided a range of services to teachers and children beyond those typically offered in Head Start, including pedagogical development sessions that were designed to help teachers alter their approach to behavioral management. We present new intervention impacts estimated approximately 11 to 14 years following the initial preschool intervention period. These outcomes span a variety of measures covering developmental domains originally targeted by the early childhood intervention, including executive function, behavioral regulation, and academic achievement. We also explore effects on more distal outcomes that could also indicate better late adolescent functioning, such as civic involvement and extracurricular activity participation. Follow-up measures were collected using a range of modalities, including surveys, direct assessments, and administrative records taken across multiple follow-up waves when participants were near the end of their high school careers. This comprehensive approach allows us to test whether long-term impacts of the CSRP intervention may be detectable into adolescence, more than a decade after individuals had received services. We also examine issues of heterogeneity based on key student characteristics and other environmental experiences.

Long Term Impact of Early Interventions

Highly cited evidence from small studies of several intensive ECE programs tested nearly 50 years ago suggests that exposure to high-quality ECE experiences can lead to improved long-term outcomes for children from low-income families (see Elango et al., 2016). However, the long-term impacts of modern-day ECE programs are not well understood. One recent review found that most effects of ECE interventions faded substantially in the years immediately following treatment (Bailey et al., 2017), and similar fadeout effects have been observed throughout the educational intervention literature (Bailey et al., 2020). Many theories have been proposed for why early effects may fade, with some arguing that fadeout happens because subsequent schooling environments are of insufficient quality to maintain the gains children make during the preschool year (Zhai, et al., 2012). Others have pointed to the possibility that some ECE programs may target skills that have natural ceilings (e.g., letter naming), such that programs targeting “unconstrained” skills (e.g., language development) may lead to longer-term persistence (McCormick et al., 2021).

This picture of ECE fadeout is further complicated by studies that have shown patterns of impact emergence in the long-run, even in instances where medium-run effects were not detected. This pattern is probably best exemplified by evidence on the Boston public preschool program. Gray-Lobe and colleagues (2021) used admissions lotteries to test the medium- and long-term effects of attending a Boston preschool. Perhaps surprisingly, they found virtually no effect of attending preschool on student achievement scores in elementary school, yet they detected statistically significant positive effects on measures of high school graduation and college attendance measured years after preschool exposure (also see recent evidence on the modern version of Boston preschool from Weiland et al., 2019). Similar patterns of “emergence” can also be found in research on Head Start participation (Deming, 2009). These findings are puzzling and suggest that the mechanism linking early ECE experiences to key long-term outcomes may not be captured by achievement test scores. Some have interpreted patterns of emergence as evidence that social-emotional skills may provide the basis for long-term ECE impact (e.g., Elango et al., 2016), underscoring the need for research that can test links between early childhood educational experiences and other developmental mechanisms—such as the development of cognitive, behavioral, and emotional regulation.

Self-Regulation and Early Intervention

In recent years, ECE research has expanded to studies of curricular interventions that aim to enhance the programmatic quality of existing ECE services. When considering the types of curricular interventions that may have long-lasting effects on disadvantaged children’s trajectories, research suggests self-regulation as a potentially potent early intervention target. Self-regulation, broadly defined, describes one’s ability to modulate behavior, thoughts, and emotions to reach a desired goal (see Bailey & Jones, 2019; Inzlicht et al., 2021). Research in this area has focused on executive function as a key cognitive regulatory mechanism that supports the development of self-regulation, as it involves the flexible allocation of attention when automatic cognitive processing cannot be relied upon (Blair, 2016). Other work on self-regulation stresses the modulation of emotions and behaviors in real-world contexts to engender positive social relations with family, peers, and teachers (Blair & Raver, 2015). Indeed, emotional regulation may also develop reciprocally with executive function, as early emotional regulation may lay the foundation for the development of the more complex cognitive skills involved in EF (Blair & Ursache, 2011).

Recently, researchers have examined the possibility that self-regulation may be a key mediator between early exposure to poverty and negative long-term outcomes (see review in Raver, 2012). This theory points to the possibility that environmental adversity in early childhood can disrupt children’s stress response physiology, leading to problems with the regulation of cognitive, behavioral, and emotional processes (Blair, 2016). These early issues with self-regulation can manifest as later risk for poor academic achievement and higher rates of behavioral problems once children reach school age (Blair & Raver, 2015). Longitudinal research suggests that such problems may have serious implications for children’s long-term development. For example, Moffitt and colleagues (2011) found that an index of measures of behavioral and attention problems (termed “low self-control”) measured during childhood strongly predicted outcomes broadly related to health and well-being at age 32 after accounting for sex, SES, and IQ (also see a recent replication from Koepp et al., in press). More recently, Ahmed and et al. (2021) found that an aggregated measure of early executive function positively predicted educational attainment by age 26, even when a host of early-life control variables were considered.

Perhaps not surprisingly, interest has grown in ECE interventions targeting self-regulation. Recent large-scale evaluations of various social-emotional learning programs that include regulatory components have produced mixed results (see review by McClelland et al., 2021), perhaps owing to the fact that these studies often feature a relatively modest treatment contrast, with control group participants receiving comprehensive ECE services themselves (e.g., full-day preschool, Head Start). Of note, the Tools of the Mind program, an ECE curriculum designed to support executive function and self-regulation, has received a substantial amount of research attention, though the efficacy of the program remains unsettled. Some evaluations of Tools of the Mind have found promising results, with students in intervention classrooms exhibiting decreased problem behaviors and improved math achievement in the short term (for review, see Baron et al., 2017). However, a recent large-scale intervention of the curriculum found largely null results across a host of cognitive and behavioral measures (Nesbitt & Farran, 2021), suggesting the program impacts are sensitive to setting and implementation factors.

The mixed findings from various Tools of Mind evaluations are similar to the findings reported from a large-scale evaluation of three different social-emotional learning curricula tested in approximately 100 Head Start centers (Morris et al., 2014). The Head Start CARES study tested the effects of the Incredible Years Teacher Training Program (which also played a role in the intervention considered in the current study), the Preschool PATHS curriculum, and the Tools of the Mind-Play curriculum on various short-term measures of classroom processes and child outcomes in the short term. This large-scale study likely represents the most rigorous evaluation of social-emotional curricula for early childcare settings to date, and it found mixed results. All three curricula led to some improvements in classroom processes, but findings for child outcomes were less consistent. The programs tended to produce positive effects on measures of child social functioning and emotional recognition, but null effects were found on measures of executive functioning and pre-academic skills.

Finally, the Head Start REDI project targeted early social-emotional competence (through the preschool PATHS curriculum) and early literacy (Bierman et al., 2008). The Head Start REDI project is unusual, as it involved a randomized control trial (RCT) design and follow-up of the original participants through adolescence. Head Start REDI follow-up results have been reported across several papers, and researchers have found evidence that the intervention had positive long-run effects on measures of children’s social-emotional functioning (Nix et al., 2016; Welsh et al., 2020), with more evidence for fadeout on measures of literacy skills (Bierman et al., 2017).

Chicago School Readiness Project

The Chicago School Readiness Project (CSRP) intervention shares some features with other ECE social-emotional curricular interventions reviewed above, as it offered teachers professional development sessions targeting how they approached student social-emotional learning in their classrooms. However, the CSRP intervention was focused on reforming teachers’ approaches to student behavioral management, as previous research suggested that teachers in low-resourced Head Start centers were poorly prepared for the behavioral challenges of students growing up in homes taxed by the stress of poverty (Raver et al., 2008; 2009). The features of the intervention are described in more detail below, but the CSRP program also targeted the stress levels of teachers themselves, and it placed licensed mental health professionals in classrooms. These program features were offered in combination with the hope that overhauling the quality and predictability of the Head Start classroom would improve child self-regulation and lead to further downstream improvements in student school-readiness.

Indeed, initial evaluations reported that the intervention was largely successful at meeting its short-term goals, as the intervention led to improvements in classroom quality (Raver et al., 2008), higher levels of child executive function skills (based on both observer ratings and direct assessments; Raver et al., 2011), reductions in children’s behavior problems (Raver et al., 2009), and improvements in children’s reading and math performance (Raver et al., 2011). The study suffered from methodological concerns that somewhat clouded the findings, including issues with power and baseline differences between experimental groups, which we describe in more detail below. However, a replication effort in additional Head Start sites in Chicago and preschool centers in New Jersey found similar positive effects on children’s behavior and executive functioning, but no effects were detected on pre-academic skills (Morris et al., 2013).

Further analyses of the longer-run effects of the CSRP intervention have produced mixed results. Zhai and colleagues (2012) found that positive effects on academic achievement and behavior problems faded in kindergarten, though children who attended higher quality elementary schools still showed some positive intervention impacts on literacy, externalizing and internalizing behaviors (see also Li-Grining & Haas, 2010). In terms of social-emotional functioning, McCoy and colleagues (2018) found that a small number of children in the intervention group displayed fewer signs of transitory issues with attention and social functioning during middle childhood. However, issues with measurement and attrition hampered the elementary and middle school follow-up efforts, making a comprehensive study of medium-term impacts difficult.

High school follow-up.

The current study represents the culmination of a multi-year follow-up effort that began during the 2015–2016 academic year, approximately 10 to 11 years after the end of the initial preschool intervention.¹ This federally funded follow-up study was designed to provide a comprehensive picture of CSRP’s impacts on adolescent functioning through the end of high school. As part of the high school follow-up, adolescents were re-randomized to a brief mindset intervention (described below), which tested whether targeting students’ purpose for learning and growth mindset beliefs might affect self-regulation and achievement (see Yeager et al., 2019). The effects of this follow-up intervention effort were reported by Gandhi and colleagues (2020), and the intervention had few detectable impacts on proximal and distal measures of adolescent functioning. This follow-up intervention is not the focus of the current study, though we revisit it below when pursuing some checks for heterogeneity.

Several additional analyses of CSRP intervention impacts have been conducted from the high school follow-up, which serve as precursors to the effects reported here. Watts and colleagues (2018) examined preschool intervention impacts on data from the first adolescent follow-up wave in 2015–2016. Their analyses found some evidence for positive long-run intervention impacts on adolescents’ EF and academic achievement, though results for measures of self-reported behavior problems were null. A mediational analysis found that the positive impacts on adolescent EF and academic performance were partially explained by preschool impacts on pre-academic skills, presenting the possibility that earlier effects on cognitive functioning may have led to longer-term developmental changes (McCoy et al., 2019). Interestingly, exploratory evidence suggests that adolescents from the preschool intervention group may have selected into higher performing high schools (Watts et al., 2020). However, it remains unclear if these potential impacts will lead to better outcomes through the end of adolescence.

Present study.

The current study provides a broad examination of CSRP’s impacts on a range of developmental outcomes taken from the later waves of the high school follow-up study, which occurred over three years between the 2016–2017 academic year (i.e., 11 to 12 years after the intervention) and the 2018–2019 academic year (i.e., 13 to 14 years after the initial intervention). Although this study was not pre-registered, we used our follow-up grant application, when possible, to guide our approach to this set of analyses. In particular, we first used adolescent measures of behavioral problems, executive functioning, emotional regulation, and academic achievement outcomes to directly test if the intervention produced long-term impacts on the same constructs that were used as posttest outcomes in preschool (Raver et al., 2009; 2011). Confirmatory tests on these outcomes likely represent our best tests of long-term impact fadeout or re-emergence, though it should be noted that operationalizations have shifted substantially over time to reflect developmental differences in these constructs. For example, our measures of adolescent academic achievement include Scholastic Aptitude Test (SAT) scores and college enrollment, whereas the original evaluation included early childhood measures of letter naming, vocabulary, and basic math performance (Raver et al., 2011). We also selected measures of more exploratory outcomes, drawing upon longitudinal literature relating self-regulation to comprehensive measures of adult functioning (i.e., Moffitt et al., 2011) and recent evidence on the broad impacts of ECE exposure on constructs like civic participation (e.g., Kitchens & Gormley, 2022).

For our analyses, we present a range of modeling approaches with the goal of both understanding the robustness of findings to various model specifications and addressing design issues that can threaten internal validity. The original CSRP study suffered from some sources of baseline imbalance at the time of random assignment, which, coupled with the relatively small number of Head Start sites, has made results sensitive to statistical approaches for modeling the multi-level structure of the data (e.g., Raver et al., 2009; Watts et al., 2018). Indeed, like many other longitudinal studies of ECE programs, analytic approaches for detecting treatment impacts in the CSRP study have varied over the years, and previous reports show that findings have varied based on the inclusion of different control variables. For example, descriptive evidence presented in the original preschool evaluations suggested little evidence of positive impacts when simply comparing means between the treatment and control group, though multi-level models with controls for child and site characteristics produced larger positive effects (see Raver et al., 2009; 2011). Follow-up work has further highlighted these issues, with more recent papers presenting a range of modeling approaches to explicitly examine the sensitivity of estimates to model specification (see Watts et al., 2018). As we describe below, we present estimates with varied sets of controls, including models designed to align with the original evaluation reports (e.g., Raver et al., 2009; 2011), and we examine the consistency of effects across these approaches to assess the sensitivity of our findings. Further, given the longitudinal structure of the data, we examine issues relating to attrition and present findings generated from multiple imputation.

Finally, we extended these main effect estimates with several important tests for heterogeneity, which we considered to be exploratory. Although we place most of our focus on estimating average effects, we recognize recent work suggesting that researchers should pay attention to how impacts vary across persons and settings (e.g., Raudenbush & Bloom, 2015). Thus, we examined if CSRP preschool intervention effects differed for key subgroups of children, largely following the heterogeneity tests pursued in the original evaluation work (Raver et al., 2009). Here, we tested if impacts differed by gender, race, baseline executive functioning, or exposure to significant poverty. Because the study included two cohorts of students, we also tested if effects differed by cohort. Similarly, we drew on recent theory regarding early impact fadeout, which suggests that the long-term impacts of a given ECE intervention may depend on the features of the environments children experience after the intervention (see Bailey et al., 2017). Thus, we tested if intervention effects differed based on the quality of the high school where the student enrolled, and we tested if exposure to a mindset intervention in adolescence moderated preschool treatment impacts.

Method

Procedure and Sample

Evaluation design.

The CSRP study design has been described in detail in the original evaluation reports (see Raver et al., 2008; 2009; 2011). Here, we provide an overview of key study details, and Figure 1 depicts the flow of participants over time with more description given to the follow-up waves used in the current analysis.

Figure 1.

Longitudinal Study Design of the CSRP Evaluation

The CSRP intervention was tested through a cluster-randomized trial that paired Head Start centers (n = 18) into blocking groups (n = 9) based on site characteristics. Centers were recruited for study participation based on three criteria: 1) location in a high-poverty Chicago neighborhood; 2) receipt of Head Start funding; 3) serving two or more full-day classrooms. Sites participated in the study across two cohorts, with the first cohort participating during the 2004–2005 school year and the second during the 2005–2006 school year. Within each of the 18 sites recruited for participation, two classrooms were selected for study participation, though one classroom later dropped out of the study due to Head Start funding cuts. Researchers recruited approximately 83% of the children in the 35 study classrooms for participation in the original round of data collection, with 602 children participating at some point during the original Head Start year (59 children entered the study later in the school year; see Raver et al., 2009). The demographic characteristics of the sample reflect the low-income neighborhoods served by the original Head Start centers. Approximately 66% of the sample identified as Black and 27% as Hispanic. The average income-to-needs ratio was 0.68.

Data on children, families, teachers, and classrooms was originally collected during the fall (i.e., baseline) and spring (i.e., posttest) of the Head Start year for both cohorts, and the effects of the intervention on classroom quality and child outcomes were reported across several evaluation reports (see Raver et al., 2008; 2009; 2011).

Follow-up design.

During adolescence, participants and their parents were re-contacted and re-consented to follow-up waves of data collection, ² with most interviews in each follow-up wave occurring during the spring semester of a given school year.³ Two follow-up waves of data collection were executed during the 2015–2016 and 2016–2017 school years for both cohorts of children (see Figure 1). These two initial adolescent follow-ups were administered via computerized assessments in participant’s high schools, which allowed for the collection of direct assessments of adolescent self-regulation (described in detail below).

As part of the 2015–2016 follow-up assessment protocol, adolescents were re-randomized to a light-touch mindset intervention. For this intervention, adolescents were randomized within their original Head Start groupings (i.e., complete re-randomization across the original treatment and control groups). For the second wave of follow-up collected during the 2016–2017 academic year (i.e., one year later), students were given a mindset intervention module. For this second module, students remained in their treatment groupings from the first mindset intervention protocol. The first module was directed toward students’ purpose for learning, and the second module targeted students’ growth mindset beliefs (see Gandhi et al., 2020).

The effects of the “2-dose” mindset intervention on proximal and distal measures of achievement and self-regulation have been previously reported, with Gandhi and colleagues (2020) finding largely null results of the intervention on student outcomes. Some models also produced negative effects on academic achievement, though these effects were not consistently statistically significant. Because the follow-up intervention was randomly assigned across conditions, it has no correlation with the preschool treatment, which allows us to examine main effects of the preschool intervention for all students in the follow-up sample without bias due to the re-randomization. However, in our key models (unless otherwise noted), we controlled for the re-randomization indicator for dose 2, and we present effects of the mindset intervention on the outcomes considered here in the supplementary file (results further suggested largely null effects of this intervention).

The current study begins with data taken from the 2016–2017 follow-up wave (i.e., when the second dose of the mindset intervention was administered; student age approximately 16 years), and we also include data taken from two additional follow-up waves executed during the 2017–2018 (student age approximately 17 years) and 2018–2019 (student age approximately 18 years) school year. These additional follow-up waves did not involve any re-randomization and were conducted via survey assessments sent electronically to parents and adolescents (with some participants completing the assessments via phone interview). These follow-ups were designed to be brief, with questions primarily focusing on educational outcomes and major life events that occurred during the previous year. These last two follow-up waves were also used to collect consent from adolescents and parents for the release of administrative data on adolescents’ standardized test scores and college enrollment.

Across the three waves of follow-up data reported here, 430 children had non-missing data on the mindset random assignment indicator and at least one outcome measure considered in our follow-up analyses (71% of the original sample). We found virtually no difference in the overall follow-up participation rate between the preschool treatment and control groups (estimated difference in the probability of participation: 0.01, ns). In the supplementary file (Table S1), we present several key baseline characteristics for children in the original sample versus those included in the follow-up sample, and we found that the follow-up sample had few noteworthy differences from the original sample. Specifically, the follow-up sample came from families with more children in the household at the fall of preschool (p < 0.10), they had slightly higher family income (p < .10), their Head Start teachers reported higher levels depression (p < 0.01), and observers rated their Head Start classrooms as lower quality (p < 0.05).

Intervention

Intervention sites.

Previous reports have carefully detailed the CSRP preschool intervention components (e.g., Raver et al., 2009), and Li-Grining et al. (2014) and Zhai et al. (2010) both provide information regarding implementation fidelity. Intervention implementation was found to be generally adequate across sites. Here, we briefly summarize the key programmatic details.

First, teachers were provided with training designed to improve teacher behavioral management with the goal of supporting child self-regulation. The 30 hours of professional development (PD) were split across five sessions scheduled on Saturdays beginning in October and ending in January. The PD sessions were led by an experienced trainer who was also a licensed social worker, and the sessions involved an adapted version of the Incredible Years Teacher Training Program (Webster-Stratton et al., 2004). This program is designed to give teachers developmentally appropriate strategies to reduce children’s behavioral problems and promote positive self-regulation.

The training sessions were also attended by master’s-level social workers, who were employed to provide hands-on services to intervention classrooms. These “Mental Health Consultants” (MHCs) visited intervention classrooms throughout the school year, working directly with teachers to help them implement the behavioral management techniques introduced during the PD sessions. The MHCs were supervised by a licensed clinical social worker throughout the school year, receiving biweekly clinical supervision.

To help guard against teacher burnout, the MHCs also organized and led a stress-reduction workshop at each intervention site, and they regularly provided teachers with strategies to help them relieve stress during their school day visits. Finally, toward the end of the school year, the MHCs provided direct services to children who had been identified as having especially difficult behavioral and emotional issues. These direct intervention services were targeted to approximately 3 to 4 children per class.

The entire intervention program was overseen by a program coordinator, who organized the PD sessions and encouraged regular teacher participation in the trainings.

Control sites.

Classrooms in Head Start sites assigned to the control group were provided with an associates-level teacher’s assistant (TA). The TAs for control classrooms were added to balance the change in the student-to-adult ratio within the intervention classrooms due to the regular presence of the MHCs.

Measures

In the supplementary file, we provide more measurement details. Here, we provide an abbreviated overview of each measure. Correlations among all the outcome measures are shown in supplementary file Table S2.

Follow-up direct assessments of emotional regulation and executive function.

Several follow-up assessments were conducted via computer-based measures administered by trained data collectors in participants’ high schools during the 2016–2017 follow-up wave. The follow-up assessment battery included the Emotional Go/No-Go Task (Tottenham et al., 2011), a direct assessment of emotional regulation, emotion discrimination, and cognitive control. The task administered in the current study involved discriminating between “neutral” faces and faces that expressed either sad or angry emotions. Following previous CSRP work (Watts et al., 2018), we included a measure of “emotional discrimination” (i.e., accuracy adjusted for false alarms) in trials when students viewed angry and sad faces, with higher “D-Prime” scores indicating better emotion discrimination. We also examined measures of reaction time, which were calculated as the difference in reaction time during sad or angry trials, respectively, and happy (i.e., baseline) trials (calculated in milliseconds).

Executive function (EF) was directly assessed using the Hearts and Flowers task, which taps working memory and inhibitory control (Diamond et al., 2007). This commonly used measure of EF asked students to press keys corresponding to visual stimuli that appeared on the screen. As with the emotional regulation measure, we again used the same variables that were employed in previous CSRP evaluation work (Watts et al., 2018). We first measured accuracy during mixed trials, in which adolescents were asked to respond to both congruent and incongruent stimuli. We also measured response time by taking the difference between reaction time in the “mixed” trials and reaction time in the “hearts only” (i.e., baseline) trials (calculated in milliseconds).

Follow-up survey measures of behavioral adjustment.

The follow-up assessment battery administered during the 2016–2017 wave included an array of self-reported measures of behavioral functioning. For these measures, we used a series of factor analyses to generate conceptually coherent subscales that we used in the current analysis. These factor analyses were fit for the full sample to maximize power and to be more representative of our full sample. However, we pursued several tests of measurement invariance to ensure that the factor structure was equivalent between the treatment and control groups. We observed little evidence suggesting the presence of factor non-invariance across all tests conducted, though it should be noted these tests may have been somewhat hampered by the low power of the current study (results available upon request from the first author). Further details regarding factor analyses are provided in the supplementary appendix.

First, following McCoy and colleagues (2011), we used responses to the Barratt Impulsiveness Scale (Patton et al., 1995) and the Behavior Rating Inventory of Executive Function (Gioia et al., 2000) to measure both cognitive and behavioral dysregulation, respectively (we used an adapted version of the BRIEF for self-report). The items for the Cognitive Dysregulation scale (α = 0.83) involved difficulty with attention and impulsivity, and items for the Behavioral Dysregulation scale (α = 0.85) tapped issues with restlessness and social misbehavior. Higher scores on both scales indicated higher levels of dysregulation.

We also measured internalizing behavioral problems and substance abuse using the “risk” items of the “Risks and Strengths” scale, a comprehensive behavioral assessment adapted from the Child Health Risk Behavior Scale (CHRBS; Riesch et al., 2006) and the Middle School Youth Risks and Behavior Survey (MS-YRBS; CDC, 2015). The Internalizing composite (α = 0.73) included items that captured depression and worrying (e.g., felt unsafe, felt unhappy, sad, or depressed), and the Substance Abuse composite (α = 0.70) tapped alcohol and drug use (e.g., smoked a cigarette, drank beer, wine, or liquor without adult permission). Higher scores on these measures indicated worse behavioral functioning.

Finally, we relied on survey items that captured prosocial behaviors, as well as items adapted from the Critical Consciousness Scale (Diemer et al., 2014), to measure several dimensions of positive behavioral functioning. Using the “strength” items from this adapted scale, we also examined critical actions, socio-political and civic actions, and extra-curricular activity participation as indicators of pro-social beliefs and behaviors. Our measure of Critical Actions (α = 0.83) included items focused on students’ motivation to affect social change (e.g., “it is important to me to contribute to my community”). Higher scores on this measure indicated increased motivation. Socio-Political and Civic Actions (α = 0.50) asked students whether they had recently participated in socially active behaviors (e.g., “contacted a public official to tell him/her how you felt about a particular social or political issue”). Finally, the Extracurricular Activity Participation scale (α = 0.62) asked students about their involvement in organized activities outside of school (e.g., “joined a team or played an intramural sport”). For these two latter measures, higher scores indicated more actions were taken by the student.

Academic achievement and attainment.

Throughout each of the adolescent follow-up waves, we consented students and parents for the release of administrative records regarding educational outcomes. Using these consents, we then pursued data sharing partnerships with Chicago Public Schools (CPS), American College Test (ACT), College Board, and National Student Clearinghouse (NSC). We then used these third-party sources to generate several measures of end-of-high-school academic achievement.

We first generated a composite SAT score, which was the total of students’ quantitative and verbal scores as provided by either CPS or College Board. For students missing SAT scores, we turned to Preliminary SAT (PSAT) and ACT scores. We used published conversion tables to link these alternative tests to the SAT. We verified this approach by using multiple imputation to generate an imputed SAT score from an imputation model that included all high school achievement scores available for each student (this also included student-reported SAT scores from survey measures). These two approaches led to two highly correlated measures of SAT performance (r (360) = 0.96)), and we ultimately generated a composite that averaged scores from the imputation and conversion table approaches. In results available upon request, we estimated models for scores generated solely from the imputation and conversion table approaches to check consistency of estimates. The point estimates were similar to those shown in our key tables.

We relied on NSC records to measure college enrollment. We received consent for educational records from 411 students, and NSC provided enrollment records through the Spring of 2021⁴ through the StudentTracker service. Thus, students who were 4 when they enrolled in Head Start in the first cohort of the study (2004–2005) would have been approximately 21 years old for this data pull, and 4-year-old students in cohort 2 (2005–2006) would have been approximately 20 years old. We first generated a broad measure of college enrollment, which was simply coded as “1” if a student had any college enrollment data from NSC, and “0” if a student had no enrollment records. Assigning students who did not return records from NSC as “not enrolled” is a typical assumption when working with this data because NSC records cover almost all post-secondary enrollment in the U.S. (96.9% for all institutions within the United States; 98.2% for institutions in Illinois; National Student Clearinghouse, 2021).

We then generated a second measure of “post-secondary enrollment in a 4-year school,” which involved more extensive cleaning of the data (see supplementary file). For this measure, students were only coded as “1” if they ever enrolled in a 4-year college (i.e., eliminating only those who enrolled in a community college), and if their enrollment could be verified as occurring after high school (i.e., eliminating those who only ever had dual-enrollment during high school).

Finally, we relied on survey data taken from the 2017 – 2018 and 2018 – 2019 follow-up waves, which collected information on students’ high school academic achievements (see Li-Grining et al., 2021). Among the surveys, we identified 4 items that were consistent across both waves and indicated college preparedness: 1) whether the adolescent took the SAT, 2) whether the adolescent took the ACT, 3) whether the adolescent applied to either a 4-year college or a community college, and 4) whether the adolescent was accepted to either a 4-year college or community college. We summed across these 4 indicators at both waves to generate an index of College Planning. This index was moderately positively correlated with our binary measure of any college enrollment (r(360) = 0.37).

Baseline Covariates.

As with other previous analyses of the CSRP (e.g., Watts et al., 2018), we include a host of baseline covariates in several of our key treatment impact models. We briefly describe these measures below. More details for these measures can be found in previous reports (Raver et al., 2009; 2011), and the full list of baseline covariates can be found in Table S3 of the online supplementary information file.

Information regarding family background was collected in the fall of the preschool year. Parent-reported background characteristics were used as covariates in the present analyses, which included child gender, age during preschool, race/ethnicity, and parent socioeconomic status, as well as a host of parent-reported sociodemographic variables. Importantly, we calculated family income-to-needs ratio at baseline based on reported total family income from the previous year divided by that same year’s federal poverty threshold for the number of adults and children in the family.

Teacher- and parent- reported child skills and behavior were measured during baseline of preschool, which included EF, effortful control (EC), attention and impulsivity control, positive emotion, math, vocabulary, letter-naming, and externalizing and internalizing behavior. EF, EC, and attention and impulsivity control were assessed using the Preschool Self-Regulation Assessment (PSRA; Smith-Donald et al., 2007). EF was an aggregated measure consisting of children’s performance on a Balance Beam task (Murray & Kochanska, 2002) and Pencil Tap task, which was adapted from the peg-tapping task (Blair, 2002; Diamond & Taylor, 1996). EC was an aggregated measure consisting of three delay tasks: Toy Wait, Snack Delay, and Tongue Task (Murray & Kochanska, 2002). After the tasks were administered, a 28-item PSRA Assessor Report was completed to provide a global picture of children’s emotions and attention during the tasks. 18 items across the PSRA assessor report were aggregated to generate a measure of attention and impulsivity control while 9 items were aggregated to generate a measure of positive emotion. For all four measures, higher scores indicated higher self-regulation.

For preschool academic skills, the current study used early skills scores of the National Reporting System (NRS) assessment. Children’s math skills were assessed using the Early Math Skills test, which included basic concepts such as addition and counting (Zill, 2003a). To assess children’s vocabulary skills, the 24-item Peabody Picture Vocabulary Test (PPVT; Dunn, & Dunn, 1997) was used for English speakers and the Test de Vocabulario en Imagenes Peabody (TVIP; Dunn et al., 1986) was used for Spanish or bilingual speakers. Both tasks asked children to point to a picture out of a group of four that corresponded with the vocabulary read aloud. To assess children’s letter recognition, a 30-item letter-naming task was given where the children were asked to name the displayed letters one at a time (Zill, 2003b). In this study, behavior problems were reported by both parents and teachers using the Behavioral Problem Index (BPI; Zill, 1990), with 18 items used to represent externalizing problems (e.g., lying, argues, bullies) and 10 items used to represent internalizing problems (e.g., clings to adults, cries too much, demands attention).

Baseline teacher and class characteristics included teacher age, education level, depression, job demand, job control, behavior management, classroom emotional climate, classroom overall quality, class size, and number of adults in class. Depression is a teacher-reported measure consisting of a six-question K6 scale of psychological distress (Kessler et al., 2002). Using the Child Care Worker Job Stress Inventory (CCW-JSI; Curbow et al., 2000), teachers also completed the 6-item Job Demand and 9-item Behavior Management scales. Classroom emotional climate and overall quality was collected using the Classroom Assessment Scoring System (CLASS; La Paro et al., 2004) and the Early Childhood Environment Rating Scale–Revised (ECERS-R; Cryer et al., 2003).

Analytic Approach

Previous studies reporting CSRP impacts have employed a range of modeling techniques, including approaches that have relied on hierarchical linear models (HLM) with random effects (e.g., Raver et al., 2009; 2011) and approaches that have relied on econometric fixed effects with clustered standard errors (e.g., Watts et al., 2018). In the current study, our primary estimates were generated from a model that adopted current best practices for analysis of multi-site RCTs by using a random effect model (i.e., HLM) that included econometric fixed effects for the clustering unit of random assignment (i.e., the blocking group pairs; see Murnane & Willett, 2011, p. 131). Essentially, this model relied on random effects to account for the multi-level structure of the data (students nested in Head Start sites) while explicitly controlling for differences between site-pair blocking groups:

Level 1: Students

$${\text{Outcome}}_{\text{ij}}={\text{β}}_{0\text{j}}+\text{χ}{\text{Child}}_{\text{ij}}+{\text{ΩClass}}_{\text{ij}}+e_{\text{ij}}$$

Level 2: Head Start Sites

$${\text{β}}_{0\text{j}}={\text{γ}}_0+{\text{β}}_1\text{T}{\text{x}}_{\text{j}}+\sum _{\text{b}=1}^9{\text{λ}}_{\text{b}}{\text{Block}}_{\text{j}}+{\text{u}}_{\text{j}}$$

where ${\text{Outcome}}_{ij}$ represents a given outcome measure for adolescent $i$ originally recruited from Head Start center $j$. We included a host of child and classroom controls (noted by $\chi Chil{d}_{ij}+\Omega Clas{s}_{ij}$) to account for possible sources of baseline differences between groups at the time of random assignment (this set of child characteristics also includes random assignment to the adolescent mindset intervention). We also included random effects (i.e., random intercepts) for Head Start site, and a set of dummy variable controls for the blocking group matched pairs (${\sum }_{b=1}^9{\lambda }_{b}Bloc{k}_j$). It should be noted that sites were blocked according to cohort, so this set of blocking group fixed effects also controls for cohort status. Importantly, ${\beta }_1$captures the effect of assignment to the CSRP intervention group on each outcome of interest, controlling for the blocking groups and the set of child and classroom baseline characteristics. Maximum likelihood was used as the primary estimation method, though we also assessed sensitivity to REML in the supplementary file.

Our approach is similar to the approach employed by Watts and colleagues (2018), which also used blocking group fixed effects and a large set of baseline covariates to adjust for sources of imbalance. However, we begin by reporting models that include no baseline controls (with only the blocking group fixed effects and mindset assignment indicator included) to present basic cluster-adjusted mean differences between the pre-k treatment and control groups. By comparing these estimates to those generated from the “fully controlled model,” we can assess the likely impact of baseline differences on our treatment impact estimates. Further, we also present results from a host of alternative modeling approaches, including an approach that closely aligns with the multi-level models used by Raver and colleagues (2009, 2011) in the original preschool treatment impact papers. As we describe below, these models relied on controlling for Head Start site characteristics rather than the blocking group pairs used in our preferred estimates.

We also explored several tests for heterogeneity based on key child characteristics. First, we tested for interactions with a set of baseline child characteristics that included gender (coded “1” for female), race (coded “1” for Black students), significant exposure to poverty (coded “1” for children with family income-to-needs ratio below 0.50), and fall-of-preschool EF (continuous variable). To limit the number of statistical tests, these interactions were entered jointly into the model, and we pursued a post hoc joint test of statistical significance to test if the entire set of baseline interactions significantly contributed to the model. We then fitted separate models to test if the impact varied based on cohort (by including a cohort by treatment interaction), later high school quality (see supplemental file for more details), or assignment to the adolescent mindset intervention.

In all of our models, continuous outcomes were standardized across the full sample, so coefficients can be interpreted as effect sizes (though we present effects on raw scores in the supplemental file). For binary outcomes (i.e., college enrollment), we used linear probability models (consistency in findings with logistic regression models was also tested). Thus, treatment impacts can be interpreted as predicted changes in probability in response to treatment assignment. Further, to account for missing data on baseline covariates, we used multiple imputation to generate 25 imputed datasets in Stata 16.0 (using the “Markov chain Monte Carlo” method). Models were then fitted using the “mi est” commands in Stata to combine estimates across the imputed datasets. Below, we also detail results from additional models that included imputation on the outcome variables to adjust for possible effects of attrition on our estimates.

Results

Baseline Equivalence

As previous evaluations have reported (e.g., Watts et al., 2018), we observed indications of baseline imbalance across several measures taken in the fall of preschool. Table 1 presents a representative set of baseline measures, and Table S3 in the supplementary file presents the full set of measures included in our models. Estimates shown in the “difference” column were generated using mixed models with blocking group fixed effects and random effects included for sites. In general, we observed that children in the CSRP intervention group tended to have higher scores on fall baseline measures of cognitive functioning and worse scores on measures of behavioral adjustment. Further, intervention classrooms were rated as more disadvantaged at the onset of the intervention, with some of these differences producing large effect sizes. We used an OLS regression model to test the joint statistical significance of the entire set of baseline controls, and this F-test was statistically significant (p < .001), underscoring the need to use control measures when estimating treatment impacts.

Table 1.

Selected Baseline Characteristics

	CSRP Treatment	CSRP Control	Estimated Differenc^
Child Demographic Characteristics
Female	0.49	0.61	−0.12*
Age (years) at preschool	4.37	4.36	−0.01
Black	0.70	0.66	−0.00
Hispanic	0.26	0.25	0.05
Bi-racial or Other	0.03	0.05	−0.02
Family/Parent Characteristics
Income to Needs Ratio	0.68	0.72	−0.04
Number of Children in the Home	2.67	2.82	−0.14
Food Stamps	0.52	0.52	−0.02
Free/Reduced Price Lunch	0.56	0.62	−0.08
Bio Parent Sees Child Everyday	0.42	0.50	−0.10+
Married/Remarried	0.19	0.24	−0.06
Parent Has Savings	0.65	0.54	0.12*
Child Baseline Skills and Behavior
Executive Functioning	0.05	−0.16	0.21*
Math	7.37	6.90	0.11
PPVT	10.46	9.88	0.15
Externalizing (Parent Report)	6.69	6.06	0.12
Internalizing (Parent Report)	3.18	3.17	−0.02
Teacher and Class Characteristics
Teacher age	37.20	42.62	−0.40+
Techer Depression (K6 Score)	3.20	2.04	0.71*
Teacher Job Demand	2.88	2.53	0.60***
Behavioral Management	4.56	5.11	−0.63***
Classroom Overall Quality	4.42	4.91	−0.61**
N	224	206	430
F (53, 349.2) = 18.33, p < 0.001

Note.

^{^}The values shown in the “Estimated Difference” column were derived from HLM results such that each respective baseline variable was regressed on treatment status and a set of blocking group fixed effects, with school random effects included. Continuous baseline variables were standardized, so coefficients for continuous variables in the “Estimated Difference” column can be interpreted as effect sizes. Binary variables were not standardized. So, differences for binary variables can be interpreted as the difference in probability (i.e., linear probability models). The F-statistic was generated by regressing treatment status on all baseline measures (with blocking group controlled), and testing whether all baseline measures were jointly statistically significantly different from 0 (using the “mi test” command in Stata). The full set of baseline characteristics used in our key models can be found in supplementary file Table S3. Descriptive statistics were generated using imputed data for missing responses. The table only includes students included in the analysis sample (n = 430), though comparisons to the full sample of 602 can be found in the supplementary file.

⁺p < 0.10

^*p < 0.05

^**p < 0.01

^***p < 0.001

Descriptive Findings

Table 2 presents descriptive characteristics (not adjusted for clustering) for our key outcomes of interest for the intervention and control groups. As Table 2 reflects, we saw few mean differences between the groups on the various outcome measures.

Table 2.

Descriptive Statistics for Outcome Measures

	CSRP TreatmentSD					CSRP Control
	N	Mean	SD	Min	Max	N	Mean	SD	Min	Max
Emotional Regulation: Emotional Go/No Go
Emotion Regulation- Angry D-Prime	207	1.57	0.84	−0.90	3.26	197	1.59	0.79	−1.83	3.57
Emotion Regulation- Sad D-Prime	206	1.35	0.87	−1.53	3.20	197	1.46	0.83	−0.71	3.20
Emotion Regulation- Angry RT	207	34.60	48.19	−109.47	181.38	197	35.56	57.47	−175.27	314.53
Emotion Regulation- Sad RT	207	43.54	45.30	−100.10	165.12	197	41.79	54.95	−331.80	233.74
Executive Function: Hearts & Flowers
Executive Function- Mixed Block Accuracy	206	0.73	0.18	0.18	1.00	190	0.72	0.19	0.24	1.00
Executive Function- Mixed Trials RT	206	180.16	76.87	−47.60	377.08	190	188.81	81.24	−204.25	357.52
Behavioral Functioning: Risks & Strengths
Internalizing	224	0.37	0.31	0.00	1.00	206	0.36	0.29	0.00	1.00
Substance Abuse	224	0.10	0.18	0.00	1.00	206	0.10	0.18	0.00	0.83
Critical Actions	217	3.91	0.82	1.00	5.00	202	3.99	0.79	1.00	5.00
Socio-political and Civic Actions	218	0.25	0.28	0.00	1.00	202	0.29	0.29	0.00	1.00
Extracurricular Activity Participation	224	0.43	0.30	0.00	1.00	206	0.37	0.29	0.00	1.00
Self-Regulation: BIS & BRIEF
Cognitive Dysregulation	224	0.37	0.23	0.00	1.00	206	0.37	0.19	0.00	0.96
Behavior Dysregulation	223	0.26	0.17	0.00	0.97	206	0.27	0.19	0.00	0.89
Academic Achievement and Attainment
Any College Enrollment (proportion)	193	0.52		0.00	1.00	181	0.56		0.00	1.00
Four Year College Enrollment (proportion)	193	0.28		0.00	1.00	181	0.30		0.00	1.00
Total SAT	182	896.29	151.89	580.00	1560.00	172	913.80	144.61	630.00	1300.00
College Planning (survey index)	195	2.31	1.29	0.00	4.00	190	2.46	1.35	0.00	4.00

Note. RT= reaction time. These estimates represent “raw” means at the student level (i.e., no adjustment for clustering). Cluster adjusted differences between treatment and control are presented in tables 3 through 5.

Main Effects

Key results.

Tables 3, 4 and 5 present main effect estimates across our key outcome measures. In Column 1 of each table, we begin with a basic model that includes only the blocking group fixed effects and the indicator for random assignment to the mindset intervention as controls. In Column 2, we include the full set of baseline child and teacher characteristics, which were generated using multiple imputation for missing data on baseline characteristics. We present both sets of estimates, because the randomized design should allow us to infer treatment impacts from simply observing mean differences after adjusting for the block randomized design (i.e., Column 1). However, due to the likelihood of baseline differences between the experimental groups, we also present covariate-adjusted models attempting to account for sources of baseline imbalance (i.e., Column 2). For the models with no controls, we observed no statistically significant treatment effects on any outcome. Across the models with full controls, we detected only one intervention effect that was statistically significant at the .05 level out of the 17 tested. Below, we note patterns in the magnitude of the observed effects and note any p-values below .10.

Table 3.

Estimated Impacts of the CSRP Preschool Intervention on Adolescent Emotional Regulation and Executive Functioning

	Block FE w/ No Covariates	Block FE w/ Full Covariates	N	Imputation on Outcomes	N
	(1)	(2)		(3)
Emotional Regulation: Emotional Go/No Go
Emotion Regulation- Angry D-Prime	−0.01	0.02	404	0.02	602
	(0.10)	(0.14)		(0.13)
Emotion Regulation- Sad D-Prime	−0.13	−0.16	403	−0.13	602
	(0.10)	(0.13)		(0.14)
Emotion Regulation- Angry RT	0.01	−0.03	404	−0.05	602
	(0.10)	(0.13)		(0.12)
Emotion Regulation- Sad RT	0.05	0.14	404	0.07	602
	(0.10)	(0.13)		(0.12)
Executive Function: Hearts & Flowers
Executive Function-Mixed Block Accuracy	0.04	−0.15	396	−0.13	602
	(0.10)	(0.13)		(0.12)
Executive Function- Mixed Trials RT	−0.13	−0.14	396	−0.11	602
	(0.10)	(0.14)		(0.12)
Controls
Blocking Group	Inc.	Inc.		Inc.
Demographic, Family and Parent Characteristics		Inc.		Inc.
Child Baseline Cognitive and Behavioral Functioning		Inc.		Inc.
Classroom & Teacher Characteristics		Inc.		Inc.

Note. All outcome variables were standardized across the full sample, so coefficients can be interpreted as effect sizes. All models included fixed effects for blocking group and random effects for Head Start site. Models with covariates included multiple imputation to adjust for missing data, and models shown in Column 3 included multiple imputation on the outcome to adjust for attrition. Models presented in Columns 1 and 2 included the mindset intervention random assignment indicator as a control.

RT = reaction time.

⁺p < 0.10

^*p < 0.05

^**p < 0.01

^***p < 0.001

Table 4.

Estimated Impacts of the CSRP Preschool Intervention on Adolescent Behavioral Outcomes

	Block FE w/ No Covariates	Block FE w/ Full Covariates	N	Imputation on Outcomes	N
	(1)	(2)		(3)
Behavioral Functioning: Risks & Strengths
Internalizing	−0.00	0.12	430	0.09	602
	(0.10)	(0.13)		(0.12)
Substance Abuse	0.02	−0.13	430	−0.09	602
	(0.10)	(0.13)		(0.12)
Critical Actions	−0.09	−0.13	419	−0.10	602
	(0.10)	(0.13)		(0.12)
Socio-political and Civic Actions	−0.16	−0.19	420	−0.15	602
	(0.10)	(0.13)		(0.13)
Extracurricular Activity Participation	0.14	0.10	430	0.05	602
	(0.10)	(0.13)		(0.12)
Self-Regulation: BIS & BRIEF
Cognitive Dysregulation	0.00	−0.16	430	−0.12	602
	(0.10)	(0.13)		(0.12)
Behavior Dysregulation	−0.05	−0.24+	429	−0.17	602
	(0.10)	(0.13)		(0.13)
Controls
Blocking Group	Inc.	Inc.		Inc.
Demographic, Family and Parent Characteristics		Inc.		Inc.
Child Baseline Cognitive and Behavioral Functioning		Inc.		Inc.
Classroom & Teacher Characteristics		Inc.		Inc.

Note. All outcome variables were standardized across the full sample, so coefficients can be interpreted as effect sizes. All models included fixed effects for blocking group and random effects for Head Start site. Models with covariates included multiple imputation to adjust for missing data, and models shown in Column 3 included multiple imputation on the outcome to adjust for attrition. Models presented in Columns 1 and 2 included the mindset intervention random assignment indicator as a control.

⁺p < 0.10

^*p < 0.05

^**p < 0.01

^***p < 0.001

Table 5.

Estimated Impacts of the CSRP Preschool Intervention on Adolescent Academic Outcomes

	Block FE w/ No Covariates	Block FE w/ Full Covariates	N	Imputation on Outcomes	N
	(1)	(2)		(3)
Any College Enrollment (binary)	−0.06	−0.11	374	−0.06	602
	(0.0 5)	(0.07)		(0.07)
Four Year College Enrollment (binary)	−0.02	−0.05	374	−0.01	602
	(0.05)	(0.07)		(0.06)
Total SAT	−0.12	−0.20	354	−0.14	602
	(0.11)	(0.14)		(0.11)
College Planning (survey index)	−0.11	−0.25*	385	−0.11	602
	(0.10)	(0.12)		(0.11)
Controls
Blocking Group	Inc.	Inc.		Inc.
Demographic, Family and Parent Characteristics		Inc.		Inc.
Child Baseline Cognitive and Behavioral Functioning		Inc.		Inc.
Classroom & Teacher Characteristics		Inc.		Inc.

Note. SAT scores and the College Planning survey were both standardized across the full sample, so coefficients can be interpreted as effect sizes. The measures of college enrollment were both binary, so coefficients should be interpreted as the predicted change in probability. All models included fixed effects for blocking group and random effects for Head Start site. Models with covariates included multiple imputation to adjust for missing data, and models shown in Column 3 included multiple imputation on the outcome to adjust for attrition. Models presented in Columns 1 and 2 included the mindset intervention random assignment indicator as a control.

⁺p < 0.10

^*p < 0.05

^**p < 0.01

^***p < 0.001

Beginning with Table 3, which presents estimates for the direct assessments of emotional regulation and executive functioning (taken during the 2016–2017 follow-up), we observed mainly negative intervention effects that were not statistically significant across the various models. A comparison of Columns 1 and 2 suggests that baseline differences might have biased observed impacts toward zero, as estimates with controls tended to produce larger treatment and control differences. However, no estimates for the direct assessment outcomes were statistically significant.

Table 4 presents intervention impacts on surveys of behavioral regulation from our 2016–2017 follow-up, and we again did not observe any statistically significant effects at the .05 level. When assessing the magnitude of the non-significant effects, some scores tended to favor the intervention group whereas others favored the control group. When full controls were included, we did observe lower levels of behavioral dysregulation for adolescents in the intervention group at a marginal level of statistical significance ($\beta =-0.24$; SE = 0.13, p < .10).

Table 5 presents estimates of the effects of the intervention on long-term indicators of academic achievement and college enrollment. Again, we did not detect any statistically significant effects at the .05 level for the academic outcomes taken from the administrative records (i.e., college enrollment and SAT scores), though impacts tended to suggest worse academic outcomes for the intervention group. We also observed that adolescents in the intervention group had lower scores on our survey index of College Planning, and this effect was the lone statistically significant result from our key models ($\beta =-0.25$; SE = 0.12; p < .05).

Adjustments for attrition.

Although we observed almost no difference between the treatment and control group on the likelihood of inclusion in our overall follow-up sample, some individual measures had more missing follow-up data than others (e.g., our SAT measure was non-missing for only 355 students in the follow-up sample). Thus, in Table S4 of the supplementary file, we present differences in the likelihood of having data on each individual follow-up measure included in Tables 3, 4, and 5, and we again observed small and nonstatistically significant differences between the preschool treatment and control groups in the likelihood of having each follow-up measure.

In the third column of each of the key results tables, we present results from models that used multiple imputation (i.e., again using the “Markov chain Monte Carlo” method) on the outcome measures to generate estimates for the original full sample of 602 CSRP participants (these models do not control for mindset treatment in order to recover the original sample), and results were largely consistent with our preferred estimates that dropped cases with missing outcome data. For these models, SEs were slightly smaller due to the increased sample size, but we still did not detect any significant effects at the .05 level. Notably, the effect for the College Planning survey measure dropped substantially ($\beta =-0.11$; SE = 0.11) and was no longer statistically significant.

Alternative specifications.

In Table S5, we present alternative modeling specifications that assessed sensitivity in our main effect estimates to our key modeling choices. Of note, we tested models that used a reduced list of control variables, and we also tested models that used mean imputation for missing control variables, and results were largely consistent with those shown in our main models. We also tested models that used OLS regression, with SE’s adjusted for clustering using an adjustment recommended for smaller sample sizes (Tyslzer, Pustejovsky, & Tipton, 2017). For these OLS models, we again observed almost uniformly null results, though the negative effects on college enrollment and SAT scores were statistically significant at the .05 level.

Finally, we examined how our results might have differed if we had used a modeling approach similar to the approach taken by Raver and colleagues (2009, 2011) in the original preschool evaluations. The modeling specifications in those papers differ slightly from one another, but both papers used a 3-level HLM with site characteristics included as controls instead of the blocking group fixed effects used in the current models. These papers also used a reduced set of family demographic controls, and for measures of child cognitive skills and behaviors, they only controlled for a lagged indicator of the dependent variable. In the supplementary file, we provide more details regarding the exact specifications used for these models. As was reflected in the original evaluations, these models tended to produce larger coefficients than those shown in our key estimates, though they also produced noticeably larger SEs across most of the outcomes tested. For these models, we detected statistically significant effects at the .05 level for 3 of the 17 outcomes tested, with adolescents in the intervention faring worse on the College Planning index ($\beta =-0.51$; SE = 0.18; p < .01) and the Socio-Political and Civic Action scale ($\beta =-0.46$; SE = 0.18; p < .05), but better when compared with the control group on the Behavioral Dysregulation measure ($\beta =-0.36$; SE = 0.18; p < .05). Finally, Table S5 also presents non-standardized results with each treatment impact estimated in the raw units of each outcome.

Heterogeneity

We pursued several heterogeneity tests to examine if CSRP’s impacts on adolescent outcomes differed for key subgroups in our sample. These tests were largely exploratory and involved adding an interaction term between a given child variable and pre-k status. As we describe in the supplement, we used a pared-down list of control variables for these models.

Child characteristics.

Following the approach of previous CSRP intervention evaluations (e.g., Raver et al., 2011; Watts et al., 2018), we tested whether effects differed by the following subgroups: 1) females vs. males; 2) Black vs. Hispanic, White, and “Other race/ethnicity” adolescents; and 3) adolescents who had significant exposure to poverty during early childhood (defined as having income-to-needs below 0.50 at baseline) vs. other children. We also tested for heterogeneity by baseline executive functioning (modeled continuously using a standardized measure of executive functioning for the fall of preschool; see Raver et al., 2011). The results from these models are displayed in Table S6 in the supplementary file, and we found noisy and mostly null effects across the subgroup models. We used a joint F-test to assess whether the set of baseline interactions significantly added to the model, and we observed no statistically significant F-statistics at the .05 level out of 17 tested. Moreover, we observed no consistent patterns of heterogeneity for any of the baseline characteristics across the set of outcomes.

We also tested for interactions based on cohort status (see Table S7). Here, we found 1 statistically significant (p < .05) interaction out of the 17 tested: treatment students in Cohort 1 had higher reaction times to Sad trials on the EGNG task (p < .01).

Later school performance.

We next examined whether CSRP intervention impacts differed based on the quality of adolescents’ high schools. For these tests, we used a composite measure of school performance, relying on school enrollment for the 2016–2017 survey. The measure of school performance was derived from Illinois School Report Card data, and it was an aggregated measure of several academic school-level outcomes made publicly available by the state (e.g., test score performance, graduation rate, etc.). The measure is described in detail in the supplementary file.

Watts and colleagues (2020) found that students in the CSRP intervention attended high schools that were rated higher on this school performance measure (we also observed better ratings of school quality on this measure for the intervention group using the current sample and modeling approach (($\beta =-0.31$, SE = 0.19, p = 0.10)). This effect suggested that participants in the CSRP intervention group may have selected into higher quality high schools relative to their peers in the control group, underscoring that interactions should be interpreted cautiously given the connection between the moderator and the treatment variable. As Tables S8 reflects, although we found that the school performance indicator had positive main effects on several outcomes, we saw few statistically significant interactions between high school performance and CSRP treatment condition (2 out of 17 interactions tested were significant at the .05 level). However, these significant interactions indicated worse outcomes for the treatment group in higher performing schools (for Internalizing and the measure of Cognitive Dysregulation). Perhaps most relevantly, we saw no indication that intervention effects were moderated by school performance on any of the academic outcomes assessed, including college enrollment and SAT scores.

Mindset intervention.

We also tested if CSRP intervention effects differed based on random assignment to the adolescent mindset intervention. As we noted above, the two-dose mindset intervention was found to have largely null impacts on outcomes related to self-regulation and achievement (Gandhi et al., 2020). In Table S9, we present models that estimated the main effect of the mindset intervention, with estimates largely suggesting null results⁵. In Table S10, we present the interaction between the mindset intervention and the preschool intervention. With these models, we detected no statistically significant main effects at the .05 level for either intervention, nor did we detect any statistically significant interactions at the .05 level.

Discussion

Although research and theory suggest that participation in high-quality ECE programs may yield long-lasting effects on children’s developmental trajectories, we lack long-term follow-up evidence from RCTs of modern ECE interventions, especially those focused on curricular enhancements. The current study attempted to address this gap by examining the long-term impacts of the CSRP, a self-regulation focused early childhood intervention implemented in Head Start centers serving low-income neighborhoods in Chicago. We examined impacts of the CSRP intervention on adolescent outcomes measured through the end of high school, with key measures of executive functioning, emotional regulation, behavioral functioning, and academic achievement constituting our confirmatory tests of the constructs originally targeted by the program. We also examined more distal measures of adolescent functioning, such as extracurricular activity participation and socio-political and civic actions.

Our descriptive results suggested that approximately half of the students who originally enrolled in the Head Start centers had taken at least one college class nearly two decades later. For comparison, the 2019 college enrollment rate for Black and Hispanic students nationally was 37% and 36%, respectively (National Center for Educational Statistics & Institute of Education Sciences, 2021). The 2018 college enrollment rate for Black and Hispanic students within CPS was 49.5% and 64.5%, respectively (Nagaoka et al., 2020). Students also reported engaging in socio-political and civic actions, and extracurricular activities, and rates of misbehavior were relatively low. However, we found little evidence for sustained preschool intervention impacts. Despite earlier work suggesting that the CSRP intervention may have had some positive effects on adolescent cognitive functioning and academic achievement measured near the beginning of high school (Watts et al., 2018), we failed to find any substantial effects on later adolescent outcomes. Across our key treatment impact models, we found impacts that scattered around zero, with some null impacts favoring the intervention group (e.g., behavioral dysregulation) and others favoring the control group (e.g., executive functioning). These results suggest that like many other ECE programs – especially those targeting curricular enhancements – the CSRP intervention alone cannot be counted on to protect children from low-income families from subsequent life and school experiences in the intervening years. That is, CSRP did not have any strong impacts on behavioral and cognitive functioning that were detectable at the end of high school.

Perhaps more troubling, we tended to find scattered impacts that moderately favored the control group on the set of academic outcomes. Our survey-based measure of College Planning produced the lone statistically significant (p < .05) effect across our preferred models, with control adolescents scoring approximately 25% of a SD higher on this measure of college preparation (i.e., registering for tests, applying for college). However, this effect did not hold in our model that adjusted for attrition. However, one significant finding out of 17 outcomes should be viewed with some skepticism. Indeed, we calculated a Bonferroni-corrected critical p-value of 0.0029 by simply dividing an alpha of 0.05 by 17 (i.e., the number of outcome measures considered in the current analysis; this p-value would be even smaller if we further adjusted for the number of sensitivity tests across each outcome). Indeed, we did not observe any p-values below 0.0029 across any of our main models or sensitivity tests (i.e., Table S5), further suggesting that any significant findings reported across our models were likely due to chance.

Still, the pattern of effects across academic outcomes tended to favor the control group. This is surprising given that earlier exploratory analyses with this sample found that students in the intervention group attended higher-performing high schools (Watts et al., 2020), pushing back against the hypothesis that intervention adolescents had fewer resources to support their academic development in high school. Although a comprehensive mediational analysis exploring subsequent school quality as a mechanism is beyond the scope of the current paper, future analyses may examine whether selection into higher-performing schools actually hampered student achievement due to negative peer effects (see Marsh, 1987). However, we did examine if treatment impacts were moderated by enrollment in higher quality high schools. This test examined the “sustaining environments” hypothesis, which predicts that early intervention impacts may only last if they are followed by high quality environmental experiences in later periods (Bailey et al., 2017), a finding supported by earlier analyses from the CSRP sample in kindergarten (Zhai et al., 2012). Interestingly, we found no indication that our measure of high school performance, which directly predicted SAT scores and college enrollment (see supplementary file), meaningfully moderated the impacts reported here. These largely null interactions between the preschool intervention and a later measure of school quality comports with the conclusions of a recent systematic review (Bailey, Jenkins, & Alvarez-Vargas, 2020) and suggest that if these impacts (or the lack thereof) were driven by differential access to later educational environments, our measure of school performance was unable to capture the key features of what such sustaining environments might entail.

Relatedly, we saw little indication that the CSRP preschool intervention effects were enhanced by later exposure to multiple doses of a brief learning mindset intervention. However, it should be noted that we estimated largely null effects for the learning mindset intervention on the set of outcomes reported here (see supplementary file; see also Gandhi et al., 2020), so it remains unknown whether preschool impacts might have received a boost if the subsequent intervention had also produced meaningful short-run effects on behavioral and cognitive functioning. The possibility of interactions between these two interventions held promise at the outset of our study, as both sets of interventions had been shown to have positive impacts on children’s academic achievement (Paunesku et al., 2015; Raver et al., 2011). Yet, we found no consistent evidence across the outcomes assessed here to suggest that students benefitted by receiving the two interventions in succession during early childhood and adolescence, respectively. In our view, the field should continue to look for complementary interventions that can make meaningful impacts on students’ lives across developmental periods.

When considering how our results inform ECE policy and practice, these results are sobering, but not entirely unexpected. A recent cost analysis suggested that it would cost approximately $1360 per child to implement CSRP in ECE settings today (https://capproject.org/general-resources). Although disappointing that such an investment did not yield detectable benefits at the end of high school, it is important to recognize that this amount is relatively small compared to the estimated cost per-child to provide high-quality preschool services in the United States ($12,500; Friedman-Krauss et al., 2021). Indeed, our results suggest that we should consider modifying our expectations for the durability of early childhood curricular interventions. Prior studies of CSRP found that the program did improve children’s experiences in Head Start classrooms (e.g., Raver et al., 2008), and simply improving the quality of care during an important period of development might be motivation enough to continue to support similar intervention efforts.

Although the estimates shown here may seem discouraging for intervention developers hoping to make long-lasting changes to children’s trajectories through early childhood programs, we do believe these findings leave open several important areas for future research. First, future work should examine how strong early intervention efforts must be to produce long-term changes. Many have noted that the large long-term effects produced by early demonstration ECE programs could be due to the unusually large contrast between the treatment and control groups (Duncan et al., 2022), as children living in poverty in the 60s and 70s had little access to high quality ECE supports. Comparing these intervention efforts to modern-day programs is difficult because the intervention contrasts have become more incremental, raising the possibility that fadeout could be partly due to advances in the quality of counterfactual conditions in ECE studies (see Bailey et al., 2020). Indeed, the CSRP intervention was modeled on the assumption that providing key supports for self-regulation in Head Start classrooms would lead to changes in early self-regulatory capacity. However, it should be noted that everyone in the control group received standard Head Start programming, meaning that the CSRP intervention was really focused on providing services over and above what children already received in Head Start. Given the comprehensive nature of modern Head Start programming, perhaps this contrast was simply not strong enough to produce long-term effects. Thus, more research should seek to understand whether further improvements to today’s programs can meaningfully affect long-term developmental trajectories. Instead, it could be the case that intervention efforts are better placed on other types of programs that can support low-income families, as the availability of comprehensive ECE services has grown substantially for most low-income families in the past several decades (see Cascio, 2021).

More research should also continue to examine the long-term development of self-regulation, though we need more causal evidence in this area. Although correlational studies suggest that childhood behavioral and cognitive regulation predict a host of adult outcomes (see Ahmed et al., 2021; Moffitt et al., 2011), we failed to find any long-term effects for an intervention that explicitly targeted children’s early self-regulation and EF. Indeed, our generally null findings for educational outcomes also comport with the conclusions of a meta-analysis that examined links between EF and academic achievement, as Jacob and Parkinson (2015) found little evidence to suggest a causal relation between the two constructs. This was perhaps disappointing, but a few caveats should be taken into account. First, it should be noted that that participants in the current study were assessed during the transition from adolescence to adulthood, preventing us from capturing many of the key measures of adult life that have correlated with early self-regulatory capacities in longitudinal work (e.g., criminal behavior, markers of adult health, etc.). Second, another recent follow-up of an intensive intervention targeting early self-control in a group of highly dysregulated boys did find impacts on measures of adult attainment (Algan et al., 2022). Thus, our study does not rule out the possibility that self-regulation may provide a lever of early intervention. We simply need more evidence on the long-term effects of exogenously driven increases in self-regulation to develop better theory about how childhood self-regulation relates to adult functioning. In our view, our study is an important early step in this direction, but it will be difficult to draw firm conclusions until more experimental or quasi-experimental evidence can be brought to bear.

Limitations

Several important limitations should be noted. First, the study suffered from a number of design issues. As has been reported in previous evaluations of this program (Watts et al., 2018), we observed indications of baseline imbalance across the intervention groups at the study outset. Thus, these effects should be interpreted with some caution because causal inference is hampered when RCTs suffer from baseline non-equivalence. However, we detected relatively consistent results across models that did and did not control for baseline characteristics. Further, our models were underpowered to detect small, but important, differences between adolescents at follow-up. Indeed, standardized standard errors tended to fall between 0.10 and 0.15, suggesting that effects smaller than approximately 0.20–0.30 SDs were not likely to be detected. This power issue is due to the cluster-randomized design of the study coupled with the relatively small number of centers (n = 18) involved in the project.

Relatedly, any interpretation of the effects reported here should be considered against the apparent sensitivity of estimates to various modeling approaches, which has been demonstrated in the past (e.g., Raver et al., 2009; Watts et al., 2018). As we described above, due to the low power and baseline imbalance, treatment effect estimates reported from this project have been dependent on multi-level modeling decisions and the inclusion of control variables. Thus, if initial effects were equivocal, then it is unclear whether our results represent a pattern of long-term fadeout or small initial effects carrying forward. This point is further complicated by the fact that our measures of adolescent functioning do not cleanly align with the preschool measures from initial evaluation, even if a given construct has been carried forward (e.g., executive function measures differ across the waves). A full consideration of the best modeling approach for each wave of the study represents a viable avenue for future research, as would approaches that integrate outcome data across multiple waves (e.g., McCoy et al., 2018). In the current paper, our focus was to provide a comprehensive analysis of late-adolescent impacts. Given the various approaches employed in previous papers, we provided a range of estimates from models that aligned with the initial evaluations (Raver et al., 2009; 2011) as well as follow-up work (Watts et al., 2018). Even with these alternative approaches, we still detected largely null results, with no clear pattern of results favoring the intervention group.

Next, our measures were limited in both validity and reliability. Despite including a broad set of measures across multiple measurement modalities, our data still lacked many key measures that could represent crucial outcomes for late adolescence and early adulthood. As we stated above, we lacked sound measures of high school graduation, adult earnings, incarceration rates, and other outcomes that have been shown in the past to be sensitive to ECE intervention (e.g., Elango et al., 2016). We were also unable to capture all possible skills and strengths of this age group that could have been impacted by the CSRP intervention (e.g., relational skills, self-efficacy). Thus, future work may find differences in additional outcomes that we were unable to examine here. Moreover, our measures were not devoid of measurement error, and several of our survey-based instruments had low inter-item reliability (the pro-social measures were particularly noisy). Some of these same measures are also relatively new to the field (e.g., critical consciousness), and it is unclear how sensitive they may be to intervention.

As with most evaluations of intervention programs, our study can provide “causal description,” but it lacks in “causal explanation” (see Shadish, Cook, and Campbell, 2002, p. 9). In other words, our study suffers from a “black box” problem as we cannot identify why the CSRP intervention failed to strongly impact late adolescent and early adult outcomes. Mediational analyses were simply beyond the scope of this paper, but this limitation could be further addressed in future work leveraging other waves of study data. However, achieving causal inference with mediational analyses is difficult, and such difficulties will be exacerbated by the lack of power in our study. Nevertheless, we recognize that our study is limited in its ability to determine mechanisms that explain the relation (or lack thereof) between the intervention and the outcomes considered here.

Finally, our results were generated from a single sample of children from low-income families growing up in the Chicago area. These results may not generalize to similar interventions in other populations (see Morris et al., 2013). Relatedly, any generalizations from this work should be considered against the historical context within which the sample lived. Indeed, the Great Recession peaked during our cohort’s childhood years, and the COVID-19 pandemic struck just at the end of our observation window for college enrollment (though as we mentioned above, we saw few indications that it affected enrollment patterns for the students most likely to be affected based on age). Moreover, the ECE context has changed considerably since these students participated in Head Start, with state-funded pre-k programs growing in influence. Thus, as with any longitudinal study in Developmental Psychology, it is difficult to know how these contextual factors might influence the long-term developmental patterns we observed.

Conclusion

Few studies of ECE curricular interventions have followed participants into the long-term, and even fewer studies have been able to collect a robust and broad set of measures of late adolescent functioning. The findings reported here indicate that the CSRP preschool intervention did not have sustained effects on adolescent behavior and cognitive development that were detectable toward the end of students’ high school careers. These results suggest that although self-regulation may be a core capacity for school readiness, targeting self-regulation in early childhood alone may not lead to strong impacts on later developmental outcomes. More work is needed to understand the important ingredients of early childhood educational programs that will consistently lead to sustained benefits for disadvantaged children. Furthermore, our findings suggest that investments across developmental periods are likely necessary to produce meaningful long-lasting change on children’s developmental outcomes.

Public Significance Statement:

This study evaluated the long-term effects of an early childhood intervention focused on self-regulation. Despite initial promise, the authors found virtually no evidence to suggest that the preschool intervention had long-lasting positive effects on key outcomes such as achievement, college enrollment, or behavioral dysregulation. This work carries important implications for our continued development of early childhood programs targeting children from low-income families.

Data Availability

A comprehensive version of the CSRP adolescent dataset is available upon request from ICPSR: https://www.icpsr.umich.edu/web/ICPSR/studies/38425. However, the data posting does not contain item-level data for most scales, nor does it contain data obtained from third parties (i.e., College Board, ACT, and National Student Clearinghouse). Because our data use agreements prohibit us from making the full data used in the current publication available for public use, we have included our Stata analytic syntax files as online supplemental material.