School Readiness in the Midwest Child-Parent Center Expansion: A Propensity Score Analysis of Year 1 Impacts

Abstract

In this paper, we evaluate the effectiveness of the first year of a federally-funded, evidence-based preschool through third grade intervention in Chicago. We use inverse probability weighting with regression adjustment to estimate the impacts of the Child-Parent Center (CPC) program on teacher assessments of school readiness for 1,289 low-income preschool and 591 comparison-group participants. Results indicated significant positive impacts of the program for all domains, including literacy, math, socio-emotional development, science and total score. The percentage of CPC children who met national norms in school readiness exceeded the comparison group by 12 to 18.5 percentage points. Full-day participants experienced greater school readiness gains while program impacts were similar by family income and home language. Compared to the original CPC evaluation of children born in 1980 in which few comparison group children attended preschool, we find evidence that the contemporary implementation performs at least as well even though the current comparison group participants had alternative preschool experience.

1. Introduction⁶

In recent years, public funding of early childhood programs has continued to expand. Expenditures for the U. S. Department of Education’s Race to the Top contest, preschool development grants to states, enhancements to federal Head Start programs, block grants to states for child care, and state expansion of prekindergarten programs totaled more than a billion dollars in new funding over the past five years (Council of Economic Advisors, 2015). Public-private sector initiatives such as Pay for Success have also been implemented to help fund public preschool programs (Temple & Reynolds, 2015). An important goal of these investments is expanded access to high-quality programs at city, state, and national levels.

Extensive research has consistently shown that participation in effective preschool and early education programs can improve school readiness skills, subject-matter achievement, and reduce the need for later remedial education services (Karoly & Auger, 2016). However, some of the evidence of the effectiveness of preschool programs is mixed, indicating that while there are developmental and cognitive effects, state-funded preschool programs may not impact special education rates, behavioral problems or parent involvement (Gilliam and Zigler, 2000). Other preschool programs have been found to have a drop off in effects as children get older (Currie and Thomas, 1993), though evidence of effectiveness persists in several areas, including increased earnings and lower likelihood of committing crime (Garces, Thomas and Currie, 2000). Exemplary and high-quality programs have also demonstrated economic returns of more than 7 dollars per dollar invested (Heckman, Moon, Pinto, Savelyev & Yavitz, 2010; Reynolds and Temple, 2008; Schweinhart, Barnes & Weikart, 1993). This evidence was cited in the President’s State of the Union Address in 2013 as the rationale for the Preschool for All initiative.

One of the leading examples of an evidence-based, early childhood program with a high return-on-investment is the Child-Parent Centers (CPC; Reynolds, Temple, Robertson & Mann, 2002). CPC has not only shown positive effects on school performance and achievement (Reynolds and Ou, 2010), but also in crime reduction (Reynolds, Temple and Ou, 2010). The CPC program is currently undergoing scale-up in the Midwest through new federal investments and Pay for Success initiatives.

The Child-Parent Centers, a high quality, early education intervention that serves students from preschool through third grade, began operation in 1967. By the mid-1980s there were 24 centers in Chicago, but despite the well-documented positive effects of the program, the number of sites fell to 10 by 2011, primarily due to lack of financial support. In 2012, with support from the Federal government through an Investing in Innovation (i3) grant and local private funds, an effort was made to scale-up the CPC program as part of the Midwest CPC Expansion project (MCPC). Five school districts in Minnesota and Illinois opened new CPC locations. This paper focuses on the 10 existing and 6 additional CPC sites operating in Chicago.

While the impact of the original CPC program has been well-documented through the Chicago-Longitudinal Study (CLS), we investigate the impacts of the scaled-up program following key changes in the implementation, including responding to parental needs by offering full-day rather than half-day programming in some sites. Additional changes were to enhance parental involvement experiences and to offer additional professional development to teachers. Another key change was the nature of the comparison group. While in the 1980s few comparison group children attended preschool, in the current research the CPC participants are compared to children who attended state-funded preschool offered in the Chicago Public Schools. A final difference between the CLS study and the current evaluation of the CPC program is the inclusion of Hispanics in the current sample. The original study of the CPC program was based on a sample that was almost entirely African-American.

We examine the impacts of the program on teacher-administered assessments that evaluate students on their mathematics, literacy, socio-emotional and science skills, as well as an overall score of school readiness. We also investigate the impacts of full- and part-day CPC programming, as well as the differential impacts by free lunch status. We also examine CPC preschool effects for children whose families speak Spanish at home. Finally, we use the same methodology to examine the impacts to the first year of the original CLS sample on school readiness to compare the effect sizes across implementations, roughly 30 years apart. We focus on the first year of the program, analyzing the impacts of one year of CPC preschool. Given significant differences in the socio-economic characteristics of the treatment and control group despite using a matching procedure to determine the control sites, we use inverse propensity score weighting regression adjustment (IPWRA; Wooldridge, 2007) to estimate treatment effects and to address dosage issues of full versus half-day intervention. The estimation approach is described as doubly robust because results are unbiased if either the regression model or the propensity score equation is mis-specified.

1.1. Child-Parent Centers and the Midwest CPC Expansion

The original CPC program was offered in schools located in high poverty neighborhoods in Chicago and targeted students between the ages of 3 and 9. The CPC program offered a high-quality preschool program staffed by teachers with four-year degrees and small class sizes of 8 or 9 children per teacher or teacher’s aide, a 17:2 child to adult ratio. Started in the 1960s and still ongoing today, the CPC program is a comprehensive, educational intervention with an intensive parental involvement component. Students may enroll in the CPC preschool program for one or two years and then continue in the elementary school component of the program that offers small class sizes, field trips, and a modest amount of additional classroom resources.

The name “Child-Parent Center” indicates that parental involvement is an important component of the program. Parents were expected to volunteer at least one-half day each week at the center in various capacities. Each CPC site had a dedicated parent-resource teacher who encourages parent participation and a parent resource room that provides a location for parent program activities.

The effectiveness of the CPC program is well-documented. The Chicago Longitudinal Study follows 1,539 low-income, minority students born in 1980. Of the total sample, 989 attended CPC preschool programs at twenty sites, while the control group of 550 attended either five randomly-selected Chicago Public Schools with alternative programs or enrolled in CPCs in kindergarten without preschool participation (Reynolds, 1999). Reynolds, Temple, White, Ou and Robertson (2011) examine the impacts of the CPC on educational achievement, special education, crime and welfare. These benefits are monetized and weighed against the costs of the program. Using adult data through age 26, the authors find a total public and private $10.83 return for every dollar invested for the preschool program and a $3.97 return for every dollar invested for the school-age program. The largest components of these benefits are the savings of reduced crime and the increased earnings capacity and tax revenue (Reynolds et al, 2011). These results are indicative of the positive effects the CPC for both the individual and society as a whole. Arteaga, Humpage, Reynolds and Temple (2014) use propensity score weighting to analyze the dosage impacts of one or two years of CPC participation using the CLS sample. Similar to previous studies, they find positive impacts of one year of program participation on academic achievement, health and adult SES outcomes. In addition, students that elect to attend CPC for a second year are less likely to receive special education or to commit crime compared to those students that only received one year of the intervention. A recent evaluation of the CPC preschool program by Gaylor et al. (2016) indicated that participation was associated with a lower probability of special education placement in kindergarten and higher rates of school readiness. These studies are a small selection of the available literature on the CPC, (others, including Temple and Reynolds (2007)), but they are representative of the positive results seen by both individual students and to the communities that implement the program.

Although the program has been in existence for 50 years, studies of the CPC are especially important in light of the recent strong national interest by policymakers in Prek-3 education. As described by Shore (2009) and Takanishi (2011), concern about the perceived lack of persistence of the benefits of preschool, especially for economically-disadvantaged children, has focused attention on the need for early programs that last longer, perhaps through third grade.

Given the success of the original CPC program, the model was modified and expanded in the 2012–2013 school year in Chicago, Evanston and Normal, IL and Saint Paul, MN. Drawing from the foundation of the original model, the expansion program focused on parental involvement, small class sizes and improving kindergarten readiness. However, the Midwest CPC expands on the original model by developing six main components that must be met: a collaborative leadership team lead by a head teacher, effective learning experiences driven by small class sizes and certified teachers, parent involvement and engagement, an aligned curriculum from preschool through 3^rd grade, a focus on continuity and stability, and professional development (Human Capital Research Collaborative, 2016). In the first year of implementation, there was a strong push to develop key leadership teams in each site, led by the principal. These teams focused on offering an aligned curriculum from preschool through third grades to reduce the possibility of fadeout effects and increased professional development for the teachers. These six components are the driving factors for increasing the human capital skills of the CPC students, but also increase the generalizability of the model.

In the first analysis of the Midwest CPC expansion, Reynolds, Richardson, Hayakawa, Lease, Warner-Richter & Englund et al (2014) analyze the impacts of full-day preschool. Using a sample of students in Chicago from schools that had both full and part day CPC programs, Reynolds et al (2014) find positive impacts of the full-day program on early school readiness, measured by the teacher assessment Teaching Strategies GOLD (TSGOLD), as well as improved attendance, both increased average daily attendance and reduced chronic absence. Compared to the original CPC program that almost exclusively served low-income, black students, the Midwest CPC program provides services to a much wider variety of students, which allows for more robust subgroup analysis.

1.2. Scaling Up Programs

While high-quality, model preschool programs have been shown to have strong impacts on children’s development (Schweinhart, 1997; Reynolds, 2000; Campbell, Ramey, Pungello, Sparling, & Miller-Johnson, 2002), not as much research is available to document the success with which model preschool programs can be brought to scale. Impact evaluations have shown mixed results on the ability of scaled-up programs to produce the same lasting impacts for children as model programs, with some studies showing gains fading out by early elementary (Puma, Bell, Cook, Heid & Lopez, 2010; Lipsey, Hofer, Dong, Farran, & Bilbrey, 2013;) and others showing sustained gains (Barnett, Jung, Youn, & Frede, 2013).

For an expansion to be successful, the scaled-up program must be “reasonably similar” to the model program (Schweinhart, 2007) so fidelity to the original model is important (Rohrbach & Dyal, 2015). Challenges to scale-up are adequate funding, as well as facility and leadership capacity (Lauter and Rice, 2008; Bumbarger & Perkins, 2008; Rohrbach & Dyal, 2015), but strong collaboration between school districts, school principals, and preschool providers can help scale-up efforts to achieve positive results for children (Lauter and Rice, 2008; Hayakawa, Englund, Candee, Lease, Sullivan, Warner-Richter, et al., 2015). Additionally, there are institutional and political factors that influence the capacity to scale, including cost, organizational commitment, and service fragmentation (Reynolds, Hayakawa, Ou, Mondi, Englund, Candee et al., in press; Cooper, Slavin, & Madden, 1997; Domitrovich & Greenberg, 2000; Greenberg, 2010).

2. Theory

At its core, the CPC program focuses on increasing the school readiness skills of its participants, including an emphasis on cognitive skills, literacy and numeracy, as well as non-cognitive skills like socio-emotional learning. These skills not only manifest in the first year of program, but also make subsequent learning more effective. The acquisition of these skills is fundamentally a human capital model (Becker, 1962; Grossman, 2000, Almond and Currie, 2010, among others) that focuses on that acquisition of skills across time periods. Duncan and Magnuson (2013) outline how preschool programs are fundamentally human capital investments that see returns in the long-run, even if there appears to be fadeout of impacts on academic achievement. Heckman (2006) notes the importance of investing in the skills of economically-disadvantaged children (which the CPC program serves).

We base our analysis on the model of human capital presented by Cunha and Heckman (2007), where the capabilities of a student are a function of the genetic and environmental conditions of the child at preschool entry, θ_t, the human capital of the parents, h, and the investment in education, I_t, at any age t, such that:

$${\theta }_{t+1}=f_t(h,{\theta }_t,I_t)$$ 1)

Thus the skills obtained at the end of the school year, time period t+1, are a function of the skills the student entered the school year with, the human capital skills of their parents, and the investment into human capital skills of the children that are made by the parents. We use this framework to help inform our model choice and to better understand the impacts of the CPC program. If we view attending CPC as increased investment in human capital skills, we should see an increase in those skills by the end of the preschool year, relative to the control sample and controlling for parent human capital and baseline human capital skills of the child. Similarly, we should be would expect the full-day group to outperform the part-day group when we examine the impacts within the CPC. Or, because:

$$\raisebox{1ex}{$\partial f_t(h,{\theta }_t,I_t)$}\!\left/ \!\raisebox{-1ex}{$\partial I_t$}\right.>0$$ 2)

where the delta term is a partial derivative and represents a predicted change in the measured skills resulting from a small change in the investment in education. Assuming that the magnitude of the education investment is described as:

$$I_{\mathit{CPC}\mathit{Full}-\mathit{Day},t}>I_{\mathit{CPC}\mathit{Part}-\mathit{Day},t}>I_{\mathit{Control},t},$$ 3)

then given equation (2) we should then see the outcomes varying as:

$${\theta }_{\mathit{CPC}\mathit{Full}-\mathit{Day},t+1}>{\theta }_{\mathit{CPC}\mathit{Part}-\mathit{Day},t+1}>{\theta }_{\mathit{Control},t+1}.$$ 4)

Moreover, we would also expect f_t(h, θ_t, I_t) to be increasing in h, the parent’s human capital. If we are to isolate the impact of increased investment due to CPC, we must ensure that we account for any differences in h and θ_t, which include measures of parent education and fall baseline test scores, respectively. Without controls for those measures, we may incorrectly attribute increases in human capital from CPC investment to unrelated family or child characteristics.

3. Method

3.1. Sample

The total enrollment sample for the first year of the Midwest CPC scale-up program consists of 1,724 students in Chicago CPCs and 868 students in matched Chicago Public School preschools. While in the total sample 79 percent of the students were present for the entire school year, frequent mobility among the other students led us to define program participation as at least four months of attendance in a CPC site and enrolled in the program by January 1st. The administrative data received from Chicago Public Schools contained information on more students than could be physically accommodated in the preschool locations at any one point in time. By defining the sample with this rule, we ensure that students were present for a minimum of half the school year and that the enrollment matches the physical capabilities of the preschools.

Our sample includes 1,289 students in Chicago CPCs and 584 students in public preschool at matched control schools that had at least one valid TSGOLD score. We analyze the 10 existing and 6 the new CPC sites in this sample. The six additional sites were chosen to help broaden the target population of the intervention. Using all Chicago public preschools that were not part of the CPC program, matched control sites were chosen based on a propensity score of school participation, estimated on key demographic characteristics of the schools, including ethnic breakdown and 3^rd grade test scores. Schools were matched individually based on these propensity scores, the neighborhood of the school and the schools’ willingness to participate. All but one school agreed to participate and another control site was chosen in that school’s place. These matched sites create a set of schools from which individual control students were drawn and compared to the CPC students. Control students received half-day preschool programming that represent the typical Chicago Public preschool services including Head Start and the state pre-k program. All students, treatment and control, received preschool programming, but the CPC group received additional services above and beyond what the control group students received.

Table 1 presents the individual level characteristics of the sample, both at the beginning of the preschool year and the end of the year.

Table 1.

Characteristics of CPC and Comparison Groups at Preschool Entry, 2012–2013

Child/Family Characteristics**	CPC Group (N=1,724)	Comparison Group (N=868)	Original Sample* p-value	End of Year p-value
Female child, %	51.6	50.1	.67	.97
Black, %	64.1	44.5	<.01	<.01
Hispanic, %	34.1	54.8	<.01	<.01
Home language is Spanish, %	27.2	48.9	<.01	<.01
School-level proficiency at state assessment (grades 3–8; %)	62.4	60.8	.03	<.01
Age in months on Sept. 1, 2012 (mean)	48.4	48.6	.42	.68
Enrolled as 3-year-olds on Sept. 1, 2012, %	40.4	38.7	.40	.79
Special education status (IEP), % a	9.6	9.1	.67	.95
Child eligible for fully subsidized meals, % a	85.4	84.0	.33	.93
Single parent family status, % a	48.8	46.7	.52	<.01
Fall score on Literacy subscale, mean (SD)	33.7(15.3)	31.4(13.0)	<.01	<.01
Fall score on Math subscale, mean (SD)	22.6 (8.5)	23.2 (7.2)	.13	.56
School readiness, Fall total scale (SD)	192.1(58.8)	190.6 (49.0)	.50	.35

^*Original sample was participants who enrolled in the program and comparison group. End-of-Year sample had valid values for one or more outcome indicators. P values show the significance of mean (or percentage) group differences for the program and comparison groups. The comparison group participated in the usual preschool programs in Chicago (Head Start and State PreK) and were matched on the school-level propensity to enroll in the program.

^**Data on child and family characteristics were collected from school administrative records with the exception of low-income status which was a combination of administrative records and parent reports.

^aChildren have an Individual Education Plan under IDEA. N for single parent family status is 1,455 (parent survey).

^bEligibility defined at <130% of the federal poverty level.

As seen above, there are significant differences in race, home language and the literacy baseline test score (adjusted for age, as the sample includes both 3- and 4-year old students), though this may be accounted for by the significant difference in percent Hispanic and ELL status. There is no significant difference in math scores or the summed total score of the six domains. Despite the attempt at school level matching, the significant differences at baseline must be addressed in order to draw casual inferences of the effectiveness of the MCPC Expansion.

3.2. Key Outcome Measures

To estimate the impact of the CPC program on school readiness, we used scores on the Teaching Strategies Gold Assessment System (TSGOLD; Lambert, Kim, and Burts, 2013a, 2013b, 2014). As a teacher-rated performance assessment measuring multiple domains of school readiness, TSGOLD ratings are completed three times during the school year (fall, winter, spring). They are routinely collected as part of the school district assessments. Based on observations of classroom performance, teachers rated students on a scale from 0 to 9 (low to high skill proficiency). For science, the items were rated from 0 to 2. Raw scores are summed to obtain subscale scores and whether students meet national norms is based on age and the norming population (Lambert et al., 2013a, 2013b; Soderberg, Stull, Cummings, 2014). Internal consistency reliabilities are high (> .90; Lambert et al., 2013a). Table 2 presents summary statistics of the domains, as well as sample items from each subscale.

Table 2.

TSGOLD subscale sample items and means

Domain	Sample Items*	Fall Mean (SD) Spring Mean (SD)	Percent at/above National Norm
Literacy	Identifies and names letters	33.3 (15.9)	10.5%
12 items	Uses and appreciates books	57.2 (17.7)	71.5%
Math	Counts	23.0 (8.9)	8.5%
7 items	Quantifies	36.3 (9.5)	69.4%
Science	Uses scientific inquiry skills	4.5 (2.2)	N/A
5 items	Demonstrates knowledge of the characteristics of living things	8 (2.3)
Socio-emotional	Manages feelings	40.6 (13.0)	10.8%
9 items	Balances needs and rights of self and others	55.4 (11.9)	60.1%
Total Score	Sum of six domains: Literacy, Math, Cognitive Development, Socio-emotional and Physical Health	192.8 (60.0)	12.0%
		277.8 (61.0)	64.1%

^*Items are rated on a scale of 0 to 9 (Science 0 to 2). Scores at or above the national average in spring are as follows for 3- and 4-year-olds: Literacy (39, 56), Math (27, 37), Socioemotional (46, 57). Meeting the national norm in total score was defined as meeting the norm in at least 3 domains in the fall and 4 in the spring..

The validity of performance assessments is well documented by the National Research Council (Snow & Van Hemel, 2008). The advantage of TSGOLD and similar assessments is that scores are based on greater knowledge of child behavior in the school context. Ratings, for example, occur after 4 to 6 weeks of observation in the classroom. The assessment is also aligned to district and state standards, covers all domains of learning, and is linked to the curriculum and opportunity to learn principles. In support of construct and convergent validity, TS GOLD scores are moderately correlated with standardized assessments, and are predictive of later learning (Joseph, McCutchen et al., 2011; Kim, Lambert, & Burts, 2013; Teaching Strategies, 2011).

We examined the impact of CPC on TSGOLD math, literacy, socio-emotional, and science domains, as well as a total score comprised of the six domains (math, literacy, language, socioemotional wellbeing, physical heath and cognitive development). We also analyze the impact of CPC on meeting the national norm standard. No national norm is available for science. Missing data were imputed using the Expectation-Maximization (EM; Schafer & Olson, 1998) algorithm, which yields valid estimates under the assumption of data missing at random. Given the extensive baseline data available for children and families, this assumption was satisfied. The input variables included race, gender, age, free lunch status, parent information including single parent status, employment status and education level and all available TSGOLD data. The sample was limited to students with at least one valid TSGOLD score (1,873 students). There was no significant difference in the TSGOLD scores by the imputed or non-imputed sample, see Appendix A. The fully imputed sample was 2,592. The pattern of findings on program impacts was similar for non-imputed and imputed data.(see Appendix B.

Table 3 presents t-tests of the unadjusted mean differences in the outcome data by CPC and comparison and by imputation sample. Columns 1–3 present the partially imputed 1,873 sample, while 4–6 examine the fully imputed sample.

Table 3.

Imputed, unadjusted mean differences in Spring TSGOLD scores

VARIABLES	Partially Imputed Sample (n=1,873)			Fully-Imputed Sample (n=2,592)
	CPC (n=1289)	Control (n=584)	p-value	CPC (n=1724)	Control (n=868)	p-value
Math	37.62	33.57	0.000	37.29	33.55	0.000
Percent at National Norm	75.25%	56.68%	0.000	77.49%	56.45%	0.000
Literacy	60.24	50.67	0.000	59.53	50.72	0.000
Percent at National Norm	79.21%	54.45%	0.000	82.14%	56.76%	0.000
Socio-emotional	57.11	51.59	0.000	56.84	51.38	0.000
Percent at National Norm	67.81%	43.49%	0.000	70.30%	40.44%	0.000
Science	7.97	7.30	0.000	7.93	7.22	0.000
Total	287.37	257.17	0.000	285.32	256.73	0.000
Percent at National Norm	70.91%	49.41%	0.000	73.84%	47.18%	0.000

Before accounting for group differences, we see significantly higher raw test scores and percent at the national norm for the intervention group, suggestive of positive impacts of the CPC program. Below we discuss the use of a robust propensity score analysis to determine the impacts of the program, accounting for differences in the distribution of baseline characteristics.

When we compare the results of the first year of the MCPC expansion program to the original CLS cohort that attended CPC in the 1980’s, we cannot use the TSGOLD for the CLS cohort as it was not administered. However, at the beginning of the kindergarten year, the CLS students were tested on school readiness skills using the Iowa Test of Basic Skills (ITBS). The ITBS is a reliable and valid (Hildebr and, Hoover and Hildebrand, 1987) assessment that captures student ability on reading and math.

3.3. Analysis Plan

To estimate the impacts of the first year of the Midwest CPC program, we use inverse propensity score weighting with regression adjustment (IPWRA). Given the significant differences in the groups at baseline, our analysis focuses on propensity score analysis (Rosenbaum and Rubin, 1983) that can potentially reduce bias in nonexperimental study designs. As described in Wooldridge (2007) as a doubly-robust estimator, inverse propensity score weighting with regression adjustment involves a comparison of two regressions modeling the outcomes – one for the treatment group and the second for the comparison group. Inverse propensity score weights are used to estimate corrected regression coefficients. The estimated treatment effect in a regression adjustment framework is the difference between the two estimated constant terms from the two regressions.

We follow guidelines laid out in Caliendo and Kopeinig (2008) to estimate the probability of program participation, focusing on variables that only occur prior to program participation and are informed by prior CPC studies (Arteaga et al, 2014; Reynolds et al., 2011) and economic theory from the human capital model. Given the importance of the parent’s investment in Heckman’s human capital model, we include parent survey data in our CPC prediction model. We estimate the propensity for CPC participation in equation 4 using multiple specifications, controlling for various baseline characteristics.

$$Pr({\mathit{CPC}}_i=1\mid X)=\beta X_i+e_i$$ 5)

Where the probability of attending a CPC preschool is a function of demographic characteristics of both the child and the family. Table 3 presents three probit models to predict CPC program participation. Model 1 uses only the administrative data available from the start of the preschool year. Model 2 includes several parent survey variables completed by families at the beginning of the school year, imputed using MVA methodology and demographic characteristics. Also included in this model is an indicator of whether or not a family completed the parent survey. The final model is included for robustness testing and includes a baseline TSGOLD test score as well as a school level achievement of 3^rd graders in the year the preschool students entered the school. The baseline TSGOLD score was not a significant predictor. We use model 2 for our analysis. The estimation results did not fundamentally change across the three prediction models, see Appendix B for results.

Next, we compute the inverse probability weights and estimate weighted regression models to obtain predicted outcomes for each treatment level, z_i.

$$w_i=\frac{z_i}{Pr({\mathit{CPC}}_i\mid X)}-\frac{1-z_i}{1-Pr({\mathit{CPC}}_i\mid X)}$$ 6)

The estimated weight, w_i, favors treatment students with a lower probability of attending CPC, based on demographic characteristics, while weighting control students with a higher probability of attending CPC (despite attending a control site) more heavily. This weighting procedure creates a control group that is more similar to the treatment group and helps minimize bias that arises from differences in the distribution of observed covariates (Rosenbaum and Rubin, 1983). We begin by estimating the probability of CPC participation or not, but extend the analysis to a multi-level weight for the probability of attending CPC full-day, CPC half-day or control preschool programming. We use a Poisson distribution for TSGOLD data and logistic distribution for the national norm dummy variables. We take the difference in the predicted, weighted outcomes by treatment status to estimate the Average Treatment Effect (ATE) to estimate the causal impact of the Midwest CPC program (Wooldridge, 2010). In the outcome model, we include all variables from the propensity score model as well as a school-level reading achievement and fall baseline TSGOLD scores for each domain.

We focus on IPWRA estimation for several reasons; first, the estimation strategy allows flexibility in estimating both the CPC participation model and the outcome model. This allows us to include variables that may influence the outcome, such as baseline test scores and school quality, but may not influence the probability of treatment participation, such as fall TSGOLD scores, for example. Another advantage of the estimation strategy is that IPWRA methods are doubly robust (Robins and Rotnitzky, 1995; Wooldridge, 2007) in that only the specification of either the treatment prediction model or the outcome needs to be correct to provide consistent estimate of the impacts of the CPC program. Finally, unlike propensity score matching, IPWRA methods use all available data without discarding observations and allow for multilevel treatment variables, so we can estimate the impact of differences in CPC full or half-day participation versus the control students.

4. Results

We begin by estimating the impacts of the first year of the CPC program on TSGOLD scores and national norm rates. By weighting the outcome regressions by the estimated weights obtained from the program participation model, we are able to create a more comparable control group, thereby reducing the standard errors of the coefficients. Table 5 describes these results using the partially imputed sample of 1,873 (see Appendix for fully imputed estimates).

Table 5.

Impacts of CPC on school readiness

VARIABLES	(1) Math	(2) % at national norm	(3) Literacy	(4) % at national norm	(5) Socio-emotional	(6) % at national norm	(7) Science	(8) Total	(9) % at national norm
CPC vs None	3.6*** (0.242)	11.9*** (0.0248)	6.4*** (0.543)	18.4*** (0.0250)	4.0*** (0.292)	16.6*** (0.0269)	0.7*** (0.0969)	22.2*** (1.472)	13.0*** (0.0247)
Observations	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873

Notes: Propensity score model controls for race, gender, special education status, age of the student in months, free lunch eligibility, Mother’s education and employment status, single parent status and an indicator if the family did not complete the parent survey. The propensity score weighted outcome model controlled for race, gender, special education status, age of the student in months, free lunch eligibility, Mother’s education and employment status, single parent status, an indicator if the family did not complete the parent survey, 3^rd grade school level reading scores, fall baseline TSGOLD score and the month the student was assessed in the fall. Robust standard errors in parentheses.

^***p<0.01,

^**p<0.05,

^*p<0.1

CPC students score significantly higher on all school readiness domains, including the likelihood of meeting the national norm standard. CPC students have significantly higher scores (from 3.6 points higher in math to 6.4 points higher in literacy) on all TSGOLD domains tested, compared to similar students receiving the typical Chicago Public preschool programming. Like the original CPC program, the Midwest CPC Expansion program is effective in increasing school readiness among its participants.

4.1 Full-Day CPC vs Part-Day CPC

Given the results in Reynolds et al (2014), it seems possible that that the impacts of the CPC program are being driven by the higher performance of students in the full-day programs and the impacts for CPC part-day students may be minimal. The IPWRA model allows us an intuitive method to analyze this question. We create a trichotomized treatment level variable, where 2 indicates attendance in a CPC full-day program, 1 indicates attendance in a CPC half-day program, and 0 indicates attendance at a control preschool. The IPWRA model allows us to estimate the probability of attending each of those levels and then weights the outcome model by those results. We use CPC part-day participation as the baseline level as this allows us to compare the CPC part-day students to the control group students, while also comparing the CPC full-day students to the CPC part-day students. This allows us to test for significant differences among intervention levels.

We find that the full-day preschool group outperforms the half-day preschool group in raw TSGOLD scores and meeting the math and total national norms, though there is no significant difference in the likelihood of meeting national norms of literacy and socio-emotional outcomes between the intervention groups. The part-day group consistently outperforms the control group in raw score and the percentage of students meeting the national norm for each domain. This demonstrates that the overall impacts of CPC are not exclusively driven by the full-day group, but there are positive, significant impacts of the part-day program as well. Students gain skills in the half-day program and continue to improve those skills with extended classroom time.

4.2 Subgroup Impacts

Given one of the goals of the Midwest CPC program was to evaluate the impacts of the program on different demographic groups, we investigate the impacts of the CPC program for two key subgroups: Spanish-speakers at home and free lunch eligible students. For each subgroup, we estimated a new propensity score model, limiting the sample to that particular subgroup. The first row of Table 7 limits the sample to only those students eligible for free or reduced price lunch (n=1,660). Given the both the CPC and control preschools serve communities with very high rates of poverty, the majority of the sample is eligible for at least some reduction in lunch price. We also estimate the impacts of CPC on those students not eligible for free or reduced lunch.

Table 7.

Impacts of CPC participation by free lunch eligibility

VARIABLES	(1) Math	(2) % at national norm	(3) Literacy	(4) % at national norm	(5) Socio-emotional	(6) % at national norm	(7) Science	(8) Total	(9) % at national norm
CPC free/reduced lunch vs Control free/reduced lunch (n = 1,660)	3.86*** (0.262)	11.0*** (0.0274)	7.30*** (0.594)	17.1*** (0.0275)	4.134*** (0.315)	16.0*** (0.0302)	0.638*** (0.104)	24.1*** (1.585)	10.7*** (0.0270)
CPC non reduced lunch vs control non reduced (n=213)	2.31*** (0.736)	21.5*** (0.070)	1.80 (1.433)	35.9*** (0.0763)	3.996*** (0.639)	34.2*** (0.068)	1.413*** (0.250)	15.0*** (3.337)	37.9*** (0.0719)
Observations	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873

Notes: Propensity score model controls for race, gender, special education status, age of the student in months, Mother’s education and employment status, single parent status and an indicator if the family did not complete the parent survey. The propensity score weighted outcome model controlled for race, gender, special education status, age of the student in months, Mother’s education and employment status, single parent status, an indicator if the family did not complete the parent survey, 3^rd grade school level reading scores, fall baseline TSGOLD score and the month the student was assessed in the fall. Robust standard errors in parentheses.

^***p<0.01,

^**p<0.05,

^*p<0.1

We find that regardless of free lunch status, CPC students outperform control preschool students of the same lunch status. Thus, the results, especially the gains seen in raw TSGOLD test scores, are not driven exclusively by more economically advantaged families. The gains in raw TSGOLD scores are comparable between CPC subgroups, though those gains are more likely to translate into meeting the national norm for the students not eligible for free lunch. Those students have higher baseline test scores so gains are more likely to translate to meeting the national norm. Thus, while the gains from CPC may be equivalent across free lunch status, it is easier for the non-eligible students to reach the national norm threshold.

Table 8 compares the impacts of CPC for students who speak Spanish at home compared to those control students that speak Spanish at home. We also compare the impacts of CPC home Spanish-speakers to CPC students that spoke all other languages at home (97.5% of the non-Spanish speakers spoke English at home). Again, we limit the sample by language spoken at home and re-estimate the propensity scores.

Table 8.

Impacts of CPC by language spoken at home

VARIABLES	(1) Math	(2) % at national norm	(3) Literacy	(4) % at national norm	(5) Socio-emotional	(6) % at national norm	(7) Science	(8) Total	(9) % at national norm
CPC Spanish at home vs Control Spanish at home Lang (n=622)	3.839*** (0.571)	26.3*** (0.0436)	3.91*** (0.956)	29.1*** (0.0424)	5.55*** (0.647)	34.7*** (0.0424)	1.473*** (0.177)	23.32*** (2.906)	35.5*** (0.0400)
CPC all other languages vs control all other lagnuages (n=1,251)	3.52*** (0.289)	5.63* (0.0312)	6.81*** (0.676)	12.4*** (0.0315)	3.34*** (0.372)	2.82 (0.0305)	0.486*** (0.119)	20.0*** (1.845)	0.53 (0.0287)
Observations	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873

Notes: Propensity score model controls for race, gender, special education status, age of the student in months, Mother’s education and employment status, single parent status and an indicator if the family did not complete the parent survey. The propensity score weighted outcome model controlled for race, gender, special education status, age of the student in months, Mother’s education and employment status, single parent status, an indicator if the family did not complete the parent survey, 3^rd grade school level reading scores, fall baseline TSGOLD score and the month the student was assessed in the fall. Robust standard errors in parentheses.

^***p<0.01,

^**p<0.05,

^*p<0.1

We find similar results to the lunch eligibility subgroup. Regardless of language spoken at home, CPC preschool students outperform similar control preschool students. When comparing the impacts of CPC for only those that spoke Spanish at home we see gains compared to the Spanish-speaking control students for both raw scores and those students meeting the national norms. Interestingly, home Spanish speakers typically had higher initial test scores in math and overall score, so comparable raw gains from CPC are more likely to translate to meeting the national norm in math and the overall score. CPC students that speak Spanish at home are more likely to meet the TSGOLD national norms, compared to control preschool students, by roughly 25 to 35 percentage points. While the mainly English speakers CPC students saw significantly higher raw scores in all domains, there was no significant difference in the rates of meeting national norms in socio-emotional learning and total summed score.

4.3 Comparison to the CLS model

An important aspect to this analysis is to compare the results of the first year of the scale up program to the results seen in the original CLS sample. With the MCPC sample, we have used TSGOLD scores at the end of the preschool year. The most consistent outcome measure in the CLS data is the Iowa Test of Basic Skills (ITBS) measured in the fall of kindergarten. IPWRA was used on the CLS data to understand if the effect size of the historic CLS model is similar to that of the CPC P-3 model. The CLS data contains a different set of covariates and does not include all the parent-level variables found in the i3 data. Instead of leaving out covariates to make the models match more closely, we tested the best possible model with the available data.

Using IPWRA to estimate the impacts of the CPC program on the total TSGOLD score, we find an estimate effect size of .38 of a standard deviation, certainly smaller than the impacts estimated in the CLS sample of students who attended CPC in the mid-1980s.

5. Conclusion

Study findings show that an expansion of the CPC program to new sites yields practically significant gains in school readiness skills. These gains occurred above and beyond those of students who attended the usual preschool programs in the Chicago Public School District. Analyses of a range of state prek (Magnuson, Meyers, Ruhn, & Waldfogel, 2004) and Head Start programs (Currie & Thomas, 1993; Puma, 2005) demonstrate that they can be effective in improving school readiness. Recent evidence on large-scale programs, including Tulsa and Boston, indicate the average effect sizes range from .18 to .63 (Yoshikawa, Weiland and Brooks-Gunn, 2016, Gormley, Gayer, Phillips & Dawson, 2005, Lipsey et al., 2013; Weiland & Yoshikawa, 2013; Wong, Cook, Barnett & Jung, 2008). With an effect size of about .40, the results of this study show that the innovations of the CPC program improve student performance significantly above already effective services. We hypothesize that these gains arise from the key requirements of CPC, which includes classes sizes of no more than 17, state-licensed teachers, a leadership team in each center, family support services, and professional development. These and other requirements were specifically designed to target and improve the school readiness of vulnerable populations.

Our findings were not exclusively driven by students with more learning time through full-day programming. Part-day preschool participants as well as their full-day counterparts outperformed the comparison group. Full-day participants made the largest gains, however. These benefits occurred for under-represented populations, including Hispanic children, and those from more diverse socioeconomic and ethnic contexts. Regardless of subgroup status, CPC students outperform the comparison students attending district programs. These results are important because the evidence of effectiveness for these subgroups has been limited in the original CLS evaluations and indicates a broader effectiveness of the program. This suggests that further scale-up of the CPC program is not only feasible but can produce larger effects than are typically found for publicly funded preschool programs (Camilli, Vargas, Ryan & Barnett, 2010). With evidence of effectiveness across multiple subgroups, including students that speak Spanish at home and by socio-economic status, these are indications of increased generalizability compared to the original CLS cohort.

In comparing the present study with CPC findings from the CLS, the larger effect size in the CLS sample is most likely due to the relative absence of preschool participation in the comparison group. Only 15% of the comparison group enrolled in publicly funded preschool whereas the entire comparison group in the present CPC enrolled in either state prekindergarten or Head Start. The .18 SD difference between the two estimates is roughly the impact of centered-based preschool versus home care in the ECLS. In addition, there were temporal differences in the school readiness measures. The TSGOLD measures were administered in the spring of the preschool year. The CLS school readiness measure was administered in the fall of kindergarten. Not only did students have additional months of development, but CLS students that attended CPC received at least one to two months of additional services. Finally, two major school district events occurred during the preschool year that may have deflated effect sizes. A nine-day strike occurred at the beginning of the year, which reduced the number of instructional days from which to contrast CPC versus usual preschool. Moreover, in the winter of the preschool year the district announced the planned closing or reconstitution of more than 50 schools, which were more concentrated among CPC schools. This process may have had a detrimental effect on the learning climate of the affected schools so important for school readiness skills. Consequently, our findings may be conservative.

Despite these caveats, the MCPC full-day students performed at a level comparable to the original implementation. This is evidence that the CPC scale-up program can be successful in contemporary educational contexts, but the program required key changes from the original model.

There are three notable limitations. First, there were significant differences in baseline characteristics between the treatment group and the control group, including race, home language and fall literacy TSGOLD assessment scores. However, using the doubly-robust propensity weighting methodology minimizes the bias that may arise from differences in the distribution of observed variables. It is also important to note that the positive impacts of CPC are still present across home language and racial subgroups.

The second limitation is the short-term nature of the study, as we only examine the impacts of the first year of the preschool program. Thus we cannot rule out the presence of drop-off effects that have been found in some other preschool interventions (Takanishi and Kauerz, 2008, Reynolds, 2003). However, previous research on the CPC program has demonstrated sustained effects (Reynolds et al., 2011). In addition, the comprehensive set of educational, family, and professional learning services is greater than most other programs. This would be expected to help sustain gains. As data are available on the performance of students in subsequent school years, this analysis should be expanded to investigate the impact of CPC on school performance in kindergarten and the elementary grades. The use of other assessments besides TSGOLD also will address robustness across outcome measures.

Finally, while we hypothesize that the key program elements account for the observed gains, further research should investigate the impacts of these elements on student achievement. Some but not all of these elements are present in state prek, Head Start, and other center-based programs. They warrant further testing and inclusion in expansion efforts. The availability of full-day services is also a key CPC feature, and the increased learning time was linked to large gains in school readiness.

The positive impacts of the CPC program should be viewed within the context of the changes since the 1980s assessed in the CLS. In the present program, six major elements are emphasized: effective learning experiences, collaborative leadership, parent involvement and engagement, aligned curriculum, continuity and stability, and professional development. The previous model emphasized only the first three, and with less intensity. For example, enhanced elements of effective learning experiences include a curriculum balance of teacher-directed and child initiated activities, full-day preschool, and progress monitoring of instruction. The current Midwest expansion also has a professional development system of coaching, provides program support by site mentors, and implements curriculum alignment and parent involvement plans in collaboration with school principals. These and other elements are likely to contribute to the positive effects. The CPC benefits on school readiness are the added value of the six elements above and beyond that of the usual services. The typically implemented district preschool programs have some but not all of these elements, and at lower degrees to intensity. For example, typical preschool class sizes are 20 whereas CPC has a maximum of 17. A leadership team is also present in each site to manage the entire program and help establish a strong environment for learning. The precise influences of these and other elements, and their contribution to sustained gains, warrant further investigation.

In conclusion, we find evidence that the CPC program is effective in increasing school readiness amongst its participants, across subgroups examined. If we allow for spring TSGOLD scores to represent human capital skills, θ_t+1, we find that, despite the limitations of this study, there is evidence that CPC serves as an effective investment for increasing human capital skills.

Table 4.

CPC Prediction Models

VARIABLES	(1) CPC	(2) CPC	(3) CPC
Black	0.546*** (0.171)	0.534*** (0.190)	0.740*** (0.197)
Hispanic	−0.309* (0.170)	−0.299* (0.178)	−0.430** (0.183)
Female	0.0308 (0.0636)	0.000200 (0.0665)	0.00530 (0.0673)
Special Education Status	0.196 (0.120)	0.262** (0.125)	0.184 (0.129)
Age in Months	0.00409 (0.00490)	0.00530 (0.00512)	0.0122* (0.00639)
Free Lunch Eligible	−0.198** (0.0976)	−0.317*** (0.106)	−0.291*** (0.108)
Mother High School Graduate		0.277*** (0.0938)	0.272*** (0.0943)
Single Parent Household		0.0905 (0.0980)	0.108 (0.0991)
Mother Employed		0.201** (0.0965)	0.223** (0.0975)
Missing Parent Survey Data		−0.978*** (0.0764)	−1.021*** (0.0774)
Fall Total TSGOLD score			−0.00860* (0.00469)
School-level Reading Score (3rd grade)			0.0157*** (0.00244)
Constant	0.261 (0.296)	0.326 (0.333)	−0.842** (0.382)
Observations	1,873	1,873	1,873

Robust standard errors in parentheses

^***p<0.01,

^**p<0.05,

^*p<0.1

Table 6.

Impacts of Full- and Part-Day CPC Participation

VARIABLES	(1) Math	(2) % at national norm	(3) Literacy	(4) % at national norm	(5) Socio-emotional	(6) % at national norm	(7) Science	(8) Total	(9) % at national norm
CPC Part-Day vs Control	2.9*** (0.249)	9.3*** (0.0261)	5.2*** (0.559)	17.0*** (0.0260)	3.4*** (0.315)	15.8*** (0.0277)	0.5*** (0.106)	18.2*** (1.530)	11.2*** (0.0259)
CPC Full-Day vs CPC Part-Day	2.4*** (0.625)	11.2*** (0.0427)	4.8*** (1.348)	6.3 (0.0395)	1.9** (0.827)	−5.8 (0.0514)	0.4* (0.161)	13.1*** (3.212)	8.8** (0.0435)
Observations	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873

Notes: Propensity score model controls for race, gender, special education status, age of the student in months, free lunch eligibility, Mother’s education and employment status, single parent status and an indicator if the family did not complete the parent survey. The propensity score weighted outcome model controlled for race, gender, special education status, age of the student in months, free lunch eligibility, Mother’s education and employment status, single parent status, an indicator if the family did not complete the parent survey, 3^rd grade school level reading scores, fall baseline TSGOLD score and the month the student was assessed in the fall. Robust standard errors in parentheses.

^***p<0.01,

^**p<0.05,

^*p<0.1

Table 9.

Comparison of Effect sizes by implementation

Program	MCPC overall	MCPC part-day	MCPC full-day	CLS sample
Effect Size (imputed sample)	.377	.309	.573	.565
Effect Size (non-imputed sample)	.365	.307	.597	.495

Notes: Propensity score model controls for race, gender, special education status, age of the student in months, free lunch eligibility, Mother’s education and employment status, single parent status and an indicator if the family did not complete the parent survey. The propensity score weighted outcome model controlled for race, gender, special education status, age of the student in months, free lunch eligibility, Mother’s education and employment status, single parent status, an indicator if the family did not complete the parent survey, 3^rd grade school level reading scores, fall baseline TSGOLD score and the month the student was assessed in the fall. The CLS model controlled for gender, Black, low-income neighborhood, single-parent family, mother’s age under 18 at birth of child, mother did not complete high school, more than four children in the family, participate in AFDC, maternal employment status, eligibility for subsidized meals, low birth weight, home environment problems, and child welfare case history by age 4.

Highlights.

• A scale-up PreK-3^rd program proves successful in increasing school readiness.

• Impacts of the program are significant across subgroups examined.

• Increases in school readiness are comparable to original program implementation.

Appendix A

Non-imputed versus Imputed, mean differences in Spring TSGOLD scores

VARIABLES	(1) Non-imputed Sample (n=1,446 – 1,582)	(2) Imputed Sample Control (n=1,873)	(3) p-value
Math	36.32	36.36	0.9108
Literacy	57.14	57.26	0.8485
Socio-emotional	55.46	55.39	0.8686
Science	7.75	7.76	0.8903
Total	276.72	277.95	0.5676

Appendix B: Results from alternative specifications

Table B.1.

Results by Imputation Sample

VARIABLES	(1) Math	(2) % at national norm	(3) Literacy	(4) % at national norm	(5) Socio- emotional	(6) % at national norm	(7) Science	(8) Total	(9) % at national norm
CPC vs None, non-imputed	3.9*** (0.366)	8.4** (0.0329)	8.3*** (0.852)	18.4*** (0.0466)	2.2*** (0.568)	0.21 (0.0348)	1.2*** (0.140)	21.5*** (2.415)	11.4** (0.0445)
CPC vs None, imputed (n=1873)	3.7*** (0.241)	12.0*** (0.0252)	6.4*** (0.538)	18.5*** (0.0252)	4.0*** (0.288)	16.6*** (0.0280)	0.7*** (.09746)	22.4*** (1.458)	12.8*** (0.0253)
CPC vs None, fully imputed (n=2630)	3.8*** (0.241)	18.2*** (0.0252)	6.4*** (0.381)	24.3*** (0.0198)	4.5*** (0.204)	27.0*** (0.0207)	0.8*** (0.0723)	24.1*** (1.018)	22.3*** (0.0198)

^***p<0.01,

^**p<0.05,

^*p<0.1

Table B.2.

Results by CPC prediction model

VARIABLES	(1) Math	(2) % at national norm	(3) Literacy	(4) % at national norm	(5) Socio- emotional	(6) % at national norm	(7) Science	(8) Total	(9) % at national norm
CPC vs None, Model 1	3.6*** (0.230)	12.2*** (0.0230)	6.5*** (0.498)	18.5*** (0.0227)	4.3*** (0.273)	18.6*** (0.0242)	0.7*** (0.0930)	23.2*** (1.358)	13.8*** (0.0224)
CPC vs None, Model 2	3.7*** (0.241)	12.0*** (0.0252)	6.4*** (0.538)	18.5*** (0.0252)	4.0*** (0.288)	16.6*** (0.0280)	0.7*** (.09746)	22.4*** (1.458)	12.8*** (0.0253)
CPC vs None, Model 3	3.6*** (0.238)	11.9*** (0.0242)	6.3*** (0.545)	18.5*** (0.0237)	3.9*** (0.290)	16.8*** (0.0267)	0.7*** (0.0948)	21.9*** (1.456)	13.2*** (0.0240)
Observations	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873	1,873

^***p<0.01,

^**p<0.05,

^*p<0.1