Unpacking the Black Box of the CSRP Intervention: The Mediating Roles of Teacher-child Relationship Quality and Self-regulation

Abstract

This study examines the theory of change of the Chicago School Readiness Project (CSRP), testing a sequence of theory-derived mediating mechanisms including the quality of teacher-child relationships and children’s self-regulation. The CSRP is a multi-component teacher- and classroom-focused intervention, and its cluster-randomized efficacy trial was conducted in 35 Head Start-funded classrooms. A series of increasingly complex and conservative structural equation models indicate that the CSRP carries its effects on children’s academic and behavioral outcomes through changes in teacher-child relationship quality and children’s self-regulation.

The last decade has witnessed the convergence of the developmental and prevention sciences in the design, implementation, and rigorous experimental evaluation of high quality early childhood programs designed to build children’s school readiness in both the academic and social-emotional domains (e.g., Bierman, Nix, Greenberg, Blair, & Domitrovich, 2008; Gormley, Gayer, Phillips, & Dawson, 2005; Raver, Jones, Li-Grining, Zhai, Metzger, & Solomon, 2009; Zigler, Gilliam, & Jones, 2006). Several recent demonstrations have moved beyond relatively narrow skill-based approaches (e.g., Preschool Evaluation Research Consortium, 2008) to focus on a set of core developmental competencies that research suggests underlie both social-emotional/behavioral and academic functioning. Specifically, research has highlighted the important role of behavioral and emotional regulation and executive functioning (e.g., Raver, Garner, & Smith-Donald, 2007). Evidence from interventions targeting these competencies indicates they are powerfully influential in building children’s readiness for kindergarten across outcome domains (McClelland, Cameron, Connor, Farris, Jewkes, & Morrison, 2007; Raver, Jones, Li-Grining, Zhai, Bub, & Pressler, 2011). Research also indicates that early intervention is particularly important for children in poverty, who are more likely to be exposed to a range of factors that place their chances for school success at substantial risk and for whom gaps in school readiness exist at kindergarten entry (e.g., Princiotta & Germino-Hausken, 2006).

What distinguishes this new generation of preventive interventions and their evaluations is their increasingly sophisticated and fine-grained multi-level theories of change, in which processes at multiple ecological levels are targeted (e.g., individuals and social settings such as classrooms). Such advances in the use of developmental-ecological theory, epidemiological data, and small-scale, highly controlled efficacy studies to articulate a coherent multi-level logic model are at the core of prevention science (Kellam & Langevin, 2003). Using such models as a guide to unpacking the black box of intervention effects by testing process mediators is a critical step for identifying the key features of programs that are likely to promote successful replication and sustainability, as well as for supporting program improvement efforts (Coie et al., 1993). However, there are relatively few empirical examples that move beyond reporting direct effects of the intervention to conduct a comprehensive test of the intervention’s theory of change. This study builds upon several reports of the direct effects of the Chicago School Readiness Project (CSRP) on children’s developmental outcomes to examine a sequence of process mediators that are core to the intervention’s theory of change. Consistent with the CSRP program theory, we address the broad question, is the intervention’s influence on children’s academic and behavioral outcomes carried in sequence through its impact on the quality of teacher-child relationships and children’s regulatory skills?

In the following pages we briefly review the CSRP program theory and its effects on children reported to date. We then review literature on the links between the quality of teacher-child relationships, children’s regulatory skills, and their academic and behavioral outcomes.

Chicago School Readiness Project: Program Theory and Impacts

It has been well documented that low income, minority and English Language Learner (ELL) students in the United States are at increased risk for academic failure and that disparities emerge early in life. Not only do children enter school with meaningful individual differences in language and literacy skills, accumulating evidence documents that disadvantaged children are not entering school with the social, behavioral, communication, and self-regulatory skills necessary to succeed (e.g., Aber, Jones, & Cohen, 2000; Evans & English, 2002). While biological factors likely contribute to some degree to these differences, there is evidence that socialization and educational experiences play a central role in their development (Blair, 2002; Fantuzzo, Bulofsky-Shearer, McDermott, McWayne, Frye, & Perlman, 2007).

With these patterns as backdrop, the CSRP was designed to support low-income children’s development of optimal self-regulation by fostering the development of emotionally close, positive relationships with teachers. Within the context of relatively few clinical resources and limited staffing support, early childhood classrooms may become chaotic and difficult to manage as children with more emotional and behavioral difficulty engage in escalating, emotionally dysregulating coercive processes with teachers (e.g., Arnold, McWilliams, & Arnold, 1998; Kellam Ling, Merisca, Brown, & Ialongo, 1998). Few teachers report receiving pre-service training in handling children’s disruptive behaviors and one concern is that chronic engagement in escalating cycles of conflict might exacerbate children’s acting-out, disruptive behavior, as well as lead to teachers’ rising feelings of exasperation, disengagement and burnout (Brouwers & Tomic,1998).

Based on this theoretical framework, CSRP provided teachers with intensive training in strategies that they could employ to provide their classrooms with more effective regulatory support and better classroom management (Raver et al., 2008; Webster-Stratton, Reid, & Stoolmiller, 2008). Additional components of ongoing classroom-based and child-focused consultation were provided by a mental health consultant (MHC) who supported teachers to try new techniques learned in the teacher training (Donohue, Falk, & Provet, 2000; Gorman-Smith, Beidel, Brown, Lochman, & Haaga, 2003). One critique might be that MHCs bring an extra pair of hands to the classroom in addition to their clinical expertise. To control for improvements in adult-child ratio introduced by the presence of MHCs in intervention classrooms, control group classrooms were assigned a lower-cost teacher’s aide for the same amount of time per week.

To date, several reports have described the direct impacts of the CSRP intervention on classroom- and child-level outcomes at the end of the Head Start preschool year. For example, there are positive effects of the CSRP on classroom levels of positive climate, teacher sensitivity, effective behavior management, and reductions in negative climate, as well as on children’s (a) executive functions and self-regulation, (b) internalizing and externalizing behavior, and (c) pre-academic skills including basic vocabulary, letter naming, and math (Raver et al., 2008, 2009, 2011). This paper is intended to bring this work together by examining the degree to which the intervention’s impacts on both behavior and academic skills is explained by a sequence of effects that move from intervention to the quality of teacher-child relationships, to children’s regulatory skills, and then to their behavioral and academic outcomes.

Teacher-Child Relationship Quality

Because most children spend a large portion of their waking hours in preschools and schools, their daily transactions with teachers and other children powerfully influence their development. Recent research has demonstrated clear links between these interactions and children’s developmental outcomes (e.g., Hamre & Pianta, 2005; Jones, Brown, & Aber, 2008). Children’s relationships with their teachers have been viewed as a social-emotional source of provisions that either help or hurt children’s chances of doing well in school (Ladd, Birch, & Buhs, 1999; Pianta, 1999). The opportunity to build emotionally close relationships with nurturing adults in preschool environments has been hypothesized to be particularly important for low-income children who may experience greater instability in other areas of their lives (e.g., Owen, Klausli, Mata-Otero, & Caughy, 2008; Silver, Measelle, Armstrong, & Essex, 2005).

Research on the quality of emotional relationships among teachers and children associates positive classroom climates with greater self-esteem, perceived cognitive competence, internal locus of control, mastery motivation (Ryan & Grolnick, 1986), school satisfaction (Baker, 1999), academic performance, and less acting-out behavior (Toro, 1985); while more negative, conflictual classroom relationships have been associated with poor peer relations, poor academic focus, and higher levels of aggression (e.g., Jones et al., 2008). Teachers play a key role in these interactions. For example, teachers who practice good listening skills (e.g., direct eye contact, paraphrasing, acknowledging comprehension) in their interactions with children and who are able to teach these skills and provide real-life, real-time examples increase the chances that children will employ them in their interactions with others. But it is not merely the practice of good listening skills by the teacher or any given child that is important; it is how the use of these skills reflects a set of transactional social processes enabling teachers and children to develop closer, more intimate relationships and a more responsive classroom overall (Tseng & Seidman, 2007).

Warm, emotionally positive relationships with teachers have been found to predict children’s later academic engagement, liking of school, attendance, and higher academic achievement (Birch & Ladd, 1998; Entwistle & Alexander, 1999). In contrast, children who engage in high levels of conflict with teachers develop more negative attributions (or internal working models) about teachers as reliable and trustworthy sources of support, about themselves as learners, and about the process of learning (Hamre & Pianta, 2005). More recent work suggests that the quality of teacher-child relationships may operate on children’s academic functioning by modulating children’s stress reactivity. For example, first-grade children with more conflictual relationships with their teachers were significantly less able to down-regulate stress compared to their peers with more emotionally supportive relationships (Ahnert, Harwardt-Heinecke, Kappler, Eckstein-Madry, & Milatz, 2012).

Given this correlational evidence, facilitating positive relationships between teachers and low-income children represents an important avenue through which intervention might support social-emotional, behavioral, and pre-academic domains of young children’s school readiness. When they feel emotionally closer to their students, teachers may invest more time in providing explicit instruction, they may use classroom time more productively to cover more cognitively engaging material, and they may provide more contingent feedback to their students regarding performance (Hamre & Pianta, 2005; Torgesen, 2002). Longitudinal analyses suggest that teachers’ perceptions of emotionally close versus conflictual relationships with students account for small but statistically significant proportions of variance in children’s academic competence over time, even after children’s earlier academic competencies have been controlled (Pianta & Stuhlman, 2004). Higher levels of teacher-child closeness in early elementary school also appear to play a strong protective role for children who are at higher academic risk (Burchinal, Peisner-Feinberg, Pianta, & Howes, 2002).

Self-Regulation

Children’s regulatory skills have clear consequences for their success in school. Research indicates that children who have difficulty regulating emotions and who therefore experience high levels of negative emotional arousal have trouble concentrating in class and recalling things they have learned (Raver et al., 2007). In addition, to the extent that the regulatory skills of students in a particular classroom help or hinder a teacher’s efforts to manage the classroom effectively and deliver high-quality instruction, they may indirectly influence language and literacy development (Jones et al., 2008).

The last decade has witnessed a great deal of research on the development of children’s emotional, behavioral, and cognitive regulatory skills. For the purposes of the present study, we anchor our brief discussion of children’s emotional and behavioral self-regulation in two domains of functioning: children’s executive functioning (EF) and their effortful control (EC). Executive functioning is often defined as a set of cognitive processes that help children remain on task and goal-directed, including: attentional control, the use of working memory, cognitive flexibility, and the ability to “inhibit prepotent responding” (also defined as inhibitory control, Blair, Zelazo, & Greenberg, 2005, p. 561). Attentional control is the ability to selectively focus on appropriate, goal-directed tasks. Working memory refers to the ability to retain and manipulate information over a relatively short period of time. Cognitive flexibility is defined as the ability to shift attentional or cognitive set among distinct but related dimensions of a task, such as considering whether two objects are similar according to two different attributes (e.g., color and shape; Zelazo & Muller, 2002). These elements of EF have been linked to academic competence in preschool, kindergarten, and the early grades (e.g., Blair & Razza, 2007).

There is considerable overlap among definitions and measures of EF and definitions of EC. Effortful control is often defined as “the ability to inhibit a dominant response to perform a subdominant response” (Rothbart & Bates, 1998, p. 137; Calkins & Fox, 2002) and refers to young children’s ability to control cognition and behavior in the service of specific goal-directed (i.e., motivated) activities. In general, EC (also described as self-control) involves the basic acts of self-monitoring and inhibiting inappropriate responses to social or contextual demands (Shonkoff & Phillips, 2000). Links between regulatory capacity (EC and EF) become clear in studies indicating that, when an individual is negatively emotionally aroused, or positively aroused and too focused on a specific goal, attention becomes inflexible, behavior is persistent, higher order thinking is shut down, and more reactive and less thoughtful responses to stimulation are activated (Arnsten, 2000).

Children with difficulties in each of these regulatory domains are likely to lose out academically in a number of ways. First, teachers provide disruptive children with less positive feedback, so disruptive children spend less time on task and receive less instruction (e.g., Arnold et al., 1998; Shores & Wehby, 1999). Second, emotionally negative, angry children may lose opportunities to learn from their peers as children gather to work on projects together, and provide each other with support and encouragement in the classroom (Berndt & Keefe, 1995). Third, children who are disliked by teachers and classmates grow to like school less and avoid school more often (Birch & Ladd, 1997).

Bringing these lines of research together, this paper employs structural equation modeling techniques to consider the degree to which teacher-child relationship quality and children’s regulatory skills serve as plausible mechanisms operating inside the black box of the CSRP intervention’s impact on preschoolers’ school readiness. Building directly on the primary impacts of the CSRP on children’s developmental outcomes described above, we address the following specific questions (see Figure 1): (1) To what degree do the effects of CSRP on children’s self-regulation operate through the quality of teacher-child relationships? (2) To what degree do the effects of CSRP on children’s academic and behavioral outcomes operate through teacher-child relationship quality and children’s regulatory skills?

Figure 1.

Path model depicting the hypothesized processes by which the CSRP intervention affects children’s pre-academic skills and problem behavior measured in the spring of their Head Start Year.

Method

School and Subject Selection in the CSRP Evaluation

In an effort to balance generalizability and feasibility, preschool sites for the CSRP evaluation study were selected on the basis of (a) receipt of Head Start funding, (b) having two or more classrooms that offered “full day” programming, and (c) location in one of seven neighborhoods that were selected on the basis of a set of criteria including high poverty, exposure to high crime, and lower rates of mobility (see Raver et al., 2008 & 2009 for a detailed discussion of exclusionary criteria, the site selection process, and the overall evaluation design). Eighteen sites completed the site selection and recruitment process and were included. Two classrooms within each site were randomly selected for participation, with a research coordinator and research staff successfully able to recruit 83% of the children enrolled in selected classrooms. Teacher reports of the quality of their relationship with each of their CSRP-enrolled students were collected for the full sample in spring (March) of the school year. In addition, direct assessments of children’s self-regulation and pre-academic skills were collected for a large proportion of the full sample in fall and spring of the Head Start year (N = 467).

Randomization

Each site was matched with another “sister” site that it most closely resembled on a range of demographic characteristics of families and site characteristics indicating program capacity and quality (see Raver et al., 2008 for details). One member of each pair of sites was then randomly assigned to the intervention and the other member of the pair was assigned to the control group. Within each of the 9 intervention sites, 2 classrooms participated, for a total of 18 intervention classrooms. Across the 9 control sites, there were 17 classrooms (2 classrooms in 8 sites, and 1 classroom in the remaining site which lost one Head-Start funded preschool classroom due to funding cuts). Intervention classrooms received the multiple components of the CSRP package across the school year, and control classrooms were paired with teaching assistants as described above. Additional information on CSRP implementation is reported in Raver et al., 2008.

Sample

The CSRP intervention was implemented for two cohorts of children and teachers, with Cohort 1 participating from fall to spring in 2004-05 and Cohort 2 participating from fall to spring in 2005-06. As with other recent efficacy trials implemented with multiple cohorts across time, regions, or racially segregated neighborhoods, the sites enrolled in Cohorts 1 and 2 differed on several program-level and demographic characteristics, and therefore those characteristics were included in all analyses. Because we planned to model child outcomes as potentially responsive to both the intervention and to teacher- and classroom-level characteristics, teachers were also included as research subjects. As with classrooms, teachers were enrolled in two cohorts which were also pooled into a single dataset (n = 90). A total of 87 teachers participated in CSRP at baseline. The number of teachers increased to 90 by the spring of the Head Start year. This net increase reflected the entry of 7 more teachers and the exit of 4 teachers who either moved or quit during the school year.

At baseline, a total of 543 children participated in CSRP. By the spring, the number of participating children was reduced to 509. Nearly all of the exits were due to children voluntarily leaving the Head Start program, though one child was asked to leave the Head Start program and one child was withdrawn from participating in CSRP by a parent. Of these participants, 273 (54%) were female. Children ranged in age from 26 months to 73 months (age M = 49.4, SD = 8.0). The majority of participants were African American (67%) and Hispanic/Latino (25%) while the remaining participants were Caucasian (3%) or of other ethnic groups (5%). Families were predominantly low income with a mean yearly income of $13,440. Most families (63%) had only one parent living in the home.

The sample for analyses of teacher-report data in this study included 467 children who had complete data on child and family background characteristics in the fall as well as valid data on measures of pre-academic skills, behavior problems, self-regulation, and teacher-child relationship quality. Representativeness analyses revealed no significant differences between children who were in the analytic sample and those children who were excluded due to missing data. Similarly, no significant differences were detected between teachers who were in the analytic sample and those teachers who were omitted due to children’s missing data.

Procedures

In the fall, families with children ages 3-4 were recruited from each of the 35 classrooms to participate in the study, with approximately 17 children in each classroom enrolling in CSRP. Consent forms for each child were signed by his or her parent or guardian, who also completed an interview, which included questions regarding child and family socio-demographic characteristics. We collected data on children’s behaviors, their demographic characteristics, and classroom and site characteristics from five sources: parents, teachers, classroom observers, children themselves, and site directors.

Children’s self-regulatory skills and pre-academic skills were collected individually from each child by a multiracial group of master’s level assessors who had been extensively trained and certified in direct assessment procedures (see Goyette et al., 2006; Smith-Donald, Raver, Hayes, & Richardson, 2007 for details of training and certification of assessors). For children’s self-regulation, data on CSRP-enrolled children’s performance on six self-regulation tasks were collected using the Preschool Self-Regulation Assessment (PSRA; Smith-Donald et al., 2007). In addition to the PSRA, we also collected a cognitively-oriented and federally-mandated assessment of Head Start-enrolled preschoolers’ vocabulary, letter naming, and math skills (Zill et al., 2003) as well as assessors’ global ratings on the PSRA Assessor Report (Smith-Donald et al., 2007). Assessments were conducted with children in quiet areas of their Head Start programs during the school day. Twenty percent of all assessments were videotaped and were double-coded by a trained assessor to establish inter-rater reliability.

In the spring of the Head Start year teachers were given the Student-Teacher Relationship Scale (STRS; Pianta, 2001), measures of children’s behavior problems, as well as a brief survey of teacher demographic information and past teaching experience. CSRP research staff dropped off questionnaire packets to teachers with prepaid postage envelopes for easy return, making follow-up phone calls and visits to collect teacher reports within a 6-week window. Teachers were subsequently reimbursed a nominal subject payment (of $20 per packet) for completing questionnaires during each round of assessment.

To account for classroom-level differences in resources and support for children’s social-emotional development, both trained observers and teachers provided classroom-level data in the fall. Trained observers, who were blind to randomization, assessed the quality of children’s classrooms using the Classroom Assessment Scoring System (CLASS; La Paro, Pianta, & Stuhlman, 2004). In the fall, administrators at each Head Start site also provided CSRP with access to site-level characteristics.

Measures

Pre-Academic Skills

Children’s comprehension of spoken English was assessed prior to cognitive testing by playing a game of “Simon Says” (α=.92) (PreLAS Simon Says; Duncan & DeAvila, 1998). The assessor played the role of “Simon” and directed the children to act out ten simple actions only when the assessor prompted the child with “Simon says.” Throughout this game, the assessor gauged how well the child understood spoken English. Children were screened in both September and May of their Head Start year. If children speaking Spanish and English passed this English screener, they were assessed twice, first in Spanish and second in English. We then compared each child’s scores based on the Spanish and English assessments and used the child’s highest score in analyses. Children who spoke English only were assessed one time in English.

Letter naming

Children were assessed for their knowledge of the alphabet during September and May of their Head Start year using the letter-naming portion of the cognitive development assessment (Zill, 2003). This test consists of the 26 letters of the English alphabet divided into three groups consisting of 8, 9, and 9 letters each (note that there were 30 letters for the Spanish assessment divided into three groups of 10). The letters are arranged in approximate order of item difficulty in both their capital or lower case forms. Because the English language assessment has 26 items and the Spanish language assessment has 30 items, the scores were calculated in terms of total percent correct out of 26 or 30, respectively (α=.92).

Mathematics

Children’s early math skills were assessed in September and May of their Head Start year using the early math skills portion of the cognitive development assessment (Zill, 2003). This test consists of 19 items that cover basic addition and subtraction (α =.82).

Vocabulary

Children’s picture/word vocabulary was assessed during September and May of their Head Start year using a 24-item version of The Peabody Picture Vocabulary Test (PPVT-III; Dunn & Dunn, 1997; Zill, 2003). This test is administered to the child by the assessor. Children are asked to identify (point to) the one picture out of a group of four that corresponds to the word spoken by the assessor (α=.78). A parallel Spanish-language version of the PPVT, entitled the Test de Vocabulario en Imagenes Peabody (TVIP; Dunn, Lugo, Padilla, & Dunn, 1986) was administered for Spanish-proficient and bilingual children (see above).

Behavior Problems

In both the fall and spring, teachers completed the Behavior Problems Index (BPI), a 28-item rating scale originally designed for parent report of child behavior and adapted from multiple studies of children’s behavior problems (Zill, 1990). CSRP modified the original version in several minor ways. For the purposes of this study, items were summed into Internalizing (alpha in spring = .80) and Externalizing (alpha in spring = .92) subscales.

Mediators

Self-Regulation

The PSRA (Smith-Donald et al., 2007) was used to capture children’s behavioral self-regulation in the late spring (May) of their Head Start year. The PSRA includes two components, where an assessor first administers tasks to a child and then completes the PSRA Assessor Report. The PSRA tasks were selected because they were brief, required few materials, and yielded useful data for 3- to 5-year-old children using lab-based protocols. For this paper, four delay tasks were included, and were adapted from the lab-based work of Kochanska and colleagues to tap children’s EC: Toy Wrap, Toy Wait, Snack Delay, and Tongue Task (see Murray & Kochanska, 2002). Two additional tasks were included as markers of EF, including Balance Beam (Maccoby, Dowley, Hagen, & Degerman, 1965) and Pencil Tap, which was adapted from the peg-tapping task (Diamond & Taylor, 1996). Assessors live-coded latencies or performance levels for each task. Children’s performance on the four EC tasks and on the two EF tasks were standardized and then averaged into two composites. Assessors were trained extensively and were required to pass three levels of testing before certification. In addition, inter-rater reliability was calculated from double-coded videotaped assessments for 20% of the sample. The consistency of the assessor and coder responses on those forms was evaluated for all continuous variables, and Cronbach’s alphas ranged from .73 - .99 across all PSRA tasks with an average alpha of .93.

Teacher-Child Relationship

To assess the impact of the CSRP intervention on children’s relationships with their teachers, The Student-Teacher Relationship Scale (STRS; Pianta, 2001) was completed by teachers in the early spring (March) of the Head Start year. The STRS is a 28-item self-report instrument that uses a 5-point Likert-type rating scale to assess a teacher’s perceptions of his or her relationship with a student, a student’s interactive behavior with the teacher, and a teacher’s beliefs about the student’s feelings toward the teacher. For brevity, the CSRP retained 15 of the original 28 items; however, these items were not altered. This measure has a high internal consistency (α= .89 to .95) as well as good convergent validity with other questionnaire measures of anger and anxiety.

Baseline Child, Family, Teacher/Classroom, and Site-Level Characteristics

Child- and family-level demographic characteristics assessed at baseline (September) were included in all analyses. These included (a) child gender, (b) child membership in the race/ethnic category of African American versus Hispanic, (c) family’s cumulative exposure to three poverty-related risks (including mothers’ educational attainment of less than a high school degree, family income-to-needs ratio for the previous year being less than half the federal poverty threshold, and mothers’ engagement in 10 hours or fewer of employment per week) (Raver, 2004), (d) single-headed household, (e) large family size (four or more children), and (f) parent’s self-identification as Spanish-speaking in the home. Children’s scores at baseline collected in fall of the Head Start year were also included as control variables for the corresponding outcome measures in the spring.

A set of teacher/classroom characteristics, assessed through teacher report at baseline, was also included as a proxy of classroom quality. These included teachers’ reports regarding their level of education (teacher’s attainment of a BA as well as teacher aide’s attainment of at least some college), as well as their report on several psychosocial characteristics that might affect teachers’ perceptions of children’s behavioral difficulty (see Anthony, Anthony, Morrel, & Acosta, 2005). To assess their psychosocial characteristics, teachers’ depressive symptoms were assessed at baseline using the 6-item K6, a scale of psychological distress developed for the U.S. National Health Interview Survey (Kessler et al., 2002). With a metric of 0 to 4, the K6 items were summed (α = .65). In addition, teachers reported job overload on the 6-item “job demands” and 5-item “job control” subscales of the Child Care and Early Education Job Inventory, which had a rating scale of 1 to 5 (Curbow, Spratt, Ungaretti, McDonnell, & Breckler, 2000). Subscales demonstrated adequate internal consistency (α = .67 and α = .56, respectively) and were based on the sum of each set of items. To calculate classroom-level covariates, scores on each variable were averaged across all teachers in each classroom. To control for additional variation in classroom quality, an observational measure was collected in fall using the CLASS (La Paro et al., 2004). The CLASS indicator used in the current analyses was a 7-point Likert score on the negative emotional climate scale (see Raver et al., 2008). Three-quarters of the observations were double coded “live” by two observers and intraclass correlation values (α) indicated adequate to high levels of inter-observer agreement (α values ranging from .66 to .87 for all classroom observational measures). In order to test the role of setting-level program characteristics, a limited number of site-level covariates were entered into models, including the availability of a full-time family worker at the Head Start site, the proportion of the teachers with BA degrees and the proportion of TAs with some college, and the proportion of the families served who were employed.

Analysis Plan

Means, standard deviations, and intercorrelations among primary study variables were examined in preliminary analyses. To investigate the processes by which the CSRP intervention affects children’s academic and behavioral outcomes, we used structural equation modeling in MPlus. Specifically, we fit a series of path models in which we examined the indirect effects of the intervention on children’s pre-academic skills (i.e., letter naming, mathematics, and vocabulary) and problem behavior (internalizing and externalizing) through the quality of the teacher-child relationship and children’s self-regulatory skills (see Figure 1). In the center of the figure we display an observed index of the overall quality of the teacher-child relationships as well as a measurement model linking observed indices of EF and EC to the hypothesized latent construct representing self-regulation skills in the spring of children’s Head Start year. Note that the factor loading for EF is fixed to a constant value of 1 to provide the metric for interpretation. On the far left side of the path model, we display a single observed variable representing intervention status (1=intervention, 0=control) and on the far right side of the path mode we display an observed variable representing children’s pre-academic skills or problem behavior measured in the spring of their Head Start year. The hypothesized direct effects of intervention on teacher-child relationship quality, of teacher-child relationship quality on self-regulation, and of self-regulation on pre-academic skills or problem behavior are depicted by solid black arrows and each pathway is labeled with a corresponding structural regression parameter (γ₀₁, γ₀₂, etc.).

To examine our theory describing a sequence of process mediators as depicted in Figure 1, we began by examining the indirect effects of the intervention on self-regulation through overall teacher-child relationship quality (referred to as Model 1 in Figure 1). Next, to determine whether the CSRP intervention impacts children’s pre-academic skills and behavior problems by improving the quality of the teacher-child relationship and subsequently children’s self-regulatory skills, we added to our model constructs representing children’s pre-academic skills (Table 2) or problem behavior (Table 3) measured in the spring of their Head Start year (referred to as Model 2 in Figure 1). Finally, to investigate the robustness of our findings, we fit a set of lagged autoregressive models that included children’s pre-academic skills or problem behavior measured in the fall of their Head Start year (referred to as Model 3 in Figure 1). Note that in these models, our estimates of the impacts of CSRP on children’s later outcomes are very conservative.

Table 2.

Estimated Indirect Effects of Treatment on Children’s Pre-Academic Skills through Student-Teacher Relationship Quality and Self-Regulation Skills.

	Letter Naming		Mathematics		Vocabulary
	1	2	3	4	5	6
Indirect Effects
TX → STRS	.159** (.057)	.159** (.057)	.160** (.057)	.161** (.057)	.160** (.057)	.160** (.057)
STRS → Self Regulation	.266*** (.061)	.302*** (.067)	.254*** (.055)	.319*** (.061)	.272*** (.060)	.360*** (.072)
Self Regulation →Pre-Academic Skills	.760*** (.127)		.868*** (.112)		.817*** (.123)
Self Regulation →Pre-Academic Skills + T1		.429*** (.087)		.561*** (.088)		.526*** (.096)
Total Indirect Effects	.032* (.014)	.015* (.007)	.035* (.015)	.029* (.012)	.035* (.015)	.030* (.013)
Model Fit Statistics
χ2 Statistic (df)	110.51*** (40)	173.16*** (43)	103.24*** (40)	258.73*** (43)	100.04*** (40)	201.84*** (43)
CFI	.853	.823	.885	.744	.869	.774
RMSEA	.054 (NS)	.071 (**)	.051 (NS)	.091 (***)	.050 (NS)	.078 (***)

Note: Coefficients presented in the table are the standardized coefficients; standard errors are presented in parentheses. Models 1, 3, and 5 represent the effects when fall scores are not controlled; Models 2, 4, and 6 represent the autoregressive models, controlling for fall scores. TX= treatment; STRS = student-teacher relationship quality; T1 = fall of Head Start year assessment.

^***p <.001,

^**p < .01,

^*p < .05,

⁺p < .10

Table 3.

Estimated Indirect Effects of Treatment on Children’s Problem Behaviors through Student-Teacher Relationship Quality and Self-Regulation Skills.

	Internalizing Behavior		Externalizing Behavior
	1	2	3	4
Indirect Effects
TX → STRS	.152** (.057)	.151** (.056)	.142* (.056)	.143* (.056)
STRS → Self Regulation	.521*** (.088)	.595*** (.096)	.709*** (.089)	.708*** (.090)
Self Regulation →Problem Behavior	-.415*** (.082)		-.585*** (.084)
Self Regulation →Problem Behavior + T1		-.423*** (.080)		-.565*** (.084)
Total Indirect Effects	-.033* (.014)	-.038* (.015)	-.059* (.024)	-.057* (023)
Model Fit Statistics
χ2 Statistic (df)	94.99*** (40)	113.69*** (43)	96.70*** (40)	126.17*** (43)
CFI	.878	.864	.898	.858
RMSEA	.048 (NS)	.052 (NS)	.048 (NS)	.057 (NS)

Note: Coefficients presented in the table are the standardized coefficients; standard errors are presented in parentheses. Models 1 and 3 represent the effects when fall scores are not controlled; Models 2 and 4 represent the autoregressive models, controlling for fall scores. TX= treatment; STRS = student-teacher relationship quality; T1 = fall of Head Start year assessment.

^***p <.001,

^**p < .01,

^*p < .05,

⁺p < .10

Models were fitted using MPlus version 5.1 All models controlled for the effects of a common set of child (i.e., child sex, race/ethnicity and total problem behavior), family (i.e., family poverty-related risks, single parent family, four or more children in the household, and Spanish speaking parent), teacher/classroom (i.e., BA, K6 scores, job demand, job control, negative emotional climate, and number of adults in the classroom), and site (i.e., family support worker on staff, percentage of teachers with a BA, percentage of assistant teachers with any college training, and percentage of families with at least one parent employed) level covariates. In addition, given that the intraclass correlations for our dependent variables tended to be higher between classrooms within sites than between sites (e.g., Raver et al., 2009), the nesting of children within classrooms was accounted for in all models. Missing data were handled using Full Information Maximum Likelihood estimation. Model fit was evaluated using three criteria: (1) Chi-square statistics, where a small, non-significant value is considered good fit; (2) CFI, where a value between .9 and 1 is considered good fit; and (3) RMSEA, where a value between 0 and .1 is considered good fit. Because chi-square statistics are highly sensitive to sample size, it is not common to obtain a non-significant value and thus the use of multiple fit indices to evaluate model adequacy is critical.

Results

Preliminary Analyses

Sample means and standard deviations for the outcome, key predictor and control variables for the total sample as well as by intervention status are presented in Table 1. Correlations among children’s pre-academic skills and problem behaviors measured in May of their Head Start year, quality of the teacher-child relationship measured in March, and children’s self-regulatory skills measured in May are available from the first author upon request. On average, mean levels of children’s pre-academic skills increased between the fall and spring while their mean level of problem behavior decreased somewhat. Children in the intervention group exhibited slightly higher pre-academic skills and more behavior problems than did children in the control group. There were few differences between the intervention and control groups on any of the mediating variables.

Table 1.

Descriptive Statistics for Outcome and Key Predictor Variables as well as Child, Family, Teacher/Classroom, and Site-Level Covariates for the Full Sample as well as by Treatment Status

	Full Sample	Treatment	Control
	Mean/% (SD)	Mean (SD)	Mean (SD)
Outcome Variables (spring, May)
Letter-Naming	.44 (.38)	.48 (.39)	.40 (.37)
Mathematics	9.80 (4.18)	9.88 (4.04)	9.7 (4.32)
Vocabulary	13.27 (4.44)	13.52 (4.43)	13.0 (4.44)
Internalizing Behavior	1.59 (1.98)	1.98 (2.19)	1.18 (1.65)
Externalizing Behavior	4.18 (4.57)	5.09 (5.16)	3.26 (3.65)
Baseline Scores (fall, September)
Letter-Naming	.22 (.30)	.24 (.31)	.19 (.29)
Mathematics	7.42 (3.82)	7.69 (3.53)	7.14 (4.08)
Vocabulary	10.54 (3.90)	10.80 (3.60)	10.27 (4.18)
Internalizing Behavior	2.25 (2.46)	2.49 (2.49)	2.01 (2.41)
Externalizing Behavior	5.76 (5.77)	6.18 (6.05)	5.32 (5.44)
Mediators (spring)
STRS Total Score (March)	62.49 (8.73)	61.99 (8.29)	62.99 (9.14)
Executive Functioning (May)	-.005 (.81)	.04 (.84)	-.06 (.76)
Effortful Control (May)	.001 (.686)	-.01 (.68)	.01 (.70)
Child Characteristics
Boys	46.7%	50.6%	42.5%
Race/Ethnicity (African American)	65.8%	66.5%	65.0%
Race/Ethnicity (Hispanic)
Race/Ethnicity (Other)	7.3%	6.2%	8.5%
BPI Total	6.33 (5.19)	6.67 (5.23)	5.99 (5.14)
Family Characteristics
Family Related Poverty Risks	1.09 (.99)	1.15 (.99)	1.03 (.98)
Single Parent Families	69.2%	71.2%	.67 (.47)
Four or More Children in HH	24.8%	24.0%	25.5%
Parent is Spanish Speaking	22.7%	20.8%	24.8%
Teacher/Classroom Characteristics
Teacher BA	63.6%	65.3%	61.9%
Teacher K6 Score	2.55 (2.01)	3.16 (1.63)	1.91 (2.16)
Teacher Job Demand	2.71 (.59)	2.88 (.624)	2.54 (.50)
Teacher Job Control	3.25 (.67)	3.33 (.68)	3.18 (.66)
Classroom Negative Emotional Climate	2.03 (.98)	2.17 (1.10)	1.89 (.81)
Number of Adults in Classroom	2.41 (.69)	2.53 (.79)	2.29 (.55)
Site Characteristics
Family Support Worker on Staff	1.17 (2.25)	40.9%	1.96 (2.98)
Proportion of Teachers with BA	43.7%	50.6%	36.5%
Proportion of Teachers with any College	48.9%	36.7%	61.8%
Proportion of Families Employed	74.1%	80.9%	66.9%

What Are the Processes by which the CSRP Intervention Affects Children’s Pre-Academic Skills and Problem Behavior?

To investigate the hypothesized sequence of mediating processes by which the CSRP intervention impacts children’s pre-academic skills and problem behavior measured in the spring of their Head Start Year, we began by investigating whether children’s self-regulatory skills improved by increasing the quality of the teacher-child relationship. Controlling for an extensive set of child, family, teacher/classroom, and site-level characteristics, the CSRP intervention was a positive and statistically significant predictor of the overall quality of the teacher-child relationship (β = .157, p < .01). That is, teachers of children who were in sites randomly assigned to receive the CSRP intervention reported more positive relationships with study children than did teachers in the control group. In turn, when the quality of the teacher-child relationship was better, children demonstrated significantly greater self-regulatory skills (β = .270, p < .001). A test of the indirect effects of the intervention on children’s self-regulatory skills through the teacher-child relationship revealed that the effect was marginally significant (i.e., indirect effect = .030, p < .10). Model fit statistics provided somewhat mixed evidence about how well our model fits the data. The relatively low χ² statistic (90.34 (38), p < .001) with its df to χ² nearing 2, and the reasonable RMSEA (.048), provide evidence that the model fits the data moderately well. The relatively low CFI statistic (.795) indicates the fit is somewhat poor.

We next examined the indirect effects of the CSRP intervention on a range of pre-academic skills (Table 2) and problem behaviors (Table 3) measured in the spring of children’s Head Start year. Specifically, we added to our structural equation model the pathway from children’s self-regulatory skills to their pre-academic skills (i.e., either letter naming, mathematics, or vocabulary) or problem behavior (i.e., either internalizing or externalizing behaviors). Models that do not take into account children’s earlier skills are presented in columns 1, 3, and 5 in Table 2 and columns 1 and 3 in Table 3. Note that the findings for the mathematics model are also displayed in Figure 2. On average, children who exhibited better self-regulatory skills also demonstrated better letter naming, mathematics, and vocabulary skills as well as fewer internalizing and externalizing behaviors than did children with poor self-regulatory skills, even after taking into account the effects of the intervention on the quality of the teacher-child relationship as well as the effects of the relationship quality on children’s self-regulatory skills. That is, net of child, family, teacher/classroom, and site level covariates, children who received the CSRP intervention had better relationships with their teachers and subsequently better self-regulatory skills; these improved skills led to better pre-academic and behavioral outcomes. Among the cognitive outcomes, the association was strongest for mathematics performance (γ = .868, p < .001) and among the behavioral outcomes the association was strongest for children’s externalizing problems (γ = -.585, p < .001). To test the robustness of our findings, we refit the models and included the fall assessments of children’s pre-academic skills or problem behavior as controls. Results from these models are presented in columns 2, 4, and 6 in Table 2 and columns 2 and 4 in Table 3. Although the magnitude of the associations between self-regulation and children’s spring outcomes were considerably smaller when we controlled their pre-test scores, the effects remained statistically significant across all outcomes, suggesting that these findings are quite robust given the conservative nature of this model.

Figure 2.

Path model depicting the processes by which the CSRP intervention affects children’s mathematics skills measured in the spring of their Head Start Year. Coefficients outside parentheses are the standardized estimates not controlling for fall scores. Coefficients in the parentheses are the standardized estimates controlling for fall scores.

Tests of the indirect effects of the intervention on children’s pre-academic and behavioral outcomes through teacher-child relationship quality and self-regulation revealed that these effects were statistically significant. The CSRP intervention appears to raise children’s letter naming (indirect effect = .032, p < .05), mathematics (indirect effect = .035, p <.05), and vocabulary (indirect effect .035, p < .05) skills as well as decrease their internalizing (indirect effect = -.033, p < .05) and externalizing (indirect effect = -.059, p < .05) behaviors via improved teacher-child relationship quality and self-regulatory skills of the children. Note that the indirect effects of the CSRP intervention on children’s outcomes were present regardless of whether we controlled children’s early skills and behaviors (see columns 2, 4, and 6 in Table 2 for pre-academic skills and columns 2 and 4 in Table 3 for problem behaviors). Fit indices suggest that model fit was reasonable, though not optimal. Not surprisingly, model fit was somewhat better for the models that did not include fall pre-test scores.

As a final step in understanding the processes by which the CSRP intervention impacts children’s subsequent outcomes, we re-fit the path models described above and examined a contrasting hypothesis. Specifically, we asked whether the intervention services led to better self-regulatory skills in children, which in turn led to higher quality teacher-child relationships, higher pre-academic skills and lower problem behavior. We found no evidence to support this pathway. That is, net of child, family, teacher/classroom, and site-level characteristics, there was no significant effect of the intervention on self-regulation or of self-regulation on the quality of the teacher-child relationship. Moreover, model fit was considerably poorer for this specification.

Discussion

We conducted a direct examination of the core theory of change of an early childhood intervention that has in prior reports shown direct and positive impacts on children’s social-emotional, behavioral, and academic outcomes, making it an important potential candidate for replication and scaling. As noted above, the CSRP was designed to support low-income children’s development of optimal self-regulation, and ultimately their positive behavior and academic skills, by fostering the development of emotionally close, positive relationships with teachers. To begin, we found support for the central pathways of the CSRP theory of change by showing intervention impacts on the quality of teachers’ relationships with their students, with intervention-assigned teachers reporting closer and less conflictual relationships with the children in their classrooms than control group-assigned teachers. It is important to note that these benefits of intervention were found net of the large number of classroom and site-level factors (including teachers’ own experiences of feeling overwhelmed by the stressors associated with their jobs) that might bias adults’ perceptions of child behavior. These findings are in keeping with the findings showing improvements in independent observations of teachers’ sensitivity in managing their classrooms in CSRP-assigned sites relative to controls, and with improvements found in teachers’ use of emotion coaching, overall classroom management, and behavioral support in other school readiness interventions (Domitrovich et al., revised and resubmitted; Pianta, Mashburn, Downer, Hamre, & Justice, submitted).

Second, we found that the quality of teacher-child relationships mediated the intervention’s effects on children’s self-regulation. Importantly, a test of the opposite path, from self-regulatory skills to quality teacher-child relationships, was not supported by the data. This finding is consistent with the CSRP program and operational theory in which support in classroom management was provided directly to teachers through training and in-class implementation assistance from mental health consultants. From a developmental standpoint, these findings highlight the modifiability of the quality of adults’ relationships with young children, as reported by the adults themselves. These findings also highlight the modifiability of low-income preschoolers’ ability to marshal their attention and impulsivity, and the potential of a comprehensive set of structured environmental supports to shape young children’s development of effective self-regulation (Pears & Fisher, 2005; Pollack, 2003; Riggs, Greenberg, Kusche, & Pentz, 2006).

Third, we found that teacher-child relationship quality and children’s self-regulation mediated the intervention’s effects on children’s behavioral and pre-academic outcomes. Our findings indicate that CSRP services in which a set of classroom-based processes were targeted directly, resulted in improvements in children’s self-regulation, which in turn led to statistically significant increases in low-income preschoolers’ vocabulary, letter-naming, and early math skills, as well improvements in their aggressive and internalizing behavior. These findings lend important support to the claims made in previous longitudinal, non-experimental studies that the social, emotional and behavioral contexts of young children’s early educational experiences matter for both their behavior as well as their opportunities to learn (Fantuzzo et al., 2007; Jones, Brown & Aber, 2011; McClelland et al., 2007; Raver, 2002).

Taken together, these findings suggest that the multi-component, teacher- and classroom directed CSRP intervention sets in motion a cascade of effects (Masten & Cicchetti, 2010) that begin with emotionally supportive, high quality teacher-child relationships, which nurture children’s emerging self-regulation, which in turn supports their classroom-based behavior, both academic and behavioral.

Implications

In an era of increasing economic uncertainty, it is gravely concerning that rates of poverty for young children are on the rise, and that the difference between low-income children’s school readiness and the readiness of their more affluent counterparts has grown larger. These economic and educational disparities represent a tremendous incentive for our field to find feasible means of supporting young children’s school readiness. To date, there is growing evidence for success in this endeavor. First, educational opportunities for children under the age of 6 are rapidly on the rise. As of 2009, 1.2 million young children were enrolled in public school pre-K classes, with the percentage of our nation’s 4-year-olds who were enrolled in state-funded pre-K rising from 14% to 30% from 2002 to 2009 (Barnett et al., 2009). The growth rate for preschool teachers from 2006-2016 is predicted to be 26%, compared to 12% for teachers overall (U.S. Bureau of Labor Statistics, 2007). Second, multi-level, multi-component early childhood interventions set in preschool contexts are showing positive effects across developmental domains. For example, evidence from “hybrid models” of cognitive and social-emotional curricula as well as teachers’ provision of emotion coaching and support result in substantial changes in children’s social-emotional, behavioral, and academic outcomes (e.g., Bierman et al., 2008).

As we consider how to take what has been learned from high-quality early childhood interventions and embed them in the growing sector of early childhood education, we are left with the essential question of what is necessary to maintain in a program when moving to scale or to a different setting with varying constraints and opportunities. Such questions are answered in part by examining the central processes or mechanisms that account for intervention effects on child outcomes. Not only does such a question address issues central to replicability and scaling of interventions, it also provides a test of the core theory guiding the design of the intervention itself, contributing not only to prevention science but to basic developmental science (Kellam & Langevin, 2003). This paper represents one such test.

Limitations

This study’s conclusions are constrained by several limitations. A primary limitation is that our examination of mediation in the context of this randomized, efficacy study is essentially non-causal as the mediating processes examined were not randomly assigned, and self-regulation and the pre-academic outcomes were measured at the same time (in May of the Head Start year). The quality of teacher-child relationships were measured in March, however, introducing some temporal sequencing to the mediators. We have tried to minimize this concern by examining an alternative specification (e.g., from intervention to self-regulation, to teacher-child relationship quality) that did not fit our data and revealed few statistically significant effects, and have conducted a set of conservative auto-regressive models in which baseline levels of our primary outcomes are controlled. A second limitation is that, despite the moderate correlations, to maintain consistency of this work with that published before (e.g., Raver et al., 2009; 2011), we decided not to combine the models across outcomes (i.e., one model for academic outcomes and one model for behavioral outcomes). While the degree of overlap among the dependent variables should temper our view of the findings somewhat (e.g., some of what we observe for vocabulary is accounted for by what we also observe for mathematics), we feel these findings indicate robustness of the mediating processes across varying indicators of the primary outcome domains (across indicators of pre-academic skills and across indicators of behavior problems). In addition, as noted in the Results, model fit was not optimal for all indices. We employed multiple indicators of model fit in order to have a more comprehensive view of fit given that each individual indicator is subject to its own biases, but the CFI was below .9 in every case. We should note, however that according to the CFI our models are between approximately 75% and 89% better than a null model. A final limitation is that this study’s external validity is constrained to some degree. We relied on the generosity of staff, teachers, and families at 18 Head Start sites in seven neighborhoods of concentrated economic disadvantage on Chicago’s South and West sides that were willing to be randomly assigned to the receipt of two different types of services (comprehensive, multi-component intervention vs. receipt of a teacher’s aide, one day a week).

With regard to future directions, building on the notion that there may be some domain specificity in the effects of EF and EC on academic versus behavioral outcomes that stem from the research traditions described in the Introduction, in future work we will examine differential associations between EF and EC and children’s academic and behavioral outcomes, respectively, building in additional data that allow us to examine more nuanced processes of change over the course of the Head Start year.