Individual Factors Associated with Professional Development Training Outcomes of the Head Start REDI Program

Abstract

This study examined factors associated with process and content outcomes of the training provided in the context of Head Start REDI (Research based Developmentally-informed), a preschool curriculum designed to enhance the quality of interactions (social-emotional and language-literacy) between teachers and children. REDI professional development included 4 days of training and weekly coaching. Data for 22 intervention teaching pairs (N = 44) were used in the study. With the exception of years of education and emotional exhaustion, distal teacher factors (i.e., professional characteristics, personal resources and perceptions of the work environment) were unrelated to end-of-year implementation fidelity while openness to consultation showed a significant association. The findings emphasize the importance of teacher engagement in the training process for program effectiveness.

Emphasis on the need for evidence-based programs has grown from both the research and policy perspectives as the demands for accountability of children’s academic outcomes intensifies. The passage of No Child Left Behind Act of 2001 (No Child Left Behind, 2002) and the Good Start, Grow Smart (U.S. Department of Agriculture, 2007) initiatives, combined with the expanding literature regarding effective interventions, has extended the emphasis on evidence-based interventions downward from elementary education to early childhood and brought it to the forefront as a potential way to improve Head Start (Lopez, Tartullo, Forness, & Boyce, 2000; U.S. Department of Health and Human Services [USDHHS], 2003; Yoshikawa & Knitzer, 1997). If Head Start is to improve children’s social-emotional and cognitive development, as well as prevent early problem behaviors, more research is needed to ensure that community programs are able to implement evidence-based interventions with quality.

Recent theory and empirical research underscores that the repeated interactions children have with teachers in the preschool setting are the critical process through which they develop the cognitive, behavioral, and social-emotional skills needed to enter school ready to learn (Howes et al., 2005; NICHD Early Child Care Research Network [ECCRN] 2002, 2004; Pianta, 2003; Pianta, Hamre, & Stuhlman, 2002; Shonkoff & Phillips, 2000). Consequently, teachers’ instructional practices and relationships with students have become important indicators of early childhood program quality (Mashburn & Pianta, 2006; Pianta, 2003; Shonkoff & Phillips, 2000; Vandell & Wolfe, 2000) and are a main focus of many evidence-based interventions. Given these trends, professional development efforts increasingly involve teachers learning specific instructional and relational practices associated with an intervention (Carnegie Corporation, 2007; Clifford et al., 2005; Educational Testing Service, 2004; Raikes, et al., 2006). This learning may take place within the formal education system, through credentials obtained by individuals or through the accreditation of early childhood education institutions (Maxwell, Field, & Clifford, 2006). More often, however, evidence-based interventions and their associated training models are introduced to teachers who are already in the field. In this context, training success can be measured in terms of both process and content outcomes. The process goals include establishing positive perceptions of the intervention and active engagement in the training activities while the content goal is the skilled use of the teaching practices associated with the intervention. From the program developer’s perspective, this professional development content outcome is equivalent to high quality program implementation (i.e., fidelity) so these terms are used interchangeable from now on in this manuscript.

As the use of evidence-based interventions expands, the need for empirical research on the factors associated with training engagement and high quality curriculum implementation increases. These issues are emerging as particularly important ones for Head Start, the premier federally sponsored program providing comprehensive early childhood services to almost a million poor children and their families each year (USDHHS, 2001). The recent Head Start Impact Study indicates small to moderate effects on several measures of cognitive development and small to no effects on social skills and behavioral problems (USDHHS, 2005), with earlier research indicating that Head Start’s impact on IQ and other cognitive measures fades out in the elementary grades (Zigler & Muenchow, 1992). These findings have motivated efforts to enhance the impact of existing Head Start programs. In this context, professional development training for existing Head Start teachers is recognized as a promising strategy to enhance the early learning experiences of low-income children (Klein & Knitzer, 2006).

Key Features of Professional Development Training: Content and Process

Training content

Several recent empirical trials have increased teacher support for children’s development through a combination of curriculum components and general teaching strategies (Dickinson & Sprague, 2001; Girolametto, Weitzman, Lefebvre, & Greenberg, 2007; Wasik, Bond, & Hindman, 2006; Webster-Stratton, Reid, & Hammond, 2001). Two dimensions of teaching behavior are particularly important for child outcomes and are often the target of these interventions. The first dimension centers on communication and includes the provision of a rich and responsive language context using play and conversation that challenges children by being just slightly beyond their current skill level but fits appropriately with the context of the ongoing activity (Nelson & Welsh, 1998). Children’s cognitive and social competence improves when teachers use rich and varied vocabulary, narratives, and decontextualized language (Beals, DeTemple, & Dickinson, 1994; Dickinson & Tabors, 2001; Snow, Burns, & Griffin, 1998). The second dimension centers on emotional and behavioral support and includes the provision of a positive and predictable classroom environment and interactions with teachers who are warm and sensitive to children’s needs (Arnold, McWilliams, & Arnold, 1998; Yates & Yates, 1990). When problems arise, children in these types of settings experience positive guidance rather than punitive discipline (Howes et al., 2005; Lamb, 1998; Love, Meckstroth, & Sprachman, 1997; Pianta, 2003; Vandell & Wolfe, 2000). Sensitive responding and positive behavioral management is associated with stronger cognitive and language skills in preschool (NICHD ECCRN, 2000a, 2000b).

Training process

Randomized trials are seldom employed to identify effective professional development training processes, but “best practices” are emerging. Effective professional development training is specific and targeted (Guskey, 2003), involves opportunities for practice with feedback in naturalistic contexts (Elmore, 2002; Putnam & Borko, 2000), and provides teachers with adequate time to reflect on their own practices, set goals, and self-evaluate (Bowman, Donovan, & Burns, 2000). High quality training extends beyond time-limited, in-service sessions conducted outside of the classroom and includes some form of on-going support (Noell, Witt, Gilbertson, Ranier, & Freeland, 1997).

Coaching is emerging as a promising professional development training strategy that integrates many of these best practices and leads to better outcomes (Haskins & Loeb, 2007; International Reading Association & National Association for the Education of Young Children, 1998; Joyce & Showers, 2002). Coaches are technical assistants who help teachers to understand the intervention, the mechanics of program delivery, appropriate ways to integrate the intervention with existing practices, and ways to resolve difficulties encountered when using the intervention (Dusenbury et al., 2007; Leach & Conto, 1999; Rose & Church, 1998). For example, elementary students whose teachers were coached in the implementation of a social-emotional curriculum experienced better outcomes (Aber, Brown, & Jones, 2003; Gorman-Smith, Beidel, Brown, Lochman, & Haaga, 2003); and early childhood teachers’ experiences of coaching and supervision were positively associated with their responsive involvement and engagement in practices that promoted emergent literacy skills (Howes, James, & Ritchie, 2003). The existing literature does clarify the mechanisms through which coaching has a positive effect on teacher behavior, but theory suggests that coaches serve multiple roles including that of model, technical assistant, facilitator, and emotional support (Ryan & Hornbeck, 2004). This suggests that successful coaching should be associated with positive teacher perceptions of the intervention and an open, collaborative teacher-coach relationship that is valued by the teacher.

Research indicates that teachers’ positive perceptions of the intervention, specifically its acceptability and positive impact, are important precursors of effective program implementation. Teachers are more likely to conduct a program with quality if its philosophy is congruent with their beliefs and fits with their personal teaching style (Han & Weiss, 2005; Ringwalt, et al., 2003). Teachers who do not see the value of fostering a specific skill in children may be more likely to skip or modify intervention activities or practices, even though those may be core parts of the program. Further, when an intervention is perceived as easy to understand, it is implemented more effectively (Dusenbury, Branningan, Falco, & Hansen, 2003; Goldman, 1994; Pankratz, Hallfors, & Cho, 2002). Teachers’ perceptions of the effectiveness or positive impact of the program are also important. Han and Weis (2005) argue that feedback provided by coaches in the context of on-going support plays a key role in teachers attributing successes to the intervention, which in turn motivates high quality implementation. Studies comparing mentoring with and without the provision of feedback confirm the importance of feedback in influencing teacher behavior (Joyce & Showers, 2002; Noell et al., 2005).

Professional development is a dynamic process that requires teachers to examine their own behavior and to remain open to suggestions provided by the coach. This type of engagement in training is more likely when coaches form supportive, cooperative partnerships with teachers that are characterized by open communication and effective problem solving. Such a relationship facilitates teacher motivation and enables the coach to provide technical assistance and work through any problems that are seen to undermine the teacher’s appropriate use of the intervention. Descriptive studies suggest that the support and encouragement provided by a coach is one of the essential ingredients that makes this process of professional development effective (Brooks, 1996; McCormick & Brennan, 2001).

Predictors of Professional Development Training Process and Content

Many studies have examined associations between teachers’ professional characteristics (education, training and experience) and global measures of early childhood program quality (Burchinal, Cryer, Clifford, & Howes, 2002; Early, Bryant, et al., 2006; Lee, Burkham, Ready, Honigman, & Meisels, 2006; Phillipsen, Burchinal, Howes, & Cryer, 1997), but only a handful include key dimensions of teacher-student interaction as outcomes or explore more proximal factors such as the role of personal resources (depressive symptoms, efficacy, exhaustion) or perceptions of the work environment (job satisfaction, organizational climate). Much less is known about whether these factors influence the professional development process that leads to high quality practice. Prevention researchers have examined individual predictors of implementation, but much of this work is theoretical and has focused on more distal factors (for reviews see, Domitrovich et al., 2008; Han & Weiss, 2005).

Professional characteristics

Research on the contribution of qualifications to teaching quality is not definitive (Haskins & Loeb, 2007), with most studies yielding only moderate associations with measures of teaching quality (Tout, Zaslow, & Berry, 2006). Participation in in-service trainings and workshops is associated with higher levels of teacher sensitivity and global ratings of classroom quality even after the effects of formal training (i.e., education degree, content and credentials) are taken into account (Burchinal et al., 2002; Epstein, 1993). Studies of Head Start teachers have shown that training is more strongly related to program quality than formal education (Epstein, 1999; Layzer, Goodson, & Moss, 1993; Love et al., 1997). The association between teaching experience and quality of care is unclear: some studies find a positive association (Burchinal, Roberts, Nabors, & Bryant, 1996; NICHD ECCRN, 2000b), others report no association (Bryant, Burchinal, Lau, & Sparling, 1994; Howes, Whitebook, & Phillips, 1992), and yet others find a non-linear association (Phillipsen et al., 1997). The only such study of Head Start settings found that experience and several other teacher and program factors were more predictive of quality than education (Administration of Children & Families [ACF], 2003; Administration of Children, Youth, & Families [ACYF], 2001).

In the substance-abuse prevention literature, higher levels of implementation have been associated with fewer years of teaching experience, professional qualifications, and greater teaching skills (Ringwalt et al., 2002; Rohrbach, Graham, & Hansen, 1993). Other school-based research has shown that years of experience are not related to program fidelity or likelihood of using an evidence-based curriculum (Ringwalt et al., 2002, 2003). In the early childhood literature, Wasik and colleagues (2006) found no relationship between level of teacher education and quality of program implementation. This suggests that factors other than teacher education may be critical in determining the quality of teaching in Head Start settings.

Teacher personal resources

While numerous studies have linked education and training to quality of care, very few have assessed the impact of teachers’ psychological functioning on their interactions with students. Hamre and Pianta (2004) found that, after adjusting for setting and caregiver characteristics, teachers who reported more depressive symptoms were rated by observers as less sensitive, more withdrawn, and more intrusive/negative. Emotional exhaustion is a problem faced by the early childhood workforce (Goelman & Guo, 1998). In the psychological and educational literatures, most research focuses on the predictors of this aspect of teacher burnout rather than its consequences. There is some evidence to suggest that burnout undermines job performance and is associated with negative student interactions (Lamude, Scudder, & Furno-Lamude, 1992; Maslach, Schaufeli, & Leiter, 2001), which, in turn, could undermine implementation quality (Han & Weiss, 2005). Conversely, teachers who have more personal resources in the workplace, such as higher levels of self efficacy, possess more positive attitudes regarding innovations and tend to implement those innovations with higher quality (Guskey, 1998; Kallestead & Olweus, 2003; Rohrbach, Graham, & Hansen, 1993).

Work environment

Healthy organizations provide positive, supportive, and safe environments for their staff. In the educational literature, staff perceptions of schools’ organizational health and high quality work life have been linked with greater efficacy, work attitudes and job satisfaction, and positive outcomes for students (Bryk & Schneider, 2002; Hoy & Woolfolk, 1993; Louis, 1998). Poor staff morale is associated with difficulty in implementing and sustaining innovations (Gottfredson & Gottfredson, 2002). Research in educational settings suggests that perceived support from administrators also plays an important role in fostering the commitment of staff and effective use of innovative programs and practices (Gottfredson & Gottfredson, 2002; Kam, Greenberg, & Walls, 2003; Payne, Gottfredson, & Gottfredson, 2006). However, comprehensive models that would allow an examination of the relative importance of these factors are rare, and, therefore, warrant attention.

The Present Study

The purpose of the present study was to examine process and content outcomes associated with the professional development training provided in the context of the REDI (REsearch-based Developmentally Informed) intervention. Given this focus, and the fact that the positive impact of the intervention on teaching practices has already been established (Domitrovich, Gest, Gill, Bierman, Welsh, & Jones, in press), we excluded the control group and focused exclusively on the intervention teachers. We used coaches’ monthly ratings of implementation fidelity as an outcome measure of training content because these are the types of measures most likely to be collected by community providers in practice settings when interventions such as REDI are disseminated on a broader scale.

Our first goal was to examine patterns of coaches’ monthly fidelity ratings over time. Given the extensive support provided by coaches in the REDI program, we expected teachers’ fidelity to the two dimensions of behavior targeted by REDI (social-emotional, language-literacy) to grow over the program year. Our second goal was to examine how a diverse set of individual factors including teachers’ professional characteristics, personal resources, and perceptions of the work environment were related to the REDI training process outcomes (positive perceptions of the intervention and a positive teacher-coach relationship) and content outcomes (social-emotional and language-literacy interactions). More distal factors such as teachers’ formal education, experience level, or amount of Head Start training received prior to exposure to the REDI model were not expected to predict features of the coaching relationship (i.e., openness to consultation, perceived usefulness) or intervention fidelity. However, teachers with more personal resources (i.e., higher efficacy, lower depression and burnout) and more positive perceptions of their work environment were expected to have more positive perceptions of the intervention and to be more engaged in coaching. Similar associations with implementation fidelity were also expected. The third goal was to examine how the process and content outcomes of the REDI professional development training related to one another. The strongest associations were expected among these measures as positive process outcomes are considered precursors to effective use of the teaching strategies promoted by the coaches.

Method

Participants

Data are drawn from assessments collected with intervention teachers as part of the REDI program, a randomized trial of an enriched Head Start intervention that included 44 classrooms across three Head Start programs in Central Pennsylvania. Data for the lead and assistant teachers (N = 44) in the experimental classrooms only (N = 22) were included in the study. In some analyses, data were not available for two teachers who left their positions after the mid-point of the year. The teachers who participated in the study were primarily English-speaking, Caucasian females (see Table 1).

Table 1.

Intervention teacher characteristics

	Lead Teachers	Assistant Teachers
Female	19 (100%)	20 (95%)
Ethnicity
White	16 (84%)	19 (90%)
Black	2 (11%)	0 (0%)
Hispanic	0 (0%)	2 (10%)
Multi-racial	1 (5%)	0 (0%)
Primary language
English	18 (95%)	20 (95%)
Spanish	1 (5%)	1 (5%)
Education
High School	2 (10%)	10 (47%)
Some post HS	3 (16%)	5 (24%)
Associate degree, vocational certificate	4 (21%)	5 (24%)
4-yr degree (or more)	10 (53%)	1 (5%)
Certification
CDA	6 (32%)	0 (0%)
Teacher cert. or lic.	8 (42%)	1 (5%)
None	5 (26%)	20 (95%)
Experience
0 to 5 years	4 (21%)	7 (33%)
6 to 10 years	4 (21%)	7 (33%)
11+ years	11 (58%)	7 (33%)

Procedure

In the spring, prior to the implementation of the intervention, lead and assistant teachers completed a set of ratings related to their professional background, personal resources, and job perceptions. Throughout the intervention year, coaches completed monthly ratings on the quality with which intervention teachers implemented the REDI program. At the end of the school year, as part of the post-intervention data collection, intervention teachers completed a survey about their experiences with the REDI program.

Intervention Design

The REDI intervention included explicit curriculum-based lessons, center-based extension activities, and “teaching strategies” targeting both social-emotional and language-literacy domains. It was designed to be used throughout the day to generalize the key intervention concepts. Intervention delivery was shared by teacher pairs and integrated into existing curricula (i.e., Creative Curriculum and High/Scope) used at the Head Start centers. Teachers received detailed manuals and kits containing all the materials needed to implement the intervention. Teachers in the comparison classrooms continued to conduct Head Start service “as usual.”

Language/emergent literacy skill enrichment

REDI targeted vocabulary, syntax, phonological awareness and print awareness with three programs. First, the interactive reading program was based on the shared reading program developed by Wasik and Bond (2001; Wasik, Bond, & Hindman, 2006) which was, in turn, an adaptation of the dialogic reading program (Whitehurst, Arnold, Epstein, & Angell, 1994). The curriculum included two books per week, which were scripted with interactive questions. Each book included targeted vocabulary words, presented with the aid of physical props and illustrations. Second, teachers were provided with curriculum materials to promote phonological awareness through a set of “Sound Games” based primarily upon the work of Lundberg and colleagues (Adams, Foorman, Lundberg, & Beeler, 1998). Teachers were asked to use a 10–15 minute Sound Game activity at least three times each week. Third, teachers were provided with a developmentally sequenced set of activities and materials to be used in their alphabet centers (e.g., letter stickers, letter bucket, materials for a Letter Wall, craft materials). They were asked to make sure that each child visited the alphabet center several times a week, and were given materials to track the children’s acquisition of letter names. In addition to these curriculum components, teachers received mentoring in the use of “language coaching” strategies, including vocabulary support, expansions and grammatical recasts, and decontextualized talk to provide a general scaffold for language development in the classroom (Dickinson & Smith, l994). The overall goal was to improve teachers’ strategic use of language in ways that would increase child oral language skills including vocabulary, narrative, and syntax.

Social-emotional skill enrichment

The Preschool PATHS Curriculum (Domitrovich, Greenberg, Kusche, & Cortes, 2005) was used to promote children’s social-emotional skills. It targeted four domains; 1) prosocial friendship skills, 2) emotional understanding and emotional expression skills, 3) self-control (e.g., the capacity to inhibit impulsive behavior and organize goal-directed activity), and 4) problem solving skills including interpersonal negotiation and conflict resolution skills. The curriculum is divided into 33 lessons that are delivered by teachers during circle time. These lessons include modeling stories and discussions and utilize puppet characters, photographs, and teacher role-play demonstrations. Each lesson includes extension activities (e.g., cooperative projects and games) that provide children with opportunities to practice the target skills with teacher support. Teachers taught one PATHS lesson and conducted one extension activity each week. The corresponding teaching strategies in the social-emotional domain included structuring the classroom with proactive rules and routines, positive management techniques (e.g., use of specific teacher praise and support), emotion coaching, induction strategies to encourage appropriate self-control, and the use of social problem-solving dialoguing to promote children’s flexible thinking and social competence (Denham & Burton, 2003; Bierman, Greenberg, & CPPRG, 1996).

Training and professional development support

In early August, prior to the beginning of the Head Start year, lead and assistant teachers attended a 3-day workshop conducted by the program developers. The workshop covered the theoretical and developmental model underlying REDI and oriented teachers to the rationale underlying the integration of the program’s curriculum components and general teaching strategies. There was approximately a half-day of general orientation, one day of language and literacy emphasis, one day of social-emotional emphasis, and a half-day of program-specific meetings about the logistics of implementing REDI. The domain-specific days of the training not only focused on the mechanics of conducting the curriculum lessons but also emphasized the importance of generalization through curriculum extension activities and the use of teaching strategies. Midway through the year, a one-day “booster” workshop conducted by the program developers provided a brief review of the REDI developmental model and intervention components. Teachers were encouraged to reflect on what had been working well and discuss ongoing challenges. Collective problem-solving occurred.

Intervention teachers also received weekly coaching support provided by local educational consultants (“REDI trainers”), who were experienced master teachers supervised by two project-based senior educational supervisors. The weekly consultations were intended to enhance the quality of implementation through modeling, coaching, and providing ongoing feedback. The REDI coaches spent an average of three hours per week (SD = .18, range = 2.69 – 3.33) in each classroom observing, modeling intervention techniques, or team teaching lessons. In addition, the REDI coaches held weekly one-hour meetings with the lead and assistant teachers individually following an agreed upon format. First, the teachers presented their weekly implementation form, describing what they had done, reflecting on the effectiveness of the various activities and lessons, and recording any teaching questions or challenges. This served as a platform for the REDI coaches to comment on specific positive teaching practices they had observed that week and to provide suggestions for improvements or offer solutions for the challenges that were encountered. During the second half of each meeting, the REDI coaches reviewed specific teaching strategies that were a formal part of the intervention. These were organized according to a teaching pyramid, and “rolled out” over the course of the year. During the first half of the year, coaches focused on teaching strategies that were at the broad base of the teaching pyramid, including the use of positive management strategies (e.g., use of specific praise, organization of transition routines), social-emotion skill promotion (e.g., emotion coaching), and problem prevention strategies. The coaches also introduced and emphasized language coaching strategies, especially the use of questions, expansions, and decontextualized talk as well as the generalized use of target vocabulary. During the second half of the year, the coaches emphasized strategies that could be used to respond to and re-focus problem behaviors in positive ways, particularly the use of induction strategies and social-problem solving dialogue. The goal was to maximize the use of teaching strategies that supported child language and social-emotional skill development and minimize the use of strategies at the tip of the pyramid, which included external controls (e.g., negative consequences, time-out). In presenting the various teaching strategies, the REDI coaches used examples and videotaped models to introduce skill concepts, encouraged discussion about the specific use of the strategy in the teacher’s classroom, and suggested practice activities for the coming week. Teachers were encouraged to identify personal goals regarding their planned use of the highlighted teaching strategies in the coming week. REDI coaches followed the same progression through these strategies with all teachers, but the pace was adjusted to match teachers’ mastery of the material. If necessary, the coaches extended the amount of time spent reviewing or practicing concepts before moving on.

Measures

Teacher characteristics: Professional background

A teacher survey was developed that included a section adapted from the Head Start Family and Child Experiences Survey (FACES, 1999) and elicited information about their demographics, education, training, satisfaction with training and experience of working with young children. Three variables were used in the present analyses. Education was coded as a continuous variable with values ranging from less than 12 years (for less than High School education) to 18 years (for completion of a Masters degree). Training was also coded as a continuous variable by summing the number of initial job training hours and hours of training received during the past two years. Experience was coded as the total number of years the individual had been employed as a teacher in a preschool setting.

Teacher characteristics: Personal resources

The teacher survey included three measures of personal resources. First, Depressive Symptoms were assessed with the Center for Epidemiological Studies Depression Scale (CES-D: Radloff, 1977). The CES-D is a 20 item, self-report measure used to assess depression symptoms in the general population. The cumulative scale score ranges from 0 to 60 and indicates how depressed the subject is (α = .82). Second, Emotional Exhaustion was measured with the 9-item Emotional Exhaustion subscale (α = .91) from the Maslach Burnout Inventory (Maslach & Jackson, 1981), a 22-item measure for assessing burnout among human services personnel. Finally, the 20-item Teacher Efficacy scale, which was adapted from two measures of teacher efficacy (Gibson & Dembo, 1984; Sodak & Podell, 1996), assessed teachers’ beliefs or expectations of teaching efficacy, personal efficacy and outcome efficacy within the Head Start program. Teachers rated agreement about their efficacy beliefs on a 6-point scale (α = .70).

Teacher characteristics: Perceptions of work environment

Four scales assessed teachers’ perceptions of their work environment. The first two scales came from the baseline teacher survey. Job Satisfaction was measured with a 13-item scale (Gill, Greenberg, Moon, & Margraf, 2007) that focused on the domains of salary and benefits, interpersonal climate, supervision, and roles and responsibilities (e.g., “How satisfied are you with communication between ranks in the agency?”, α = .95). Organizational Climate was measured using 28 items from the Organizational Climate Scale (OCS; Glaser, Zamanou, & Hacker, 1987; e.g., “My Head Start supervisor is a good listener”) and 11 items regarding organizational trust and motivation (e.g., In this Head Start program, teachers and other professionals trust each other; overall α = .95). Two scales from the post-test teacher survey described the teachers’ perceptions of support for the REDI intervention from the Head Start Director and the Education Manager. One scale described the extent to which each administrator Valued the goals of the REDI program and the other assessed how much they Supported Integration of REDI with the existing Head Start curriculum. For each scale, ratings for the two administrators were averaged.

Professional development: Training process

Four measures described the teacher-coach relationship and teachers’ perceptions of the process of implementing the REDI intervention. At the end of each month during the Head Start year, REDI coaches used information from their weekly teacher meetings and classroom visits to complete a set of 26 items describing each teacher. Three items describing the teacher’s receptivity to consultation, commitment to REDI, and collaborative efforts were summarized in an Openness scale (e.g., “Meetings feel like a collaborative working session. The teacher actively engages in the meetings by providing examples of situations being discussed and asking questions.” α = .94). On the post-test survey, teachers rated the Usefulness of the consultation process (“How useful did you find the consultation with the REDI coach?” On the same survey, teachers were asked, “How much did REDI coaching strategies add to the impact of REDI lessons on children’s skill development?”, then rated the impact of three language and literacy strategies (e.g., “recasting and expanding comments”) and three social-emotional strategies (e.g., “emotion coaching and dialoguing”). These six items formed an Impact scale (α = .92). Similarly, teachers rated these same six teaching strategies in response to the questions, “How well were the strategies explained and how easy were the concepts to understand?” and “How well did the strategies fit your personal teaching style?” Ratings for these 12 items formed an Acceptability scale (α = .93). All items used a 5-point Likert scale (1 = not at all, 5 = very much).

Professional development: Training outcomes

The monthly coach ratings were used to assess the extent to which teachers implemented the teaching strategies targeted by the REDI intervention model. Ratings were based on the overall impression the REDI coaches had of teachers’ demonstrated skills after observing and meeting with teachers each week. Each rating was on a 5-point Likert scale (1 = almost never, 3 = sometimes, 5 = almost always). Each item consisted of a teaching strategy label plus a one to two sentence narrative description of the strategy. Four scales were derived from these ratings: Language Richness (7 items, α = .93; e.g., fosters conversation; recasts to support grammatical understanding; uses rich and varied vocabulary); Social-Emotional Support (4 items, α = .86; encourages emotion expression; encourages emotion regulation through self-control techniques); Behavior Management (8 items, α = .93; e.g., consistency/routine; positive behavior management; uses induction strategies to promote autonomy), and Sensitivity-Responsiveness (4 items, α = .92; availability; warmth; responsiveness). To establish the validity of these monthly ratings, we examined their associations with ratings made by outside observers who visited the classroom for two hours on one occasion at post-test and completed ratings that covered very similar domains (Domitrovich et al., in press). We averaged the coaches’ ratings across the final three months of the year to ensure a similar time frame of comparison. In the social-emotional domain, coach-rated Behavior Management was positively correlated with outside observers’ ratings of Management (r = .47, p < .01) and Positive Discipline (r = .42, p < .01). Coaches’ ratings of Social-Emotional Support correlated weakly with the outside observers’ ratings of Positive Emotion (r = .28, p < .07). In the language and literacy domain, coach ratings of Language Richness and Sensitivity-Responsiveness correlated significantly with outside observers’ ratings of (Language) Richness-Sensitivity (r = .45 & r = .43, respectively, p <.01). Strong correspondence would not be expected given the vast differences in information that went into the ratings (i.e., weekly meetings and classroom visits spread over three months vs. a single 2-hour classroom visit), but these reliable correlations suggest that coaches’ ratings have some validity as indicators of teaching practices.

Results

Professional Development Training Content Outcomes: Growth Over Eight Months

We used Hierarchical Linear Modeling (HLM) to test hypotheses about growth over time in teachers’ use of the strategies emphasized by the REDI model (i.e., training content outcomes). Three-level models accounted for the nesting of period of measurement (8 monthly ratings) within teachers and teachers (lead and assistant) within classrooms. Preliminary analyses suggested that growth patterns did not vary across the three Head Start programs, so a variable representing program was not included in the final models. We modeled growth patterns for each of the four dimensions of training content outcomes (behavior management, language richness, social-emotional support, and sensitivity-responsiveness) across the eight monthly ratings, testing for linear and quadratic terms to determine the most parsimonious description. We did not test for higher-order growth terms because we had no substantive theory postulating complex non-linear effects. The quadratic growth term approached significance only for language richness (p = .07), but a plot of this term revealed a barely perceptible deviation from a linear trend, so we concluded that it was most parsimonious to model only linear growth for each dimension.

Linear growth patterns were statistically reliable and similar in magnitude for the four dimensions (see Figure 1). The slope for behavior management was β = .12 (p < .001, Table 2), indicating that coach ratings of behavior management skills increased by an average of .12 scale points per month; observed ratings increased from M_Sep = 3.02 to M_Apr = 3.71 across the eight months. This indicates that there was an increase in teachers’ use of constructive and positive classroom behavior management through the use of routines, limit setting, preparedness, and problem solving. The slope for language-richness also indicated significant linear growth across the year (β = .12, p < .001), with observed scores increasing from M_Sep = 2.78 to M_Apr = 3.49. This indicates that coaches perceived an increase in teachers’ use of modeling, recasting, rich and varied vocabulary, elaboration, and decontextualized language to support children’s emerging language and literacy skills. Coach ratings of social-emotional support in the classroom also revealed significant linear growth (β = .14, p < .001), increasing an average from M_Sep = 2.80 to M_Apr = 3.63 over the 8-month period. Coaches noted positive change in teachers’ use of feeling words in the classroom, encouragement in self-regulation of emotions, validation of children’s feelings, and modeling. Significant linear growth was also evident in coaches’ ratings of teachers’ sensitive-responsiveness (β = .08, p < .001). The overall growth pattern showed an increase of M_Sep = 3.42 to M_Apr = 3.93, indicating that coaches noted increments in teachers’ engagement and warmth and responsiveness in interactions with children. Overall, these linear growth patterns suggest improvement from “sometimes” using the particular teaching strategies (3 = sometimes) to more routine use of the practices.

Figure 1.

Mean scores of coaches’ monthly ratings of the degree to which teachers implemented the REDI teaching strategies

Table 2.

Regression models predicting content training outcomes from teacher characteristics

	Content Training Outcomes
	Behavior Management	Language-Richness	Social-Emotional Support	Sensitivity-Responsiveness
Linear Growth in Outcomes	.12***	.12***	.14***	.08***
Professional Characteristics
Education	.06 (.06)	.15** (.06)	.15*** (.05)	.08+ (.05)
Training	.01 (.01)	.01 (.01)	.01 (.01)	−.004 (.009)
Experience	−.002 (.002)	.00 (.00)	.00 (.00)	−.001 (.001)
Personal Resources
Depressive symptoms	−.02 (.01)	.00 (.01)	−.01 (.01)	−.01 (.01)
Efficacy	.09 (.20)	.37 (.22)	.26 (.19)	−.16 (.18)
Emotional Exhaustion	.19* (.09)	.20* (.09)	.11 (.08)	.14+ (.08)
Work Environment
Job Satisfaction	−.07 (.26)	.18 (.21)	−.17 (.23)	.06 (.23)
Org. Climate	−.22 (.27)	−.58** (.29)	.05 (.25)	−.14 (.24)
Values Program	.04 (.14)	.31* (.13)	.11 (.14)	.22 (.14)
Supports Integration	.01 (.09)	−.18* (.07)	−.04 (.08)	−.03 (.08)

^*p < .05

^**p < .01

^***p < .001

Predicting Variations in Training Process and Training Content Outcomes

Next, we tested a series of models in which we added conceptually grouped sets of variables as predictors of variations in growth patterns. By treating the growth term as a random effect, we tested whether a given set of predictors (e.g., education, training, experience) accounted for variation around the typical growth pattern. The conceptually grouped sets of predictors were the measures of teacher characteristics (professional background, personal resources, and perceptions of work environment) and the measures of the training process (coach-rated openness to consultation, teacher-rated usefulness of consultation, and teacher-rated acceptability and impact of the intervention). In all models, we tested for two-way interactions between teacher role (lead vs. assistant) and each predictor because most research focuses on lead teachers and it is not clear whether determinants of quality would be the same for assistant teachers. Reliable interaction effects that replicated across two or more dependent variables are reported below. Results indicated only two statistically significant predictors of variations in growth patterns out of a total of 68 tests (17 predictors for each of the 4 outcomes), which is no more than one would expect by chance. This indicated that the measures of teacher characteristics and training process were not accounting for variations around the typical growth pattern across the eight monthly ratings. By specifying the intercept at the eighth and final monthly rating, however, we could interpret significant fixed effects for the predictors as effects of those predictors on end-of-year implementation fidelity. We present results for these fixed effects terms in three sections, focusing on teacher characteristics as predictors of content outcomes, teacher characteristics as predictors of process outcomes and training process measures as predictors of training content outcomes.

Teacher characteristics predicting training content

Among the measures of professional characteristics, years of education uniquely predicted language-richness (β = .15, p < .01, Table 2) and social-emotional support (β = .15, p < .001), whereas hours of training and years of experience were not uniquely associated with any of the measures of quality. Among the measures of personal resources, emotional exhaustion was uniquely and positively associated with behavior management (β = .19, p < .05) and language-richness (β = .20, p < .05), meaning that teachers who reported being more emotionally exhausted were rated by coaches as engaging in more effective behavior management and richer patterns of language interactions. In contrast, depressive symptoms and teaching efficacy were not uniquely associated with any of the dimensions of teacher quality. None of the measures of teachers’ perceptions of the work environment uniquely predicted multiple dimensions of quality, but three of the four predictors were uniquely associated with language richness. Teachers who perceived their program administrators to have more positive views of the REDI program provided richer language input to students by the end of the year (β = .29, p < .05); but, counter intuitively, organizational climate (β = −.51, p < .01) and teachers’ perceptions of administrators’ support for teacher integration of REDI in daily program activities (β = −.18, p < .05) were significantly but negatively associated with language richness.

Teacher characteristics predicting training process

There were only two reliable associations between teacher characteristics and their perceptions of the intervention or their engagement in the coaching relationship (Table 3): teachers with more hours of training were more likely to perceive the program as having a positive impact on children’s skills (β = .04, p < .05); and teachers who reported that program administrators valued the REDI intervention were more likely to perceive the intervention to have a positive impact (β = .36, p < .05).

Table 3.

Regression models predicting process training outcomes from teacher characteristics

	Perceptions of Intervention		Engagement in Consultation
	Acceptance	Impact	Openness(coach rating)	Usefulness(teacher rating)
				Odds Ratios
Professional Characteristics
Education	.06 (.07)	.13 (.08)	.05 (.07)	.77 (.33)
Training	.01 (.01)	.04* (.014)	.00 (.01)	1.01 (.02)
Experience	.00 (.00)	.00 (.00)	−.001 (.002)	1.02 (.07)
Personal Resources
Depression	−.02 (.01)	−.01 (.02)	.00 (.02)	.98 (.10)
Efficacy	.33 (.21)	−.12 (.26)	−.07 (.25)	.59 (.62)
Emotional Exhaust.	.01 (.08)	−.15 (.11)	.06 (.11)	.92 (.31)
Work Environment
Job Satisfaction	.23 (.21)	.02 (.28)	−.32 (.21)	2.66 (2.31)
Org. Climate	.08 (.20)	.26 (.33)	−.02 (.21)	.12+ (.15)
Values Program	.09 (.10)	.36* (.15)	.24+ (.13)	.98 (.57)
Supports Integration	.05 (.07)	−.11 (.11)	.08 (.08)	1.48 (.64)

^*p < .05

Training process predicting training content

Teacher reports of the acceptability of the REDI intervention (i.e., fit their personal teaching style, easy to use) were uniquely associated with the provision of social-emotional support (β = .40, p < .05). In contrast, teacher perceptions of the program’s impact on children’s skill development did not predict end-of-year implementation fidelity ratings on any of the four dimensions. Openness to consultation (averaged across coaches’ eight monthly ratings) uniquely predicted all four dimensions of end-of-year quality, but these effects were qualified by teacher-role by openness interaction effects that took the same form across dependent variables. A representative plot of these interactions (Figure 2) indicates that as compared to the assistant teachers, the association between openness to ongoing mentoring and consultation end-of-year implementation fidelity was stronger for lead teachers.

Figure 2.

Interaction effects of teacher role*openness to consultation

Discussion

Early childhood programs such as Head Start are striving to improve their impact on children’s social and cognitive development as the current climate of educational accountability extends downward into preschool. One strategy toward this end is to bolster existing curricula with evidence-based interventions that facilitate instructional and relational practices that are proximal determinants of social and cognitive development. As new programs or practices are introduced into community settings, the interests of program administrators and intervention developers converge on the critical role of professional development training programs. From the Head Start administrator perspective, professional development training is both a long-term investment in teachers’ skills and a way to enhance their ability to use a specific program or practice effectively. From the intervention developer perspective, professional development training is a critical step towards achieving fidelity and the high quality implementation necessary to realize intended intervention effects (Durlak & Dupree, 2008). Given the importance of effective professional development training, it makes sense to examine both its process and its content. Content outcomes are obvious targets because they represent the critical components that make an intervention effective. Process outcomes are also critical because very little is known about the most effective ways to bring about behavior change in teachers. The present study examined these issues in the context of the Research-Based Developmentally Informed (REDI) project, an efficacy trial of an evidence-based preschool enrichment program.

Growth Patterns in Training Content Outcomes

Coach ratings may be particularly relevant to community settings where the roles of professional development coach and quality evaluator are often combined. In this regard, it is noteworthy that ratings made by the REDI coaches, based on weekly meetings with teachers and classroom visits over a three month period, were reliably correlated with corresponding dimensions rated by outside observers who visited the classroom for two hours on one occasion (Domitrovich, et al., in press). Correlations were most robust for behavior management and language richness and more modest for emotional climate. This suggests that judgments about emotional climate may be more sensitive to time spent with the teacher and in the classroom, reflecting either the inherent variability of emotional climates or changes over time in the cues that the raters used to judge this dimension. In either case, the fact that coach ratings correlated reliably with outside observers’ ratings supports the viability of using coaches’ ratings to track training outcomes.

Coaches’ weekly ratings made in the context of the on-going support they provided to the teachers suggested significant linear growth over time in teachers’ use of the targeted teaching strategies.

Predicting Professional Development Process and Content Outcomes

Analyses indicated that teachers varied substantially in their rates of linear growth in coach-rated quality. However, the number of reliable associations between teacher characteristics and the linear growth terms did not exceed chance levels, indicating that the variability in growth patterns was generally not accounted for by the present measures of professional characteristics, personal resources, or workplace perceptions. On the one hand, this unexpected null finding has positive implications for professional development efforts in that no particular professional or personal characteristics were associated with poor prospects for professional growth. On the other hand, the significant variability in professional growth remains essentially unexplained and is an important target for future research.

In predicting end-of-year training content outcomes, teacher education uniquely predicted two of the four dimensions of teaching quality. This is consistent with research on structural predictors of quality in childcare settings (Blau, 2000; de Kruif, McWilliam, Ridley, & Wakeley, 2000; NICHD ECCRN, 2000; Phillipsen et al., 1997) but not Head Start or public pre-K classrooms (ACF, 2003; Early, Bryant, et al., 2006). Unlike the global measures of classroom quality (rated by outside observers) used in the latter studies, we focused on teaching strategies that are linked to the specific REDI intervention components and that help to generalize concepts from those intervention components throughout the day (e.g., language recasting as a generalization of dialogic reading activities). It may be that the foundation provided by formal education provides a better base for teachers to generalize these specific teaching strategies outside of their activity formats. As anticipated, years of experience and hours of previous Head Start training were unrelated to end-of-year implementation fidelity.

Among the measures of personal psychological resources, teachers who rated themselves as more overwhelmed and fatigued by their job at the end of the preceding school year were rated by coaches as more effective in their behavior management skills and as engaging in richer conversations with children by the end of the REDI year. This finding is counter-intuitive but makes sense if one considers that exhaustion at the end of the previous school year may be an outcome of a teacher’s past motivation to expend significant energy in teaching activities, which in turn may be a positive indicator of motivation to expend energy learning new teaching strategies in the subsequent year. In the only study examining emotional exhaustion and implementation of an intervention (as opposed to other aspects of job performance), researchers in the Netherlands (Evers, Brouwers, & Tomic, 2002) found a negative univariate association between emotional exhaustion (i.e., burnout) and lesson dose, but this association was not reliable in multivariate analyses. There are a number of studies linking teacher efficacy with high quality implementation (Kallestead & Olweus, 2003; Rohrbach et al., 1993) but this association was not replicated in the current study.

Teachers’ perceptions of the work environment were related to their use of language-related behaviors promoted by the REDI program, but mostly in counterintuitive ways that did not replicate across other training content outcomes. Consistent with prior research on school-based programs (Gottfredson & Gottfredson, 2002; Payne et al., 2006), teachers who reported that their program’s administration valued the REDI program achieved higher levels of language-richness, but contrary to expectations, teachers who described a more negative organizational climate and less administrative support for the integration of the curriculum achieved higher levels of language-richness. It is possible to offer post hoc explanations for the latter effects (e.g., teachers who perceived their programs negatively may be particularly responsive to the alternative source of support provided by the REDI coach), but because these effects emerged for only one of the four training outcomes, caution in interpretation is warranted pending replication.

Variations in teachers’ perceptions of the intervention and engagement in the coaching relationship were generally not predicted by the teachers’ professional characteristics, personal psychological resources or perceptions of the workplace (e.g., only two reliable associations out of 40 tested). This failure to identify predictors of variations in training process outcomes represents an important gap in our understanding because some of these training process measures (teachers’ acceptance of the intervention and their openness to consultation) were associated with end-of-year implementation fidelity.

The most consistent training process predictor of training content outcomes was the degree to which the REDI coaches perceived teachers as being open to consultation. The finding was consistent with our hypothesis and confirms the expectation that active engagement in the immediate training context (i.e., the coaching relationship) is highly relevant to teachers’ ability to implement an intervention with fidelity. The importance of openness was somewhat greater for lead teachers as compared to the assistants. Previous research on teacher outcomes associated with REDI has established that lead and assistant teachers made similar intervention gains over time (Domitrovich et al., in press). It is possible that because lead teachers have more formal education and are accustomed to taking the lead role in establishing classroom teaching practices, changing their behavior was more dependent on being willing to work with the coach. In contrast, assistant teachers may be more accustomed to taking their cues from the lead teacher and thus their behavior change may be less dependent on their openness to coaching. The failure of teacher perceptions of the usefulness of coaching to predict implementation outcomes may be misleading due to the highly restricted range of these ratings: over 70% of all teachers gave the maximum possible rating of 5 so that there was very little variability in perceived usefulness to relate to implementation fidelity.

Teachers’ perceptions of the impact of the REDI teaching strategies were not associated with implementation fidelity, which is not consistent with the findings of Han and Weiss (2005). It is important to note that the REDI program included curriculum components that consisted of explicit lessons and structured activities in addition to the teaching strategies that were assessed by the REDI coaches. It may be that teachers’ perceptions of the effectiveness of REDI were more strongly linked to these explicit curriculum components as opposed to the more general teaching strategies designed to reinforce them.

Limitations and Future Directions

This study had several strengths, including monthly ratings of implementation fidelity and a diverse set of predictors of variations in quality, but several limitations are noteworthy. First, reliability data for the coaches’ ratings was not available because, unlike measures based on relatively brief classroom observations (LaParo & Pianta, 2003), the monthly coach ratings were based on information gleaned from four weekly meetings with teachers and four classroom visits. It was not feasible to obtain ratings from a second individual with equally extensive classroom contact. However, the reliable correlations between coach ratings and outside observer ratings provide some assurance of the reliability of the coach ratings and offset this constraint to some extent. Second, teacher perceptions of the intervention were collected at the end of the intervention year so it is not clear whether these perceptions were a cause or a consequence of their end-of-year implementation fidelity: for example, it is possible that teachers who implemented the program with quality came to perceive it as congruent with their teaching style (i.e., a “convert effect”). In future studies of REDI sustainability we will examine these constructs longitudinally to clarify the direction of effects. Third, a larger sample size would allow for greater confidence in the findings and the exploration of how the teaching pairs might influence each others’ practices. Finally, ratings of teacher openness to consultation and implementation fidelity were both collected from the perspective of REDI coaches. Shared method variance inflates the correspondence between these variables. Future studies will include assessments of the teachers’ perspective and will measure other aspects of the coach-teacher relationship process to clarify the essential ingredients of this type of professional development experience. In this regard, research on the “therapeutic alliance” between psychotherapists and their clients may offer some useful insights. Meta-analyses confirm that the therapist-client relationship is an essential element through which the process of change occurs (Martin, Garske, & Davis, 2000), with most researchers including measures of the affective bond, degree of collaboration and goal or activity alignment between the therapist and client from both individuals’ perspective. Other intervention studies that employ coaches are beginning to include measures such as these (J. Wehby, personal communication, September 9, 2008).

Our findings are consistent with the conceptual models of implementation described by Greenberg and colleagues for school-based preventive interventions (Domitrovich et al., 2008; Greenberg, Domitrovich, Graczyk, & Zins, 2005) in which characteristics of two distinct components must be considered to make evidence-based interventions work effectively in community settings: the intervention itself, and its support system (Chen, 1998; Klein & Sorra, 1996). It is around issues related to support systems that the interests of the prevention field converge with that of professional development. Researchers in both these areas can benefit from examining each others’ theories, practices, and measurement strategies. What this study suggests is what many of us already believe, that the motivation and engagement of teachers in the moment are as important as the more distal experiences that have traditionally been considered the primary pathway to quality practice.

Table 4.

Regression models predicting content training outcomes from process training outcomes.

	Behavior Management	Language-Richness	Social-Emotional Support	Sensitivity-Responsiveness
Perceptions of Intervention
Acceptance	.32+ (.19)	.32 (.20)	.40* (.17)	.09 (.20)
Impact	−.13 (.15)	−.01 (.16)	.002 (.13)	.09 (.16)
Engagement in Consultation
Openness (coach rating)	.18*** (.05)	.12* (.06)	.15** (.06)	.22*** (.05)
Usefulness (teacher rating)	−.06 (.10)	−.07 (.10)	−.02 (.10)	−.15 (.11)
Lead-Teacher* Openness	.15* (.07)	.34*** (.08)	.17* (.08)	.12+ (.07)

^*p < .05

^**p < .01

^***p < .001