Assessing Generalization During Professional Development in Child Care: A Pilot Randomized Controlled Trial

Abstract

Early childhood care and education providers provide instruction to diverse learners, including children with developmental delays, but often lack training in the use of evidence-based instructional strategies to support children’s meaningful learning engagement. This preliminary study examined effects of the Early Achievements for Child Care Providers professional development program with and without generalization training on center-based child care providers (N = 12). Outcomes assessed included providers’ implementation fidelity, knowledge, and self-efficacy as well as children’s (N = 43) social-communication skills and engagement in shared book reading. Results demonstrated that providers trained in a generalization context made larger gains in implementation fidelity in a generalization context; no other significant between-group differences were found for providers. Children in both groups made comparable, significant gains. Findings suggest that explicit generalization training is needed for providers to achieve fidelity when transferring newly learned evidence-based practices to additional classroom instructional contexts, but a brief interval of implementation likely is insufficient to differentially benefit child outcomes. Researchers should elicit providers’ perceptions when designing professional development programs to maximize feasibility of investing additional time and resources for generalization training.

Early childhood care and education (ECCE) providers’ (hereafter, providers) use of evidence-based instructional strategies is critical during children’s early years to support their meaningful engagement in learning activities. Consistent high-quality language stimulation in inclusive ECCE settings during a formative period of neurodevelopment is critical for promoting development and school readiness in young children with developmental delays (DD) (Terrell & Watson, 2018). Yet a paucity of research has focused on the design and effectiveness of high-quality professional development (PD) targeting providers’ implementation of evidence-based practices with children with DD in inclusive ECCE classrooms (Brunsek et al., 2020; Markussen-Brown et al., 2017). This preliminary study examines the effects of a PD program with and without training in a generalization context on providers’ fidelity, knowledge, and self-efficacy to inform whether a single training context is sufficient to support providers in generalizing new skills to an untrained context.

ECCE providers deliver instruction to a diverse set of learners, including those with an identified learning-related DD (Boyle et al., 2011), experiencing language disparities (Walker et al., 2020), or having as yet unidentified delays (Lipkin et al., 2020). To narrow the learning gap that currently exists for these children, providers’ proficient and consistent implementation of evidence-based development-enhancing instructional strategies is critical. Indeed, providers have identified child development and developmentally appropriate practice as areas of need for further training and cited lack of training on how to teach children with DD as a specific barrier to providing high-quality care (Vesay, 2008). Not surprisingly, educators working in inclusive classrooms report lower self-efficacy for working with children with disabilities (Guo et al., 2021). This is likely due to a lack of knowledge and skills that would enable them to effectively support children with diverse learning needs (Guo et al., 2021). There is a critical need for high-quality PD for providers focused on (a) empowering their implementation of evidence-based instructional strategies to support the learning and development of children with DD and (b) doing so across instructional contexts.

Practice-Based PD

Effective PD is practice-based (Dunst et al., 2015), providing instruction and coaching to implement targeted instructional practices and content delivery in job-embedded (Dunst et al., 2015; Elek & Page, 2019) contexts. To promote high-fidelity implementation, our PD program focused on a key instructional context in early childhood curricula (Lorio & Woods, 2020): shared book reading. Shared book reading provides an enriching context for teaching vocabulary (Toub et al., 2018; Wasik et al., 2016), which can be enhanced with opportunities for adult-supported play with book-related props (Toub et al., 2018) to increase engagement and language use, even in children with DD (Engelstad, Holingue, & Landa, 2020; Feuerstein & Landa, 2020). Further, book reading is often enjoyable (thus motivating and reinforcing) for both adults and children (Sim & Berthelsen, 2014). Embedding training of new skills that builds on existing expertise within a motivating and established practice is expected to increase adult learning success (Lorio & Woods, 2020). Provider implementation of evidence-based practices during shared book reading is expected to effect child language and literacy gains (Wasik et al., 2016).

PD Delivery Features

Pairing interactive workshops with job-embedded coaching improves providers’ ability to implement early language and literacy interventions (Brunsek et al., 2020; Lorio & Woods, 2020). Coaching leads to greater effects in providers’ outcomes related to their interactions with children and use of language and literacy materials (Markussen-Brown et al., 2017). Coaches in the current study built collaborative partnerships with providers using the Practice-Based Coaching model (Snyder et al., 2022), which emphasizes consideration of providers’ unique needs and classroom dynamics.

Subject-Specific PD

One ingredient of effective PD is subject specificity, emphasizing a particular set of child learning targets within the PD program (Dunst et al., 2015; Learning Forward, 2022). Given the importance of language and meaning-related emergent literacy instruction for children’s long-term educational success (Terrell & Watson, 2018), we selected these for our PD program’s subject focus. Such focus is likely to be perceived by educators as relevant since these skills are essential components of state early learning standards, and thus motivate buy-in and change in practice (Piasta et al., 2017). Educators’ language and literacy knowledge is important for informing classroom practices (Ottley et al., 2015). This includes discipline-specific content knowledge and knowledge of effective strategies to support children’s language and emergent literacy skills. PD programs can enhance providers’ knowledge of, and skill in implementing, evidence-based practices to enhance children’s language and emergent literacy skills (Feuerstein & Landa, 2020).

Relevant and Accessible Strategies as PD Targets

Our training focused on implementation of selected naturalistic developmental behavioral intervention (NDBI) strategies (Schreibman et al., 2015) for supporting children’s language and literacy (Fleury et al., 2021) development. NDBIs are a suitable match for providers’ implementation in inclusive classrooms due to their naturalistic focus on early child development (Maye et al., 2020) and alignment with the National Association for the Education of Young Children’s (NAEYC, 2022) recommended child-contingent/responsive approaches. They utilize a three-part contingency (clear and developmentally appropriate antecedent – child behavior – contingent consequence) to support children’s motivation, participation, and learning. Further, early intervention providers recognize NDBIs as effective for young children with and without autism spectrum disorder (ASD) in inclusive ECCE settings (Maye et al., 2020), supporting their applicability to whole-group classroom activities.

PD Ingredients to Promote Implementation Generalization

Generalization occurs when a learned behavior continues to be implemented across time, setting, and target in the absence of the conditions that promoted its acquisition (Stokes & Baer, 1977). Generalization of learned behaviors is a crucial prerequisite for sustainability, the continued implementation of a behavior after the conditions in which it was acquired are removed or changed (Burns et al., 2013). While research examining generalization of specific teaching practices is limited, extant evidence suggests that generalization training supports providers’ abilities to apply learned skills across contexts (Hemmeter et al., 2015; Scheeler et al., 2009).

The Current Study

The current study sought to preliminarily evaluate the impact of including an explicit generalization training component within a PD program by comparing implementation fidelity outcomes of providers trained in only one context (book sharing only; no generalization training) versus two contexts (book sharing and another related activity; generalization training). This study was needed to guide refinement of our PD program for a larger, iterative development study, culminating in a pilot randomized control trial. Further, studies examining the inclusion of explicit generalization training in ECCE classrooms are warranted to identify the levels of support needed to enhance providers’ language and literacy practices across multiple classroom contexts. This information is crucial to inform development of effective, efficient, and sustainable high-quality PD programs that positively impact both providers’ evidence-based practices and children’s development. The results of such PD programs will provide evidence for the creation of improved policies guiding the delivery of early language and literacy instruction for children with delays in inclusive learning environments.

We examined four research questions:

How does generalization training impact:

1. providers’ implementation of evidence-based instructional strategies during book sharing and in classroom activities other than book sharing?

2. providers’ knowledge of evidence-based instructional strategies?

3. providers’ self-efficacy regarding implementation of evidence-based instructional strategies?

4. children’s social-communication skills and engagement in shared book reading?

We hypothesized that providers receiving explicit generalization training would make greater gains in all areas examined due to their increased practice implementing the instructional strategies across contexts. Also, we hypothesized that children in both groups would exhibit gains in social-communication and engagement during shared book reading.

Materials and methods

Design

This 16-week study presents data collected during a pilot randomized controled trial as part of a larger iterative development study to guide the refinement of the PD program to enhance its adoptability, feasibility, and sustainability in inclusive ECCE classrooms. All study procedures were approved by the Johns Hopkins University School of Medicine Institutional Review Board (IRB).

Recruitment

Providers were recruited from toddler classrooms in licensed child care centers in an urban area in Maryland. Child care centers were identified from a list of state-licensed centers provided by the Maryland Family Network. Child care directors were contacted via email; in-person meetings subsequently were scheduled to discuss the study further. Interested directors identified eligible teachers to meet with study staff to review the study procedures and consent process. Providers could consent to participate during a face-to-face meeting or later using an online IRB-approved consent form. All children and families in participating classrooms were invited to join the study via a recruitment flyer and in-person classroom visits. Families consented via an IRB-approved electronic consent form. Caregivers were emailed secure links to complete the Ages and Stages Questionnaires, Third Edition (ASQ-3) (Squires et al., 2009) and the Ages and Stages Questionnaires: Social Emotional, Second Edition (ASQ: SE-2) (Squires et al., 2002).

Participants

Providers

Eligibility criteria for participating providers included: (a) currently working in a state-licensed center-based ECCE center, (b) currently teaching a class with children ages 18 to 42 months, and (c) having at least one participating child in their classroom with social and/or communication delays. Table 1 provides demographic characteristics of the 12 participating providers by group. Figure 1 displays a CONSORT flow diagram for providers (Schultz et al., 2010), from enrollment through data analysis.

Table 1.

Provider Demographic Data

		Group		Total (n = 12)
		BO (n = 6)	B + RA (n = 6)
Sex (%)
	Female	100%	100%	100%
Race (%)
	White	50%	33.33%	41.67%
	Black/African American	33.33%	66.67%	50%
	Prefer not to answer	16.67%	0%	8.33%
Ethnicity (%)
	Hispanic/Latino	0%	0%	0%
Education Level
	High school diploma/GED	16.67%	16.67%	16.67%
	Associate’s degree/some college	50.00%	66.67%	58.33%
	Bachelor’s degree	33.33%	16.67%	25.00%
Years teaching experience, Mean(SD)		13.17(8.66)	10.67(5.13)	11.92(6.91)
Years working at current center, Mean(SD)		7.67(7.23)	2.00(2)	4.83(5.86)

Note. BO = Book Only; B + RA = Book + Related Activity

Fig. 1.

Provider Recruitment and Randomization Flow Diagram, Enrollment Through Data Analysis. (Note: BO = Book Sharing Only. B + RA = Book Sharing and Related Activity)

Children

Child participants met the following eligibility criteria: (a) age between 18 and 42 months at the start of the study, (b) exposure to English at least 50% of the time at home, and (c) if a hearing or visual impairment was present, having a corrective hearing device or glasses/contact lenses. Figure 2 displays a CONSORT flow diagram for children (Schultz et al., 2010) from enrollment through data analysis. A total of 23 children participated in the ‘Book Sharing Only’ (BO) group (14 with typical development, nine with delays), and 20 participated in the ‘Book Sharing and Related Activity’ (B + RA) group (13 with typical development, seven with delays). Table 2 provides child demographic characteristics by group. Children were classified as having a delay if they met any of the following criteria: (a) their parent or legal guardian reported a delay or disability or reported that the child was receiving developmental support services (e.g., speech-language pathology), (b) they had one or more ASQ-3 subscale scores falling in the ‘Monitor’ or ‘Further Assessment’ ranges, or (c) they had one or more ASQ-SE:2 subscale scores falling in the ‘Possible Delay’ or ‘Further Investigation’ ranges.

Fig. 2.

Child Recruitment and Randomization Flow Diagram, Enrollment Through Data Analysis

Table 2.

Child Demographic Data

		Group		Total (n = 43)
		BO (n = 23)	B + RA (n = 20)
Sex (%)
	Male	52.17%	65%	58.14%
Chronological Age (Months) Mean (SD) Race (%)*		31.78 (6.39)	25.2 (5.35)	28.72 (6.73)
	White	73.91%	50%	62.80%
	Asian	17.39%	10%	11.63%
	Black/African American	8.70%	45%	25.58%
	Prefer not to answer	4.35%	5%	4.65%
Ethnicity (%)
	Hispanic/Latino	8.70%	10%	9.30%
Caregiver’s Highest Level of Education
	High school diploma/GED	4.35%	5%	4.65%
	Associate’s degree/some college	4.35%	10%	6.98%
	Bachelor’s degree	21.74%	25%	23.26%
Graduate degree Total Reported Annual Household Income $10,000-$49,000 $50,000-$99,000 $100,000-$149,000 ≥$150,000 Prefer not to answer		69.57% 4.35% 0% 17.39% 52.17% 26.10%	60% 10% 20% 0% 45% 25%	65.12% 6.98% 9.30% 9.30% 48.84% 25.58%

Note. BO = Book Only; B + RA = Book + Related Activity. *Total percentages are greater than 100% because caregivers were asked to select all that applied

Randomization

Providers were randomized at the center level to training in one context (book only) (BO group; n = 6), or two contexts (book sharing and related activity) (B + RA group; n = 6). Using R statistical package (version 3.6.1), a statistician simulated all possible randomizations, selecting the one that achieved the following parameters: (a) both groups have six providers, (b) all providers within the same center are randomized to the same group, (c) both groups have equal or nearly equal numbers of children, and (d) the proportion of children with DD are balanced between groups. The final solution was identified as having the best balance of family income, both provider and child race, and providers’ years of experience between groups.

Early Achievements for Child Care Providers (EA-CP) PD

The PD program targeted providers’ implementation of selected NDBI strategies and use of book-related props to create embodied child social-communication learning experiences (Landa et al., 2011) embedded within book reading instruction (Schreibman et al., 2015). This intervention was translated for use in inclusive child care classrooms from its initial iteration in preschool classrooms serving children with ASD (Engelstad et al., 2020) and in a clinic-based intervention for toddlers with ASD (Landa et al., 2011). The EA-CP PD consisted of a combination of workshops and coaching.

Workshops

All 12 providers participated in two six-hour workshops at a research center during the first and fourth weeks of the 16-week intervention. Providers received continuing education hours through the State Department of Education Office of Child Care for workshop participation. Workshops were facilitated by expert coaches. Coaches delivered didactic, lecture-based instruction using PowerPoint presentations with video models of strategy implementation (Coogle et al., 2022). Coaches utilized adult learning principles to create a supportive community of learners by integrating active learning opportunities into didactic elements of the PD including observation, feedback, collaboration, and discussion (Knowles et al., 2015). The workshops consisted of practice identifying strategy implementation by other providers in group video analysis, role-playing the instructional strategies in small groups and providing feedback and participation in a respectful learning environment, where providers learned from one another by sharing their successes and perceived barriers within a community of peers. These collaborative whole-group discussions created opportunities for providers to be resources for one another and collaboratively plan implementation of the newly learned skills in their unique classroom contexts (Nicholson & Reifel, 2011).

The six providers in the B + RA group also attended one additional 4-hour workshop during the tenth week of intervention to learn how to apply the instructional strategies within another classroom activity of their choice (e.g., snack, art, dramatic play). We incorporated four practices recommended by Stokes & Baer (1977) to target provider skill generalization. First, we provided a separate workshop focused solely on generalizing the NDBI strategies for targeting language and emergent literacy skills to a different instructional context. Secondly, the content of the generalization workshop built upon the concepts targeted in the primary instructional context (shared book reading), enabling providers to build upon implementation successes established prior to being introduced to the generalization contexts. Thirdly, we trained providers to incorporate stimuli that would be the same across the shared book reading and generalization contexts. Finally, we provided numerous examples of ways to implement the EA-CP strategies in other contexts (e.g., art, dramatic play), with explicit instruction on integration of the target books used during the PD program. Study coaches demonstrated a sample dramatic play activity, using an adaptable soft script. Coaches then led a whole-group role playing activity and received feedback from the study coaches and other providers. Providers also had the opportunity to develop and receive feedback as they implemented the EA-CP instructional strategies in another activity of their choosing.

Workshop Fidelity. Two trained study staff members assessed workshop fidelity using a 22-item Workshop Implementation Form examining adherence (e.g., facilitating group discussion, engaging providers in role-playing of instructional strategies) and quality variables (e.g., eliciting providers’ self-reflections during breakout sessions). Adherence (M = 90.63%, SD = 6.25%) and quality (M = 96.43%, SD = 4.12%) were high across workshops suggesting that the coaches delivered the PD workshops as intended.

Coaching

All providers in both groups received 16 coaching sessions across the PD period using PBC (Snyder et al., 2022), consisting of focused observation, reflection plus feedback, and shared goals and action planning. Coaches and providers collaboratively engaged in these components in a cyclical process (Snyder et al., 2022) to support implementation of the EA-CP instructional strategies. Coaches conducted a focused observation during the providers’ book reading while providing in-vivo coaching, modeling or prompting as appropriate to support providers’ implementation of the EA-CP instructional strategies. Then, the provider and coach engaged in a 10- to 15-minute collaborative debriefing discussion in the providers’ centers. Providers in the B + RA group received coaching during book reading only for the first nine weeks and then received coaching in both book sharing and book-related activity contexts for the last seven weeks, following their attendance at the third (generalization) workshop. During these last seven weeks, coaches completed a focused observation of the B + RA providers’ shared book reading and their book-related activity and then conducted one 10- to 15-minute debriefing session.

During debriefing discussions, the coach engaged the provider in guided reflection about her implementation of the instructional strategies, while providing supportive and constructive feedback. The coach and provider ended each session by co-constructing three goals and an action plan including required resources (Snyder et al., 2022). Providers’ goals targeted their implementation of specific instructional strategies and/or engagement of their students during book sharing and/or the book-related activity (if applicable). Examples included using book-related items (e.g., animal figurines or story-related props) within a generalization context (a classroom center activity) to provide opportunities to target child embodiment of story vocabulary or events or using event casts through narrating children’s actions (e.g., “Kate is cracking the dinosaur egg!”).

During shared book reading, study coaches taught providers how to create opportunities for six instructional targets: expressive language, receptive language, initiation of joint attention, response to joint attention, play, and peer-to-peer engagement (Feuerstein & Landa, 2020). Providers were encouraged to use the following instructional strategies throughout the book sharing activity: (a) provide clear antecedent cues, (b) prompt (including modeling and pausing), (c) reinforce child attempts, (d) expand child utterances, and (e) provide event casts (describing ongoing activities of the child or a peer). These instructional strategies are reflected in the DEC’s Recommended Practices and have been shown to positively impact toddlers’ social and language skills when implemented during book reading instruction (Feuerstein & Landa, 2020; Division for Early Childhood, 2014).

Coaching Fidelity. Two trained study staff members rated four randomly selected audio tapes from each of the providers’ sessions using a coaching fidelity protocol, consisting of two domains: adherence (M = 98.33%, SD = 3.76%) and quality (M = 99.31%, SD = 2.89%). Overall average coaching fidelity was high, indicating that the coaching protocols were followed as intended (M = 98.77%, SD = 2.99%).

Provider Outcome Measures

Provider Implementation: Book Sharing Fidelity

All providers were video recorded in their classrooms during a book sharing activity at pre- and post-training by trained research assistants on days a coach was not present. Post-training videorecording occurred during the week immediately following final coaching sessions (M = 3.42 days, SD = 2.02 days, range 1–7 days). Providers’ implementation of the targeted evidence-based instructional practices was examined using an adaptation of the EA-CP Book Sharing Implementation Fidelity Form and coding manual (Feuerstein & Landa, 2020). The adapted form consisted of four sections with 38 total items: (a) arranging the environment (e.g., provider positioned so that all children can see the book and her) (3 items); (b) structure of the book sharing activity (e.g., introducing the activity, reading the book title) (7 items); (c) target behaviors for providers (e.g., using NDBIs and instructional targets) (23 items); and (d) overall qualitative characteristics during book sharing (e.g., exaggerating voice inflection to highlight targeted vocabulary) (5 items). Each item was rated on a 3-point Likert-type scale (0 = no implementation; 1 = attempted but did not meet criteria for full and correct implementation; 2 = full and correct implementation). Percent fidelity was calculated as the total sum of points divided by the number of total points possible (76) multiplied by 100. Inter-rater reliability was conducted by two independent, trained coders, blinded to group membership and data point on 33.33% (8/24) of the providers’ pre- and post-training videos. The mean weighted K of 0.55 (SD = 0.18) indicates a moderate level of inter-rater agreement (Landis & Koch, 1977) provide the following kappa statistic interpretation values: <0.00 poor, 0.00-0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81-1.00 almost perfect.

Provider Implementation: Book-Related Activity Fidelity

All providers were also videorecorded at pre- and post-training in their classrooms by trained research assistants during a self-selected classroom activity (e.g., art, dramatic play) on a day of their choosing when a coach was not present. Providers’ implementation of the EA-CP instructional strategies during that activity were evaluated with the Provider Implementation Book-Related Activity Fidelity form, adapted from the EA-CP Book Sharing Implementation Fidelity Form (see above). The fidelity form consisted of 17 items rated on the same 3-point Likert-type scale used for the EA-CP Book Sharing Implementation Fidelity Form: (a) environmental arrangement (4 items); (b) provider’s use of instructional strategies (8 items); and (c) provider’s elicitation of EA-CP child target skills (5 items). Percent fidelity was calculated using the same method described above with a maximum of 34 possible points. Inter-rater reliability coding was conducted by two independent, trained coders, blinded to data point on 25% (6/24) of the providers’ pre- and post-training video recordings. The mean weighted K of 0.85 (SD = 0.11) indicates a substantial level of inter-rater agreement.

Video Knowledge Assessment

We developed the Video Knowledge Assessment as a proximal measure to assess providers’ knowledge of the study-specific EA-CP instructional strategies by identifying them in brief video clips. The assessment consisted of 14 multiple-choice items, each worth one point and corresponding to a video of an EA-CP-trained provider from a previous cohort implementing one of the EA-CP instructional strategies during book sharing. Providers completed the assessment at pre-training (at the start of the first workshop) by pencil-and-paper and, due to COVID-19 pandemic infection control restrictions, electronically over Zoom at post-training (within one month of the last coaching session).

Provider Self-Efficacy Scale

A retrospective Provider Self-Efficacy Scale was administered to all providers at post-intervention. This 16-item scale was used to examine providers’ perceptions of the impact of the PD intervention on their implementation of the EA-CP instructional strategies. Providers completed two sets of ratings after the PD program ended: one for their retrospective perceived self-efficacy before participating in the PD program and one for their perceived self-efficacy after participating in the PD program. Sample items included: (a) facilitate children’s expressive language development, (b) bait the classroom environment with opportunities to say and talk about target vocabulary words, and (c) use natural reinforcers. Each item was rated on a 3-point Likert-type scale: 1 = not at all confident, 2 = somewhat confident, and 3 = highly confident. A total score was calculated by summing all individual item scores (48 maximum points possible).

Child Outcome Measures

Social-communication Assessment in Book Sharing (SABS)

The SABS is an experimental measure developed to assess change over time in specific social communication behaviors in young children (Feuerstein & Landa, 2020). This assessment was administered individually at baseline and post-testing by research clinicians (with whom children were unfamiliar) in the participating child care centers and videorecorded for later coding. Examiners engaged each child in a shared book reading activity while administering five structured probes per each of six social-communication behaviors (initiation of joint attention, response to joint attention, motor imitation, expressive vocabulary, receptive vocabulary, and following directions in use of book-related object play). Examiners administered these probes at standardized pre-determined points in the story, pausing after each one to allow child initiation or response. Coding was performed by research assistants blinded to group and data point. Responses were scored as incorrect/correct (0, 1) for each of the 30 items, and a total score (maximum of 30 possible points) was calculated by summing the number of correct responses.

Provider Rating of Child Behavior During Book Sharing

The Provider Rating of Child Behavior During Book Sharing is a study-developed measure adapted from the Teacher Rating of Oral Language and Literacy (Dickinson et al., 2003). Providers completed this brief 7-item measure for each of the child participants in their classroom at baseline and post-training. This measure was designed to capture children’s engagement and enjoyment of shared book reading as well as their use of book-related words and ideas during the school day outside of shared book reading. Sample items included: (a) How often does the child join story time when stories are read in a group? and (b) How often does the child say words during non-book sharing activities during the school day that they heard during story times? Providers rated each item using a 3-point Likert scale (1 = rarely or never, one day a week or less; 2 = sometimes, some days in the week; 3 = often, nearly every day). A total score was calculated by summing each item score, for a maximum of 21 possible points.

Data Analysis

Data analyses were completed using IBM SPSS Statistics (Version 29). Mann Whitney U tests, a nonparametric method, were used to compare average differences in providers’ assessment results between groups due to the small sample sizes of providers in each group. Mann Whitney U tests were also used to compare average differences between the two groups of children with delays. Independent sample t-tests were used to make between-group comparisons of all children in each group. The alpha level was set at 0.05 for all analyses. Effect sizes were calculated using Hedges’ g (Hedges, 1981) due to small sample sizes (0.2 small effect, 0.5 medium effect, 0.8 large effect).

Results

Provider Measures

Provider Implementation: Book Sharing Fidelity

Providers in both groups implemented the EA-CP instructional strategies less than 25% of the time during group shared book reading time prior to the onset of training (see Table 3). At post-training, providers in both groups had made statistically significant gains, achieving over 85% fidelity. On average, providers in both groups demonstrated similar gains of 65% in book sharing fidelity scores; no between-group differences were detected using a Mann-Whitney test, U = 16.50, z = − 0.24, p = .81.

Table 3.

Provider within-group results

Group	Pre-training		Post-training
	M	SD	M	SD	t	df	p	g
	Provider Implementation Book Sharing Fidelity
BO	23.33%	13.32%	88.33%	6.00%	17.60	5	< 0.01	6.29
B + RA	21.04%	6.35%	86.04%	4.97%	15.63	5	< 0.01	11.40
	Provider Implementation Book-Related Activity Fidelity
BO	33.89%	12.37%	48.89%	18.09%	1.47	5	0.20	0.97
B + RA	47.78%	18.82%	90.56%	4.91%	5.01	5	< 0.01	3.11
	Video Knowledge Assessment
BO	72.62%	8.35%	88.10%	8.65%	3.31	5	0.02	1.82
B + RA	66.67%	5.83%	88.10%	12.50%	3.22	5	0.02	2.20
	Provider Self-Efficacy Scale
BO	36.33*	7.26	47.33	0.82	3.81	5	0.01	2.13
B + RA	31.83*	8.54	47.83	0.41	4.52	5	0.01	2.65

Note. N = 12 (n = 6 per group). BO = Book Only; B + RA = Book + Related Activity. *A retrospective pre-training rating was collected for the provider self-efficacy scale post-training

Provider Implementation: Book-Related Activity Fidelity

Providers in the B + RA group made statistically significant gains in implementation fidelity during the book-related activity, while those in the BO group did not (see Table 3). Comparison of the BO and B + RA providers’ fidelity (U = 7.00, z = -1.76, p = .08) revealed an average of 27.78% greater gain in fidelity in the B + RA than in the BO group from pre- to post-training.

Video Knowledge Assessment

At pre-training, providers in both groups scored below 75% accuracy. Providers in both groups made statistically significant gains from pre- to post-training (see Table 3). Providers in the BO and B + RA groups exhibited an average gain of 15.45% and 12.50%, respectively. No significant between-group difference was detected (U = 12.00, z = − 0.98, p = .33).

Provider Self-Efficacy Scale

Providers in both groups displayed statistically significant gains in self-efficacy from pre- to post-training, where pre-training ratings were made retrospectively (see Table 3). Providers in the B + RA group gained, on average, five points more than providers in the BO group, a non-significant difference (U = 14.00, z = -0.65, p = .52).

Child Measures

SABS

Change scores indicated that children in both groups improved significantly on the SABS from baseline to post-testing (BO: M = 7.00, SD = 4.66; B + RA: M = 7.70, SD = 4.33) (see Table 4), but no between-group differences were detected per an independent samples t-test, t(41) = − 0.51, p = .61, g = 0.16. Further, when comparing average gains between groups for children with delays (BO: M = 6.13, SD = 5.67; B + RA: M = 9.71, SD = 3.25), the result of the between-group comparison was not significant, U = 16.00, z = -1.40, p = .16.

Table 4.

Child within-group results

Group	Pre-testing		Post-testing
	M	SD	M	SD	t	df	p	g
	Social-communication Assessment in Book Sharing
BO	15.22	4.87	22.22	2.56	7.20	22	< 0.01	1.80
B + RA	9.15	5.92	16.85	5.63	7.95	19	< 0.01	1.33
	Provider Rating of Child Behavior During Book Sharing
BO	14.26	2.73	17.61	2.95	4.85	22	< 0.01	1.18
B + RA	14.30	3.63	17.15	3.15	4.90	19	< 0.01	0.84

Note. N = 43 (BO group: n = 23; B + RA group: and n = 20). BO = Book Only; B + RA = Book + Related Activity

Provider Rating of Child Behavior During Book Sharing

In both groups, there was a significant increase from baseline to post-testing in children’s engagement in book sharing and book-related language production per Provider Rating of Child Behavior During Book Sharing scores (see Table 4). There was no statistically significant between-group difference in children’s average gain scores (BO: M = 3.35, SD = 3.31; B + RA: M = 2.85, SD = 2.60) per an independent samples t-test, t(41) = 0.542, p = .59, g = 0.17. Similarly, when examining only children with delays, no significant between-group differences were detected (BO: M = 3, SD = 4.42; B + RA: M = 4.29, SD = 2.56), U = 20.50, z = -1.18, p = .24.

Discussion

The current study sought to preliminarily address gaps in the ECCE provider PD literature by focusing on providers’ implementation fidelity in a core instructional context as well as by examining a PD component designed to enhance their generalization of the instructional ingredients to a separate instructional context (Walker et al., 2020). Providers with and without generalization training demonstrated gains in implementation fidelity during book sharing, as well as in knowledge of evidence-based practices and in self-efficacy from pre- to post-training. Notably, however, providers whose training included an explicit generalization component (the B + RA group) demonstrated much larger gains in the generalization context, book-related activity implementation fidelity (average gain of 15% in the BO group and 42.78% in the B + BP group). Children in both groups showed significant gains during the provider training period but no significant between-group differences were detected.

Generalization Training

Our finding that providers without generalization training did not generalize the implementation of the EA-CP instructional strategies to a different instructional context, despite book-related procedural similarities to the primary training context, supports our hypothesis that providers receiving generalization training would attain greater implementation fidelity in their generalization activity. We propose that three EA-CP PD ingredients supported the B + RA group’s attainment of high fidelity in the generalization context. First, the generalization workshop focused solely on explicit discussion of the strategies to be embedded in the generalization context. This focus, along with role-playing paired with supportive and constructive feedback within the workshop, likely fostered providers’ metacognitive conceptualization of how to implement EA-CP strategies within the self-identified additional instructional context. Secondly, we systematically scaffolded for providers’ generalization success by first coaching them to fidelity within a focused and familiar routine instructional activity (book sharing), developing a strong base from which to transfer implementation skills to a new context. Finally, the strategically selected (by the research team) books may have afforded providers a salient, concrete, and visual pseudo script to enhance their ability to successfully: (a) embed book-related objects, (b) target children’s language and play skill goals, and (c) implement the EA-CP strategies in a generalization context (e.g., art activities, dramatic play).

The three ingredients discussed above align with four strategies from Stokes & Baer (1977) that are particularly relevant to research examining PD focused on teachers’ skill generalization (Markelz et al., 2017): sequential modification, training sufficient exemplars, programming common stimuli, and mediating generalization. The EA-CP generalization PD employed sequential modification (the modification of settings, responses, or students to provide contingencies similar to those during the initial training context) by first training (via workshops and coaching) in book sharing and then expanding to another classroom context of the providers’ choosing. We employed the train sufficient exemplars strategy by discussing and illustrating numerous generalization implementation exemplars in potential extension activities (e.g., art, dramatic play) during the generalization workshop and coaching sessions. To implement the program common stimuli strategy, we trained providers to incorporate the same stimuli across the shared book reading and generalization contexts (e.g., story mascot, story props, presence of the book). Coaches encouraged providers to mediate generalization of the EA-CP evidence-based strategies from the book sharing activity to the B + RA during debrief discussions by asking providers to identify which strategies were easiest and most challenging to implement.

This study adds evidence supporting inclusion of a generalization training component in educator PD to support providers’ ability to apply newly learned skills across contexts (e.g., Hemmeter et al., 2015). Specifically, generalization training should include explicit instruction via coaching across different activities (Hemmeter et al., 2015).

Impacts of Coaching

Our finding that individualized coaching and active learning supports ECCE providers’ gains in fidelity of implementation, knowledge, and self-efficacy aligns with previous research (e.g., Elek & Page 2019; Ottley et al., 2015; Piasta et al., 2017). Providers who received generalization training did not achieve larger gains in book sharing fidelity, knowledge, or self-efficacy in this short-term PD program as we hypothesized. Our results expand upon the extant literature by examining effects of providing coaching support in not just one, but two, instructional contexts. In the current study, coaches built collaborative partnerships with the providers to offer both supportive and constructive feedback, a key feature of effective PD (Elek & Page, 2019). These coaching principles likely played a major role in supporting the providers’ fidelity gains and facilitated their ability to individualize their use of the instructional strategies for their own unique classroom environments.

Child Outcomes

The significant gain in child performance detected in both groups over the 16-week PD period is a promising indicator that providers’ increased implementation of EA-CP instructional strategies and intentional targeting of specific social and language skills has positive effects on child development. The significant effects are compelling because provider-report results were corroborated by results of an examiner-administered behavioral measure where the examiner and materials were unfamiliar to children. Our findings of child gains contrast with reports from reviews and meta-analyses that providers’ participation in PD programs is associated with negligible to no gains in children’s performance (Markussen-Brown et al., 2017). This contrast in findings may be at least partly explained by our use of investigator-created measures focused specifically on child skills that providers were trained to target during the EA-CP PD program. More research is needed before causal inference is possible in interpreting the relation between the EA-CP PD and child gains since all children in the present study received instruction by EA-CP trained providers.

Lastly, our results did not indicate an additive benefit of provider generalization training for children’s short-term social or language outcomes. This may have been due to the brevity of the generalization training or to frequency with which providers implemented the strategies each week. The seven-week training period may have been an insufficient interval to permit detection of effects on child learning.

Limitations and Future Directions

Here we consider limitations of this study. First, group sample sizes were small in this pilot investigation, limiting power to detect statistically significant group differences and limiting generalizability of findings to other providers and children. Second, this study did not employ standardized child assessments due to their insensitivity to child change in short-term interventions. Our investigator-administered measures will continue to undergo psychometric evaluation to assess reliability and validity for use by other investigators. Third, the generalization training period was brief, making it difficult to assess whether additional provider training has value-added impact on child learning, and thus merits additional time and fiscal investment. Lastly, we did not collect sustainability data due to time and resources needed to adequately assess long-term impacts on providers’ instructional practices.

Future research should further examine different types and amounts of generalization training, as well as the timing of introducing generalization training during PD interventions. Further, researchers should work closely with early childhood programs when developing PD programs to maximize feasibility of implementation given providers’ many competing demands.

Conclusion

This study extends the current PD literature by examining two active intervention conditions within one study (as opposed to a training and control group) distinguished by the inclusion of an explicit generalization component in one condition. Preliminary findings demonstrate promise that the EA-CP PD program positively affects providers’ implementation of evidence-based instructional strategies and that doing so is associated with children’s learning gains. In additions, findings indicate that providers require explicit generalization training across multiple classroom contexts during PD programs to apply newly learned evidence-based practices to multiple classroom contexts. Including this generalization training may have important implications for providers’ sustainability of implementation of newly learned instructional practices (Burns et al., 2013). Numerous factors must be considered when determining whether to invest additional time and fiscal resources into generalization training for ECCE providers, especially since the generalization training did not differentially impact provider fidelity in the primary training context (book sharing), knowledge or self-efficacy improvement, or children’s learning gains; more research, including a focus on frequency of providers’ implementation of the instructional strategies in the generalization context, is warranted. The differences found in providers’ generalization of skills in the present study informed refinement of the EA-CP PD program for our larger, iterative development study, that culminates in a pilot randomized control trial to permit causal inference regarding effects of the EA-CP PD on provider implementation fidelity and on child learning outcomes.

Availability of data and material

Participants of this study did not agree for their data to be shared publicly, so supporting data are not available.

Declaration of Interest Statement

We have no relevant financial or nonfinancial interests to disclose.