A Pre-Implementation Enhancement Strategy to Increase the Yield of Training and Consultation for School-Based Behavioral Preventive Practices: a Triple-Blind Randomized Controlled Trial

Abstract

As the most common setting where youth access behavioral healthcare, the education sector frequently employs training and follow-up consultation as cornerstone implementation strategies to promote the implementation of evidence-based practices (EBPs). However, these strategies alone are not sufficient to promote desirable implementation (e.g., intervention fidelity) and youth behavioral outcomes (e.g., mitigated externalizing behaviors). Theory-informed pragmatic pre-implementation enhancement strategies (PIES) are needed to prevent the lackluster outcomes of training and consultation. Specifically, social cognitive theory explicates principles that inform the design of PIES content and specify mechanisms of behavior change (e.g., “intentions to implement” (ITI)) to target increasing providers’ responsiveness to training and consultation. This triple-blind parallel randomized controlled trial preliminarily examined the efficacy of a pragmatic PIES based on social cognitive theories (SC-PIES) to improve implementation and youth behavioral outcomes from universal preventive EBPs in the education sector. Teachers from a diverse urban district were recruited and randomly assigned to the treatment (SC-PIES; n_treatment = 22) or active control condition (administrative meeting; n_control = 21). Based on the condition assigned, teachers received the SC-PIES or met with administrators before their EBP training. We assessed teachers’ ITI, intervention fidelity, and youth behavioral outcome (academic engagement as an incompatible behavior to externalizing disorders) at baseline, immediately after training, and 6 weeks afterward. A series of ANCOVAs detected sizeable effects of SC-PIES, where teachers who received SC-PIES demonstrated significantly larger improvement in their ITI, intervention fidelity, and youth behaviors as compared to the control. Conditional analyses indicated that teachers’ ITI partially mediated the effect of SC-PIES on intervention fidelity, which in turn led to improved youth behaviors. Findings suggest that theory-informed pragmatic PIES targeting providers’ ITI can boost their responsiveness to implementation strategies, as reflected in improved implementation behaviors and youth behavioral outcomes. The results have implications for targeting motivational mechanisms of behavior change and situating preventive implementation strategies at the intersection between the preparation and active implementation stages of an implementation process. Limitations and implications for research and practice are discussed. Clinicaltrials.gov: NCT05240222. Registered on: 2/14/2022. Retrospectively registered. https://clinicaltrials.gov/show/NCT05240222

Background

Across virtually every service sector, training and follow-up consultation are cornerstone implementation strategies designed to promote provider uptake and use of evidence-based practices (EBPs; Lyon et al., 2017). While training and consultation are essential implementation strategies, there is mounting evidence indicating that these strategies alone are insufficient to translate what is known to work in research into routine practices that ultimately improve service recipient outcomes (Edmunds et at., 2013; Powell et al., 2017). Pre-implementation enhancement strategies (PIES) that complement and increase the yield of training and follow-up consultation are needed, particularly ones that are pragmatic and theoretically informed to target precise mechanisms of behavior change (McCleod et al., 2018; Lyon et al., 2019). PIES that bolster the yield of implementation strategies are especially needed in the context of youth behavioral health where many youths in need of care do not access quality prevention or treatment services even though numerous EBPs exist (Phillippi et al., 2020; Williams & Beidas, 2019). The education sector is one of the best settings to develop and test PIES as it is the most common setting where youth access behavioral health services (Duong et al., 2021; Fazel et al., 2014; Weist et al., 2017). With a triple-blind parallel randomized controlled trial, this study aimed to test a pragmatic PIES designed with principles rooted in social cognitive theories (SC-PIES) as a preventive complement to EBP training and follow-up consultation. The goal was to enhance teachers’ intentions to implement and then subsequent implementation in the service of improving youth behavioral health outcomes in the education sector.

Youth Behavioral Health and EBPs in the Education Sector

As a prevalent and impactful youth behavioral health problem, externalizing behavioral disorder and its prevention and treatment remain top priorities for policymakers, researchers, and healthcare professionals in the education sector (OSG, 2022). Externalizing behavioral health problems are outer-directed behaviors that harm relationships with others and violate social norms and rules (Splett et al., 2019). Youth who exhibit externalizing behavioral problems are likely to experience negative outcomes, including academic difficulties (Lee & Bierman, 2015), strained relationships with others (Leflot et al., 2011), elevated risk for dropout (Bevilacqua et al., 2018), substance abuse (Regan et al., 2020), repeated exposure to exclusionary discipline practices, and contact with the juvenile justice system (Mitchell & Bradshaw, 2013). To prevent these untoward outcomes, researchers have established several EBPs for use in the educational sector. For example, proactive classroom behavior management (PCBM) includes a suite of EBPs that are prevention-oriented and aim to promote high levels of behavioral academic engagement in class as incompatible with externalizing behavior problems (Rathvon, 2008). Extant research has established the effectiveness of various PCBM practices for consistent use by teachers in the context of classrooms where youth spend significant amounts of time (Simonsen et al., 2008). PCBM practices promote youth’s behavioral and academic success in classrooms which leads to short-term success in school (e.g., reduced externalizing behaviors, increased academic achievement, and school engagement; Cook et al., 2018; Nagro et al., 2019). These short-term outcomes are determinants of longer-term beneficial outcomes that reach into adulthood (Rodwell et al., 2018). Although PCBM has been disseminated through multiple outlets (e.g., refereed journals, books, and media outlets), passive implementation efforts have proven insufficient to produce scalable changes in teachers’ delivery of PCBM (Brownson et al., 2018).

Teachers as Primary Implementers with Varied Response to Training and Consultation

In the education sector, the implementation of universal prevention-oriented EBPs targeting youth behavioral health outcomes is primarily the responsibility of classroom teachers who spend the most time during the school days interfacing with youth (Lyon et al., 2018). For this reason, teachers are the recipients of training and consultation, which are the most common implementation strategies used to promote teachers’ uptake and delivery of universal EBPs (Edmunds et al., 2013). While training and follow-up consultation are considered core implementation strategies, these strategies alone are insufficient to produce quality and consistent implementation across practitioners. For instance, didactic training is necessary to increase knowledge of practices, but they are ineffective alone in producing changes in teachers’ classroom practices (Robertson et al., 2021). Moreover, teachers may demonstrate resistance or ambivalence to change during follow-up support (e.g., consultation; Lyon et al., 2019).

As primary implementers, teachers’ motivation plays an instrumental role in determining whether they will respond to training and follow-up consultation to initially adopt and then persist in using an EBP with fidelity (Dusenbury et al., 2005). Indeed, teachers are a heterogeneous group of implementers who vary significantly according to motivational factors that influence whether they are likely to adopt new practices as part of their regular classroom routines (Abry et al., 2013; Dickie & Shuker, 2014). Variability in motivational determinants of change exists regardless of whether teachers work in school settings with optimal organizational characteristics in place (e.g., supportive leadership, quality training, and coaching; Kincaid et al., 2007). Therefore, there is a need for complementary pre-implementation strategies that occur before training and consultation to target motivational determinants of behavior change to enhance teachers’ responsiveness to training and consultation (Low et al., 2016).

Preventive Pre-Implementation Enhancement Strategies for Training and Consultation

Theory-informed and pragmatic PIES offer promise as preventive complementary strategies that can increase the yield of training and follow-up consultation on both implementation and youth behavioral health outcomes. PIES occur before more active implementation supports such as training and consultation. EBP training and consultation more directly focus on increasing providers’ knowledge of and following through with specific practices. A common implicit assumption of training and consultation is that providers are already motivated to change their behaviors. However, this is not always the case. PIES occur before providers participate in training and consultation activities and are designed to motivate and prime providers to engage and respond more to active implementation strategies. Essentially, PIES sit at the immediate intersection of the preparation and active implementation phases of an implementation process (Lyon & Bruns, 2019).

Consistent with recommendations for developing implementation strategies (Cook et al., 2019), the development of PIES should be grounded in theory to ensure the content targets precise motivational mechanisms of behavior changes before receipt of training and consultation. Theoretically, effective PIES can enhance providers’ motivational mechanisms (e.g., intention to implement), which in turn improve their responsiveness to EBP-specific implementation strategies (e.g., training and consultation) that lead to enhanced implementation outcomes (e.g., adoption, fidelity) that ultimately optimize youth behavioral outcomes.

Use of Social Cognitive Theory to Develop and Test PIES

Applied social cognitive theory and research offer principles and evidence that could inform the development of pragmatic and effective PIES (Steinmetz et al., 2016). In the context of implementation, strategies informed by social cognitive theory could serve to alter teachers’ perceptions and motivation regarding the uptake and use of EBPs, in the context of actively acquiring knowledge about them (training) and participating in ongoing support and feedback to deliver them with fidelity (consultation). Social cognitive research suggests that implementers’ behavioral intentions are malleable mechanisms of behavior change that can be targeted via intervention (Cialdini & Goldstein, 2004). The theory of planned behavior (TPB; Birken et al., 2020) is a widely established social cognitive theory that has been used to predict and target behavior change (Godin et al., 2008). The central tenet of the TPB is that one of the best predictors of behavior is a person’s behavioral intentions. Behavioral intentions “capture the motivational factors that influence a behavior; they are indicators of how hard people are willing to try, of how much effort they are planning to exert, to perform the behavior” (Godin et al., 2008). Previous research using the TPB has shown that implementation strategies targeting implementers’ behavioral intentions are linked to improved implementation outcomes during the active implementation phase when EBP uptake and use are critical (Fishman et al., 2021; Kortteisto et al., 2010; Mangurian et al., 2017). Moreover, a study conducted with teachers serving youth with autism indicated that, following in-service training, teachers endorsing high intentions to implement were five times more likely to adopt and deliver EBPs than teachers endorsing low intentions to implement (Fishman et al., 2021). As such, implementation strategies that target behavioral intentions before initiating active implementation may serve as an effective approach to increasing teachers’ responsiveness to training and consultation.

Several empirically established social cognitive principles can be used to guide the content development of PIES (Birken et al., 2020; Fishman et al., 2021). We strategically selected three social cognitive principles based on two criteria: (1) empirical evidence related to behavior change and (2) ease of designing content and activities that are perceived as acceptable and pragmatic by teachers. First, the principle of growth mindset was selected as it is a popularized construct that has gained widespread attention in education and can be leveraged for purposes of face validity to engage teachers (Yeager & Dweck, 2012). Research has shown that people’s mindsets or implicit theories about the malleability of their cognitive and physical abilities have significant impacts on their goals, effort, and performance (Blackwell et al., 2007).

Second, the saying-is-believing principle was selected, because it involves individuals advocating for an idea or action to others to increase their commitment to the idea or action regardless of their initial beliefs about it (Aronson et al., 2002). For instance, college youth randomly assigned to write letters to incoming youth endorsing the importance of overcoming social and academic adversity and using problem-solving strategies to do so showed significant improvements in their school-related behaviors and achievement (Walton & Cohen, 2007). Teachers who are provided with opportunities to advocate ideas relevant to adopting and implementing new practices may in turn exhibit greater intentions to implement and are more likely to implement after receiving training and follow-up consultation (Cook et al., 2015).

Last, we selected the principle of commitment and consistency (Wood, 2000), which involves evoking a state of psychological and emotional tension that gets activated when individuals recognize a discrepancy between a belief and their behavior. The tension and discrepancy lead to an increased likelihood of individuals striving to maintain consistency between their beliefs and actions (Wood, 2000). Specific desired behaviors (i.e., intervention fidelity) can be increased by evoking commitments that are active, public, and voluntary (Cialdini, 2001). Once individuals make a commitment, they are more motivated to maintain consistency between their beliefs and actions (i.e., commitment or follow through with the stated action). Research has leveraged this principle to induce cognitive dissonance to promote behavior change (Pratkanis, 2011). This technique has been applied to a wide range of behaviors, including voting behavior, fund-raising, and recycling (Spangenberg & Greenwald, 2001; Filter & Brown, 2019).

Gaps in Current Literature and Study Aims

There is limited research on the development and evaluation of PIES as complementary strategies to prevent providers’ non-responsiveness to training and consultation, as well as to increase their yield on both implementation and youth behavioral health outcomes. Moreover, the literature on implementation strategies from other service sectors has largely neglected the specification of change mechanisms by which strategies impact implementation and client outcomes (Lewis et al., 2018). A focus on theory-informed mechanisms as mediators of behavior change is vital to identifying why and how implementation strategies (e.g., PIES) enhance the yield of prevention programs, which can subsequently inform the development, testing, and refinement of high-quality implementation strategies and prevention programming (Birken et al., 2020; Kazdin, 2007). Such inquiries must be informed by established theories to explain how strategies at different implementation stages (e.g., pre-implementation vs active implementation) influence implementation and youth behavioral health outcomes (Proctor et al., 2013).

This study aimed to experimentally examine the efficacy of pragmatic PIES based on social cognitive theories (SC-PIES). The SC-PIES was hypothesized to influence teachers’ intentions to implement EBPs, which would in turn improve their intervention fidelity of EBPs and youth academic engagement. Specifically, we conducted a preliminary triple-blind parallel randomized controlled trial (RCT) to address four research questions: (1) As compared to control, does SC-PIES improve teachers’ intentions to implement (ITI) PCBM practices at posttest after adjusting baseline and covariates? (2) Compared to control, does SC-PIES improve teachers’ intervention fidelity and youth academic engagement at the 6-week follow-up after adjusting baseline and covariates? (3) Does teachers’ ITI mediate the association between study condition and intervention fidelity? (4) Does teachers’ intervention fidelity mediate the association between their ITI and youth academic engagement?

Method

Setting and Participants

This study took place at two elementary schools in a large and diverse urban district in the Pacific Northwest. The district was in the first year of a district-wide implementation initiative for a universal prevention program for youth behavioral health. The two participating schools had no experience implementing universal prevention programs targeting youth behaviors before the study. General education teachers were recruited if they have limited training and implementation experience about proactive classroom behavioral management strategies (PCBM) before this study (n_eligible = 56). Among the 43 teachers consented to participate, the majority were White (n = 29; 67.44%) and female (n = 35; 81.40%). The average teaching experience was 8.7 years (SD = 6.4), and about a third had a master’s degree (32.55%; Table 1). Twenty-two teachers reported taking one pre-service training about PCBM, while the remaining had no former training/experience in PCBM. The participant demographics were consistent with the education literature (Christofferson & Sullivan, 2015).

Table 1.

Descriptive statistics of participating teacher demographics (N = 43)

Variables	Categories	N	%
Gender	Female	35	81.40
	Male	8	18.60
Grade level	Unknown	8	18.60
	1.00	7	16.28
	2.00	7	16.28
	3.00	7	16.28
	4.00	7	16.28
	5.00	7	16.28
Ethnicity	White	29	67.44
	Black	6	13.95
	Latino	5	11.63
	Asian	3	6.98
Teaching experience (years)	≤ 5	17	39.53
	6 to 9	12	27.91
	> 10	14	32.56
Condition	Active control	21	48.84
	Treatment	22	51.16

Procedures

Recruitment, Randomization, and Data Collection

This study was approved by the university IRB and the evaluation department of the partnering district. A single-level two-arm RCT was selected because the theoretical change mechanism of SC-PIES, treatment, and outcomes of interests were all at the teacher/classroom level (Taljaard et al., 2020). First, school administrators introduced this study to teachers and connect those interested to the authors. Forty-three teachers consented to participate, while 13 did not due to random reasons (e.g., job changes, workload, and turnover), which are common in the educational sector. With an online randomization tool (Uribank & Plous, 2013), a graduate assistant randomly assigned teachers to the treatment (n_treatment = 22) or control condition (n_control = 21). The assistant did not participate in data collection, analysis, or reporting. The triple-blind design ensured that none of the participants, researchers, or observers who assessed outcomes were aware of the treatment allocation (i.e., allocation concealment). This study rolled out in multiple steps in the fall semester as depicted in the CONSORT diagram (Fig. 1; Moher et al., 2012). We followed the STARi and CONSORT checklists in reporting (Online Resources 2 and 3). To enable proper causal inferences, we separated and sequenced the study components into (1) baseline pretest, (2) SC-PIES, (3) PCBM training, (4) immediate posttest of change mechanism (intention), (5) follow-up consultation, and (6) 6-week follow-up test (Fig. 1).

Fig. 1.

The CONSORT diagram delineates the study design timeline of treatment (P-PIES), assessment, and training, consultation, as well as pre-/post-/follow-up tests and variables assessed at each time point. ITI, teachers’ intention to implement; FI, intervention fidelity; AET, class-wide youth academic engaged time. The brackets on the right indicate specific implementation stages that correspond to each step in this study

Pretest data were collected from all participants about their intentions to implement EBPs (ITI), intervention fidelity, and class-wide youth academic engaged time (AET). Then, the treatment group received the SC-PIES, and the control group met with their school administrators. Two days after the SC-PIES or meeting with administrators, participants from both conditions received a 2-day training (2 h per day) about proactive classroom management strategies (PCBMs; Online Resource 1). The 2-day delay between SC-PIES/meeting with administrators and PCBM training was an intentional part of the experimental design to ensure adequate time for the public commitment postings to be displayed and internalized by teachers receiving SC-PIES (see SC-PIES procedure). The delay isolated and prevented potential confounding between the public commitment posting activity and PCBM training. The PCBM training sessions followed a tell-show-do approach, where the trainer detailed the core components of PCBM to participating teachers, modeled implementation of these core components, then led teachers to plan how to implement PCBM in their classrooms and rehearse. In the end, teachers received one-on-one performance-based feedback from the trainer. After the PCBM training, the posttest (ITI only) was conducted immediately because social cognitive theories support that implementers’ intention, as a predictor of their subsequent behavioral change, can be most effectively assessed before their actual enactment of implementation behaviors (Fishman et al., 2021). Following the PCBM training, the authors provided two follow-up consultations to teachers (1- and 4-week). Follow-up data (fidelity and AET) was collected after teachers completed the training and implemented PCBM for 6 weeks.

Control Condition: Meeting with Administrators

Teachers in the control condition met with their school administrators to discuss the math/reading curriculum, teaching schedule, and logistics. The meeting was of the same duration as the SC-PIES (1 h) to control for dosage. The meeting served as attention control and intentionally precluded any active components of the SC-PIES.

Treatment Condition: SC-PIES

The SC-PIES is a 1-h professional development for teachers before attending the training about PCBM. To maximize the pragmatism of SC-PIES, all content and activities were streamlined to be deliverable in 1 h. The SC-PIES was grounded in three social cognitive principles: (a) growth mindset, (b) saying-is-believing, and (c) commitment and consistency.

Growth Mindset

Consistent with prior research, growth mindset was not directly implicated as something teachers need to acquire and exhibit. Instead, the growth mindset component of SC-PIES was modeled after prior research (Larson et al., 2021), where teachers learned about neuroplasticity (i.e., the brain can change with effort and persistence to engage in growth-oriented experiences) as it relates to youth development. Specifically, teachers first watched two videos and then were prompted to engage in guided reflective discussion with colleagues. During the discussion, teachers were encouraged to (a) explore the implications of neuroplasticity for promoting youth’s growth, enabling them to persevere in the face of adversity when learning new knowledge and skills, and (b) discuss ways they could encourage youth to internalize the notion of neuroplasticity. Next, the facilitator led a whole-group discussion with teachers to share their responses. Overall, the growth mindset component of SC-PIES attempted to instill in teachers a practice-oriented notion of growth mindset toward PCBM, where we encouraged teachers to act as role models for youth through their efforts to learn new things and persevere through adversity.

Saying-Is-Believing

Building on the growth mindset component and prior research (Aronson et al., 2002), the saying-is-believing component of the SC-PIES involved teachers writing letters to new teachers in which they explained the concept of neuroplasticity and illustrated the concept by describing how they were able to overcome challenges as a new teacher to learn new practices and better manage the conditions of leading a classroom. Specifically, they described a real-life scenario where they experienced adversity as new teachers but were able to overcome it by stretching themselves and putting in the effort to learn and apply a new practice. This activity puts teachers in the role of mentors for new teachers and encouraged them to advocate a message about learning new things and expressing the positive outcomes associated with persistence. Teachers were informed that these letters would be shared with the new teachers to normalize common difficulties and emphasize the importance of effort and perseverance to learn new skills over time as a teacher.

Commitment and Consistency

This component involved encouraging teachers to make public commitments regarding supporting youth’s growth mindsets. Based on the commitment and consistency principles (Cialdini & Goldstein, 2004), this strategy hypothesizes that once individuals have made a public and explicit commitment, their beliefs tend to shift to maintain consistency between their commitment and actions. Teachers were asked to publicly sign a billboard that would be posted for youth and parents to see that read, “At our school, we believe in our youth’s ability to grow their brains and academic skills through hard work, putting in extra effort to learn new skills, seeking assistance when confronted with challenges and persevering in the face of failure through the feedback we provide them and modeling effort and perseverance ourselves.” In addition to posting the billboard in a highly visible area near the school office, a photocopy of the signed billboard was sent out to every teacher in the treatment condition to post it as a reminder in their immediate environments (e.g., office or classrooms).

Example Preventive EBP: The PCBM

Based on a consensus-driven approach facilitated by school administrators, school staff selected four evidence-based preventive PCBM practices as non-negotiables to implement across every classroom (Online Resource 1), including (1) greeting youth at the door, (2) behavior-specific praise, (3) providing numerous opportunities to respond, and (4) post, teach, review, and provide feedback about positively stated behavioral expectations (PTRP). Teachers implemented the PCBM for 6 weeks in their classrooms before assessing fidelity and student outcome.

Measures

Intentions to Implement

The Modified Intentions to Use Scale was adopted to assess teachers’ intentions to implement new EBPs (Kortteisto et al., 2010). The scale is originally developed from research on practitioners’ intention to adhere to measurement-based care and is consistent with the recommendations for assessing behavioral intentions based on the theory of planned behavior (Godin & Kok, 1996). The scale consists of five items on a 7-point Likert scale ranging from “greatly disagree” to “greatly agree.” The items were modified to specifically assess teachers’ intentions to implement EBPs. The scale showed acceptable reliability and validity in prior research (Williams, 2015) and the current study (α = 0.75; Larson et al., 2021).

Intervention Fidelity

A team of trained professionals, who were blinded to the random assignment results, conducted structured observations to assess intervention fidelity, which is operationalized in this study as “adherence to core components of the classroom management practices.” The data collection took place at the pretest and 6-week follow-up to allow enough time to detect and compare changes in fidelity. First, during core instructional times, trained observers observed two occasions at pretest and 6-week follow-up to code the presence or absence of the core components of the PCBM. Immediately following the observations, these observers completed a fidelity rating rubric, which was created by operationalizing three core components of each PCBM such that observers could reliably observe and rate them (Sanetti & Kratochwill, 2009). The average of the observation coding and the rubric scores represents each teacher’s fidelity. Adequate inter-rater reliability (Spearman’s r) was found for all PCBMs (Mean r = 0.79), which was based on ratings from two observers for the same teacher for 20% of the same occasion.

Class-Wide AET

AET was defined as “any instance where youth are attending to instruction, watching the teacher or speaker, or concentrating on their classwork.” This study used direct behavior ratings (DBRs) for AET with which a teacher observes a youth’s AET throughout a predetermined interval. Teacher-completed DBRs had strong correlations to structured direct observations in assessing AET (r = 0.81; Chafouleas, 2011). The DBRs were completed at pretest and 6-week follow-up. For comparability across classrooms, each teacher completed DBRs during the language arts block at the same time each weekday for five randomly selected youths, whose results were averaged for that class. This approach has demonstrated adequate reliability and validity to capture class-wide AET (Chafouleas, 2011). To secure the temporal sequences in the mediation analyses, the AET was measured after intervention fidelity at 6-week follow-up.

Data Analysis

First, assumptions for all planned analyses were tested and generally met (e.g., normality of residuals and homoscedasticity). Then, Chi-square tests and independent samples t-tests were performed to assess baseline equivalences between the study conditions in the pretest values of teachers’ demographics (experience, grade, ethnicity, and gender) and three dependent variables (DVs; teachers’ ITI, intervention fidelity, and class-wide AET). For RQs 1 and 2, we used ANCOVA to estimate the treatment effect of SC-PIES on each of the three DVs. The ANCOVA models were configured the same way where the post-treatment score of a DV was regressed on the binary variable of study condition (reference category = control) as the focal predictor while controlling for the pretest score of the DV and demographics as covariates. The literature generally supported ANCOVA as the most appropriate and informative treatment effect analytic approach for pre/post control group designs because it can enable (a) adjustment of the pretest score as a covariate because it is not an outcome by definition, (b) convenient and interpretable adjustment of other covariates at baseline, and (c) handling of covariance matrix heterogeneity of parameters (Wan, 2021). To adjust for the bias caused by the covariance matrix heterogeneity in relatively small samples, we used the heteroskedasticity-consistent standard error estimator (HC4; Hayes & Cai, 2007). Given the relatively small sample and multiple tests performed, traditional null hypothesis tests with p-values are suboptimal (i.e., inflated type I error). The FDR-corrected p-values (false discovery rate; q-values) were estimated to correct for potential false positives with an intended level of significance corresponding to q = 0.05 (Benjamini & Hochberg, 1995; Wason & Robertson, 2021). Power analysis was performed with the G*Power version 3.19 (Faul et al., 2007). Given n = 43, six predictors, and alpha level of 0.05, we will have sufficient power (≥0.80) to detect an effect size (Cohen’s d) as small as 0.19.

To examine the hypothesized change mechanisms of SC-PIES (RQs 3 and 4), we sequentially explored the mediation effect of ITI on the relationship between study conditions and intervention fidelity, followed by a secondary model where teachers’ fidelity was hypothesized to mediate the relationships between their ITI and class-wide AET. To control for baseline status, the mediation models were fitted with change scores that were calculated by subtracting pretest values from the posttest (Vandenberghe et al., 2017). The nonparametric bootstrapping mediation analysis was used rather than the causal steps approach (Baron & Kenny, 1986) because of its superior capacities to handle small samples, estimate robust standard errors, and detect mediation effects without significant total effects (Hayes, 2017). We interpreted results based on statistical significances and standardized mean differences with pooled pretest standard deviations (SMDES; Morris, 2008). The SMDES is interpreted as the units of standard deviations of the mean difference between the pretest vs posttest changes of teachers’ outcomes in the treatment condition and those in the control. All analyses were performed with SPSS version 26.

Results

Treatment Effects of SC-PIES (RQs 1 and 2)

The t-tests confirmed that the random assignment created probabilistically equivalent groups in terms of the baseline status of key variables (Table 2). For the ANCOVA on teachers’ ITI (RQ 1; Table 3), results showed a significant positive effect of SC-PIES where the post-treatment scores of teachers’ ITI in the treatment condition were significantly higher than the scores of those in the control (b = 2.18, p < 0.01). The standardized mean difference effect size (SMDES) was 0.40. Of note, teachers’ work experience (b = −0.13, p < 0.05) was a significant predictor of post-treatment scores of their ITI, which suggests that 1-year increase in experience is associated with a 0.13 unit decrease in the scores of teachers’ ITI controlling for the other covariates in the model. For the ANCOVA on intervention fidelity (RQ 2; Table 3), results indicated that the intervention fidelity of teachers in the treatment condition was significantly higher than the fidelity of those in the control at 6-week follow-up (b = 2.39, p < 0.001). The SMDES for fidelity was 0.91. For class-wide AET (RQ 2; Table 3), results indicate that youth’s AET in the classrooms of the teachers from the treatment condition was significantly higher than that of the teachers from the control (b = 0.06, p < 0.01). The SMDES for AET is 0.47. No demographic covariates were significant for either fidelity or AET.

Table 2.

Independent samples t-tests of pretests of variables between treatment and control conditions

	Variables	Mean		SD		t-test for equality of means			Levene’s test for equality of variances
		Ctrl	Trt	Ctrl	Trt	t	df	p	F	p
Process and outcome variables	Intentions to implement	20.57	20.23	5.69	5.50	−0.20	41	.84	0.01	.93
	Intervention fidelity	7.90	7.59	2.84	2.63	−0.38	41	.71	0.15	.70
	Academic engaged time	0.68	0.72	0.12	0.10	1.12	41	.27	0.21	.65
Teacher demographics	Years of teaching experience	8.71	8.68	6.73	7.17	−0.02	41	.99	0.08	.78
	Grade	Pearson Chi-square		df		Asymptotic significance (2-sided)
		2.98		5		.70
	Ethnicity	0.55		3		.91
	Gender	0.51		1		.48

Trt, treatment condition (P-PIES), Ctrl active control condition, SD standard deviation

Table 3.

ANCOVAs controlling for pretest values and demographics as covariates

Outcomes	Predictors	b	Beta	HC-SE	t	p	q	ES	Adj. R2	F (p)
Intentions to implement (ITI; posttest)	Intercept	7.52	-	1.95	3.85	< .001	.01*	-	.81	35.07 (< .001)
	Treatment	2.18	.23	0.65	3.34	< .01	.02*	.40
	ITI (pretest)	0.71	.83	0.07	9.84	< .001	.01*	-
	Experience	−0.13	−.19	0.05	−2.54	<.05	.03*	-
	Grade	0.04	.01	0.18	0.21	.84	.04	-
	Ethnicity	−0.05	−.01	0.33	−0.14	.89	.05	-
	Gender	−0.95	−.08	0.79	−1.2	.24	.04	-
PCBM fidelity (posttest)	Intercept	3.78	-	0.97	3.9	< .001	.02*	-	.67	24.01 (< .001)
	Treatment	2.39	.43	0.53	4.51	< .001	.01*	.91
	PCBM fidelity (pretest)	0.71	.69	0.09	7.77	< .001	.01*	-
	Experience	−0.07	−.17	0.05	−1.44	.16	.03	-
	Grade	−0.01	−.01	0.14	−0.1	.92	.05	-
	Ethnicity	−0.11	−.04	0.32	−0.35	.73	.04	-
	Gender	−0.18	−.03	0.8	−0.22	.82	.04	-
Academic engaged time (AET; posttest)	Intercept	0.24	-	0.07	3.48	< .01	.01*	-	.67	16.02 (< .001)
	Treatment	0.06	.28	0.02	3.03	<.01	.02*	.47
	AET (pretest)	0.69	.66	0.1	6.69	< .001	.01*	-
	Experience	−0.002	−.15	0	−1.11	.27	.03	-
	Grade	0.004	.06	0.01	0.50	.62	.04	-
	Ethnicity	0.01	.08	0.01	0.77	.45	.04	-
	Gender	−0.01	−.04	0.05	−0.25	.81	.05	-

q = the FDR-corrected p-value for the corresponding hypothesis test for a variable.

The asterisk (*) following a q value indicates that, for this test, the p-value is less than the q, which suggests a statistically significant result after FDR adjustment.

Treatment = study conditions (reference = the attention control condition), b = unstandardized coefficients, Beta = standardized coefficients, HC-SE heteroscedasticity-consistent standard error, “-” = not available, Adj. R² adjusted R square

Mediational Mechanisms of Change (RQs 3 and 4)

For teachers’ ITI, we first fitted Model 1 by entering the binary variable of study condition as the predictor (reference = control), the change score of ITI as the mediator, and the change score of intervention fidelity as the outcome (Table 4; Fig. 2A). Results indicated that SC-PIES exerted a significant direct effect (DE) on the changes in teachers’ fidelity (DE = 1.21, p < 0.05). Also, there was a positive indirect/mediation effect (IE) where changes in teachers’ ITI partially mediated the relationship between study condition and fidelity (IE = 1.04, 95% CI = [0.38, 1.79]). This finding suggested that teachers who received the SC-PIES demonstrated greater improvement in their intentions to implement PCBM, which in turn enhanced their fidelity of PCBM, as compared to those in the control condition. For Model 2 (Table 4; Fig. 2B), resrdts indicated that teachers’ fidelity was a significant positive mediator between their ITI and youth’ AET (IE = 0.02, 95% CI = [0.01, 0.03]). However, the total and direct effects of teachers’ ITI on their youth’ AET were non-significant (TE = 0.01, p > 0.05; DE = −0.01, p < 0.001). This constitutes the phenomenon called “inconsistent mediation,” where a significant indirect effect is present with a nonsignificant total effect because the direct and indirect effects possess opposite directions and cancel each other out (MacKinnon et al., 2000; Altikriti, 2022).

Table 4.

Configurations and results of the mediation models

Model #	IV	M	DV	Direct effect	Indirect/mediation effect	95% CI of mediation effect	Total effect	Mediation type
1	Trt	ITI	Fid	1.207* (0.497)	1.036*	0.377, 1.791	2.242*** (0.61)	Partial mediation
2	ITI	Fid	AET	−0.014*** (0.004)	0.019*	0.012, 0.027	0.005 (0.003)	Inconsistent mediation

n = 43,

^*p < .05,

^**p < .01,

^***p < .001.

IV = predictor, M = mediator, DV = outcome, ITI = change score of teachers’ intentions to implement EBPs. Fid = change score of intervention fidelity of PCBM. Trt = treatment vs control conditions. Statistics in the parentheses are standard errors. A mediation effect is considered significant if its 95% confidence interval based on 5000 bias corrected bootstraps (CI; two tailed) does not include 0. Three decimals are reported to enhance precision

Fig. 2.

*p < .05 **p < .01, ***p < .001. N = 43. Intentions to implement, intervention fidelity, and AET are change scores calculated by subtracting pretest score from posttest to account for baseline status. AET, academic engaged time. All beta weights are based on 5000 bias corrected bootstrap estimations, βa is the beta weight of the predictor regressing on the mediator, and βb is the beta weight for the mediator regressing on the outcome, βc is the beta weight for the predictor when the mediator is controlled (direct effect), while βd is the beta weight for the predictor when the mediator is not controlled (total effect)

Discussion

There is a need for theory-informed PIES to prevent lackluster implementation and youth outcomes that result from training and follow-up consultation about universal preventive practices in school-based youth behavioral healthcare. This study offers preliminary evidence for PIES, which is based on social cognitive theories (SC-PIES) and designed to be pragmatic (e.g., 60-min group delivery). PIES preventively target a motivational mechanism of behavior change (ITI) before school-based service providers receive EBP-specific training and follow-up consultation. Using a triple-blind parallel randomized controlled trial, this study found that the SC-PIES was effective at boosting teachers’ responsiveness to EBP-specific training and consultation as evidenced by improved ITI, intervention fidelity, and youth behavioral health outcomes (i.e., behavioral academic engagement as an incompatible response to externalizing problems).

Our findings preliminarily supported that implementers’ intention to implement can be an effective and amendable change mechanism. The SC-PIES appears to function by first enhancing teachers’ intentions to implement EBPs, which in turn leads to higher levels of intervention fidelity. Higher levels of intervention fidelity were then associated with improvements in a youth behavioral health outcome (i.e., behavioral academic engagement, AET). The findings are consistent with previous research on the theory of planned behavior indicating implementers’ intentions as a potential change mechanism that leads to increased intervention fidelity (Fishman et al., 2021). The significant mediation effect implied that teachers’ intentions to implement new EBPs are important to target before receiving training and consultation about those EBPs.

The inconsistent mediation effect of fidelity (MacKinnon et al., 2000) echoes the mixed finding of extant research on the hypothetical causal chains between implementers’ cognition, implementation behavior, and client outcomes (e.g., Godin, et al., 2008; Lewis et al., 2018; Williams, 2015). Consistent with behavior change literature, the positive indirect effect (Table 4) supported that improved ITI led to better implementation fidelity, which in turn led to improved student AET (Larson et al., 2021; Williams, 2015). Seemingly counterintuitive, the negative direct effect of ITI on student AET can be explained in multiple ways. First, mediation effect estimates can be unstable in small samples. Thus, the small yet negative estimate implies an unmeaningful effect that is a statistical artifact of fluctuating standard errors (Hayes, 2017). Second, unobserved moderators may confound the direction of the association between ITI and AET, such as student–teacher race match and relationship (Diaz et al., 2017; Wittrup et al., 2019). Similarly, the pathway between ITI and AET may contain unobserved mediators with opposite effects on ITI and AET, which lead to the negative association observed in this study (Wu & Huang, 2007). Future research should explore moderators and mediators with large samples to test these hypotheses to elucidate the causal chain among ITI, implementation fidelity, and student behaviors (Birken et al., 2020).

Implications for Prevention Research and Practice

This study contributes to the emerging research to investigate PIES as a generic and EBP-agnostic pre-implementation strategy situated between the preparation and active implementation stages in the process of implementing universal prevention/intervention practices (e.g., Larson et al., 2021; Lyon et al., 2019) in school settings. In field application, practitioners may integrate the SC-PIES seamlessly into the beginning of any EBP training. Notably, the 2-day delay between SC-PIES and EBP training in this tightly controlled study is not necessary in practice because field practice does not need to isolate study components or control for confounding. This study also demonstrated the importance and utility of social cognitive theories (e.g., TPB; Birken et al., 2020; Godin & Kok 1996) to inform the design of PIES for application in school-based mental health settings. Consistent with prior research on TPB, our findings supported the promise of a specific PIES (SC-PIES) targeting teachers’ intention to implement (ITI) which served as a mechanism of behavioral change to enhance the yield of training and follow-up consultation of EBPs as evidenced in improved implementation and youth outcomes.

Moreover, our findings suggest that ITI may serve as a tailoring variable to differentiate the intensity and frequency of implementation supports. For teachers with weak pretraining intentions, PIES can be used to boost their intentions to implement EBPs, which has the potential to increase practitioners’ engagement with and responsiveness to training and consultation supports. For teachers with strong pretraining intentions, active implementation support alone (e.g., training) may serve as an effective strategy to produce successful implementation (Collier-Meek et al., 2017). Future research should explore a precision-based approach, especially in low-resource settings (e.g., public schools), to develop implementation strategies that are adaptive to individual characteristics as a cost-effective way to improve implementation and client outcomes using methods such as SMART designs (Eisman et al., 2020). Research in several fields (e.g., public health, schools) indicates that adaptive strategies are more likely to produce superior implementation and client outcomes than a “one-size-fits-all” approach (Sohl & Moyer, 2007).

The field of implementation science is at risk of replicating the problem it seeks to avoid (i.e., the science-to-practice gap) by creating a divide between implementation research and practice (Westerlund et al., 2019). This gap is especially prevalent in school-based implementation science and practice. The complexity, cost, and expertise required to deliver implementation strategies impact whether these strategies can be feasibly transferred into routine practice (Brownson et al., 2017). Too often researchers develop strategies that are unlikely to be adopted or used under real-world conditions to support the delivery of prevention services. This exacerbates the science-to-practice gap. The brevity and intuitiveness of the SC-PIES were intentional to ensure that it was pragmatic and fit the context of the educational sector where staff are often burnt-out with limited time available for training or delivery of new EBPs (Fiorilli et al., 2020). A focus on the pragmatics of implementation strategies increases their potential to be translated into real-world practices if effectively disseminated (Glasgow, 2013).

Limitations and Future Directions

Several limitations should be considered when interpreting the findings of this study. First, our findings may be moderated by the characteristics and complexity of EBPs. In this pilot trial, we selected one type of EBP commonly used and well-established in schools (i.e., PCBM). It is unclear how our findings would hold up with EBPs that are more complex and demanding regarding learning and time (e.g., multi-component complex therapies). Future research should extend our study by testing the effect of SC-PIES on a spectrum of EBPs with varying levels of complexity and demand for resources. Second, this preliminary study utilized a single-level sample of teachers from two schools. The teachers from the same building may have communicated with each other, which would lead to treatment contamination (Danga & Korb, 2014). However, contamination was unlikely in this study because teachers had to have access to the training material and participate in the activities of SC-PIES to obtain an effective dosage of treatment exposure (Teerenstra et al., 2006). Also, this contamination effect may be leveraged in future studies by implementing the SC-PIES as a universal preventive implementation strategy that has a school-level effect and broad reach. Future research should replicate this study with a large sample using cluster or pseudo-cluster randomized trials to estimate the school-level effect of SC-PIES and control for contamination (Pence et al., 2015). Third, self-selection bias may limit the representativeness of our sample and the generalizability of our findings. Future research should address this bias with large and diverse samples, novel participant recruitment and retention strategies, and the “intent-to-treat” analysis (Elston, 2021). Although our findings supported the efficacy of the SC-PIES, not all teachers demonstrated comparable changes in response to the SC-PIES on their intentions to implement, implementation behavior, and youth outcome. Future research needs to explore school contextual moderators (e.g., student demographics, teacher stress, or burnout), which can elucidate for whom SC-PIES works better to guide a tailored approach to implementation support. Lastly, the SC-PIES should be replicated in other youth behavioral healthcare settings to test if its effects will transfer and hold in different provider and client populations (e.g., therapists in community clinics; Anderson & Maxwell, 2017).

Conclusion

This study offers insights into how to design theory-informed and pragmatic PIES to preventively boost the yield of training and consultation. Our findings of the SC-PIES suggested that promoting teachers’ intentions to implement before training and consultation about preventive EBPs can be a viable way to enhance their subsequent intervention fidelity and student outcomes in school-based mental health settings. The findings highlight the importance of mechanism-oriented approaches in developing and testing implementation strategies, especially those grounded in social cognitive theories. We call for further investigation of cost-effective ways to elevate practitioners’ motivation as a mechanism of behavior change linked to their engagement and responsiveness to training and follow-up consultation. Last, we hope that this study stimulates researchers to develop and test motivationally focused PIES that are situated between the preparation and active implementation phases in the process of implementing prevention or intervention services in different youth-serving settings.

Abbreviations

PIES

Pre-implementation enhancement strategies

SC-PIES

Social cognitive theory-informed pre-implementation enhancement strategies

EBP

Evidence-based practice

TPB

Theory of planned behavior

ITI

Intentions to implement

AET

Academic engaged time

Availability of Data and Material

The de-identified datasets are available in the Open Science Framework repository, (osf.io/d5t4m/).