Bringing Evidence-Based Interventions into the Schools: An Examination of Organizational Factors and Implementation Outcomes

Abstract

Although schools are one of the largest providers of behavioral health services for youth, many barriers exist to the implementation of evidence-based interventions in schools. This study used the Stages of Implementation Completion (SIC) to examine school-based implementation outcomes for a computer-assisted cognitive behavioral therapy intervention for anxious youth. Organizational factors and predictors of program startup also were examined. Results indicated that the SIC detected implementation variability in schools and suggested that spending more time completing pre-implementation activities may better prepare schools for active implementation of program delivery. Furthermore, proficiency emerged as a potentially important organizational factor to examine in future school-based implementation research.

Children with identified mental health problems often do not receive mental health services (Merikangas et al., 2011; Olfson et al., 2015), and those who do typically receive them through schools (Green et al., 2013). Given the significant amount of time that children spend at school, this setting is particularly appropriate for targeting mental health problems and providing interventions to address them. School-based interventions (SBIs) are defined as programs, interventions, or strategies “designed to influence students’ emotional, behavioral, or social functioning” (Rones & Hoagwood, 2000, as cited by Paulus et al., 2016). Although preliminary evidence suggests that evidence-based SBIs, such as cognitive behavioral therapy (CBT), can be effectively implemented in school settings (Barry et al., 2013; Masia Warner et al., 2007), they are infrequently implemented and often done so with poor fidelity (Locke et al., 2015; Mandell et al., 2013; Pellecchia et al., 2015; Suhrheinrich et al., 2013).

Several barriers to implementing both prevention and intervention programs in schools have been identified (Kern et al., 2017). For example, competing responsibilities, limited space, and the constraints of the 9-month academic calendar create challenges when trying to introduce SBIs (Langley et al., 2010; Nadeem et al., 2011; Owens et al., 2014). Moreover, barriers such as financial constraints (Forman et al., 2009), confidentiality concerns (Health, 2004), and fears of youth embarrassment and stigmatization (Aguirre Velasco et al., 2020; Gronholm et al., 2018) hamper efforts for bringing evidence-based SBIs to children at risk for mental health disorders (Paulus et al., 2016). Thus, it is critical to understand the factors that increase the likelihood of successfully implementing evidence-based SBIs in order to improve the provision of mental health services to youth.

Recently, computerized and internet-based programs have gained traction for implementation in schools as an alternative to traditional manualized treatments that may be easier to implement (Champion et al., 2013; Griffiths & Christensen, 2007). These programs are cost effective and provide treatment standardization. Although there is compelling evidence for the effectiveness of computer-assisted programs for adults (Newman et al., 2011), relatively fewer studies have examined the effectiveness of such programs for youth, particularly in schools (Kendall et al., 2011; Richardson et al., 2010). One computer-assisted intervention that has been developed for anxious youth is Camp Cope-A-Lot (Khanna & Kendall, 2008), a 12-session computer-assisted CBT program based on the Coping Cat program (Kendall & Hedtke, 2006). Similar to its traditional manualized in-person delivered counterpart, Camp Cope-A-Lot has been demonstrated to be an efficacious treatment for youth anxiety (Khanna & Kendall, 2010) and has been implemented successfully in community mental health settings (Storch et al., 2015) and in an after-school program (Sanders et al., 2018). Much like other computer-assisted treatments, Camp Cope-A-Lot addresses several barriers to the implementation of evidence-based SBIs by requiring minimal therapist training, staff time, and financial burden. Thus, it is well-suited to being implemented in school settings.

When introducing new interventions to a setting, the field of implementation science (Eccles & Mittman, 2006) emphasizes the need to consider contextual factors, including features of the organization when implementing an intervention (Damschroder et al., 2009). Organizational factors such as culture, climate, and leadership have been shown to influence the implementation and sustainability of evidence-based interventions (EBIs) over time (Fixsen et al., 2005; Novins et al., 2013; Williams & Beidas, 2019). In the school context, studies have emphasized the importance of administrative support, financial resources, and a sense of consistency or “fit” between the school philosophy and the intervention (Demby et al., 2014; Domitrovich et al., 2015; Forman et al., 2009; Forman & Barakat, 2011; Owens et al., 2014; Thaker et al., 2008). In addition, teacher and administrator collaboration, competing responsibilities, and buy-in to the intervention are relevant factors to successful implementation (Baweja et al., 2016; Ingemarson et al., 2014; Langley et al., 2010; Nadeem et al., 2018). One of the only published studies to date that used quantitative methods to assess organizational culture and climate in the implementation of SBIs was conducted by Williams and colleagues (2019). This study examined organizational culture and climate profiles as a predictor of fidelity to three EBIs for autism and found that schools with comprehensive profiles (i.e. high proficiency and positive climate) had higher fidelity to two of the EBIs. The only other study examining organizational factors and provider EBI use examined data from the same study as Williams and colleagues (2019) and did not find a relationship between implementation leadership/climate and EBI use (Locke et al., 2019). The authors suggest that this may be because the organizational constructs measured, such as school-level implementation leadership, may not directly affect what is happening at the classroom level where the intervention took place. Taken together, these studies highlight the need for more examination of organizational factors in schools and their relationship to implementation outcomes.

Because implementation is a complex and multi-step process that requires consideration of multiple domains (i.e. organization, intervention, individuals involved; Damschroder et al., 2009), it is helpful to operationalize and identify steps within the implementation process. The Stages of Implementation Completion (SIC; Saldana, 2014) is an 8-Stage tool that was developed to delineate implementation processes and milestones of newly adopting organizations. The SIC was developed to examine successful and failed implementation efforts that span across three implementation phases (i.e. pre-implementation, implementation, and sustainability). These phases of implementation include activities such as planning, stakeholder engagement, training, and adapting interventions. The SIC has been adapted for a number of different interventions and contexts, including schools (Nadeem et al., 2018). Given the specific contextual considerations within schools, the application of the SIC within this unique context merits continued examination.

This observational study is part of a larger trial examining the implementation and sustainability of the computer-assisted Camp Cope-A-Lot intervention to treat anxious youth in schools (see Crane et al., in press for a description of qualitative outcomes from the larger trial). The present study represents a unique opportunity to learn about the implementation process in schools using the SIC (Saldana, 2014), whose initial school-based adaptation was developed for this study. Our primary aim was to describe and operationalize implementation activities based on the SIC. To achieve this, we (a) present quantitative SIC data and (b) describe case examples of the implementation process for representative schools that demonstrated successful and unsuccessful program startup. Our secondary aim was to assess the relationship between SIC activities, organizational factors, and program startup. Differences in organization-level factors were explored across two cohorts of implementing schools—one in a district in the United States (US) and one in Canada. To achieve our second aim, we examined (a) differences in organizational factors and SIC scores across geographic locations (i.e. US and Canadian sites); (b) whether organizational factors predicted SIC duration and proportion scores; and (c) the extent to which the SIC and organizational factors predicted implementation success and provider competence.

Methods

Participants

Schools

Schools were located in two general geographic areas: the suburbs of a large city in the Northeastern US (n = 7), and the western suburbs of a Canadian city (n = 13)¹. There were nine schools that were approached to participate in the study who declined prior to providing consent. SIC data for these schools indicates that all except one discontinued in Stage 1 (i.e. during initial engagement) and the remaining school discontinued in Stage 2 (i.e. during feasibility consideration). Schools in the US were recruited individually through outreach presentations to teachers and special services staff (e.g. school counselors, child study team members). Schools in Canada were recruited via a single school district administrator who was enthusiastic about the study and collaborated with personnel across the district to champion the project. Initial communication about the study, as well as school agreement to enroll, was managed directly by this administrator rather than by study staff. Based on information received by the administrator, individual school principals were given the opportunity to opt in or out of possible participation. Schools that agreed to participate completed informed consent procedures with study staff. Schools were initially contacted and recruited between 2011 and 2013, and study participation for all schools was completed by 2016.

Although attempts were made to provide a standardized implementation strategy across all sites, variation occurred between schools in the US and Canada in training; training dates varied for the US schools, but providers from Canadian schools were all trained on one of two dates. Otherwise, procedures were consistent across all schools, with variations described in the resulting outcomes.

Providers

Within each participating school, interested school staff members (“providers”) received training in Camp Cope-A-Lot to treat anxious youth (N = 38). Providers did not need prior mental health training and participation was voluntary. For participating in the larger study, providers at the schools in the US were compensated with a small honorarium ($300 after year 1; $100 after year 2; $100 after year 3); providers in Canada were not permitted to accept compensation. All US schools had more than one provider per school (M = 3.57, SD = 1.40), whereas all Canadian schools had only one provider per school. Provider demographic characteristics are shown in Table 1.

Table 1.

Provider demographic characteristics.

	US schools n=25		Canadian schools n=13		Overall N=38
	Mean/N	SD/%	Mean/N	SD/%	Mean/N	SD/%
Age (years)	41.24	11.82	45.69	10.87	42.76	11.55
Race/Ethnicity
Asian	0	0	0	0	0	0
Black/African American	1	4	2	15.4	3	7.9
Hispanic/Latinx	1	4	0	0	1	2.6
Not reported	1	4	2	15.4	3	7.9
Other	0	0	1	7.7	1	2.6
White	22	88	8	61.5	30	78.9
Gender
Female	21	84	12	92.3	33	86.8
Male	4	16	1	7.7	5	39.5
Years therapy experience	5.57	6.79	9.33	8.50	6.86	7.53
Position in school
Counselor	8	32.0	0	0	8	17.0
Other	1	4.0	1	7.2	2	4.3
Psychologist	7	28.0	6	46.2	13	27.7
Social worker	2	8.0	6	46.2	8	17.0
Teacher	7	28.0	0	0	7	14.9
Previous experience providing psychotherapy	16	64	13	100	29	76.3
Previous experience treating anxiety	13	52	10	76.9	23	60.5
CBT identificationa	4.92	1.68	5.31	1.44	5.05	1.59
Opinions about EBIsa	5.84	1.49	6.54	0.66	6.08	1.30
Confidence in EBI efficacya	5.32	1.49	5.62	1.39	5.42	1.45

^aBased on baseline responses on the Clinician Demographics and Attitudes Questionnaire 7-point scale, where 7 indicates strong agreement and 1 indicates strong disagreement.

Abbreviations: CBT: cognitive behavioral therapy; EBI: evidence-based interventions.

Youth

Youth with anxiety were identified via school-wide screening of students in grades 1 through 4 using the Behavior Assessment Scale for Children – Second Edition (BASC-2). Teachers were given protected time to complete the BASC-2 for their students within approximately the first three months of school. Students with elevated BASC Anxiety scores were eligible to receive Camp Cope-A-Lot. In addition, students could be referred to Camp Cope-A-Lot via “teacher nomination” if the classroom teacher or guidance counselor believed the child would benefit from treatment for anxiety. Parents of youth who were deemed eligible to participate in Camp Cope-A-Lot based on either of these referral methods were contacted and provided with additional information about the study. If parents agreed for their child to participate in the larger study, they underwent informed consent and the child provided assent. After receiving parent consent and child assent, students met with their provider weekly.

Measures

Stages of implementation completion (SIC)

The 8-Stage SIC examines implementation process and milestones across three phases of implementation, including pre-implementation, implementation, and sustainability. Stages include Engagement (Stage 1), Feasibility Assessment (Stage 2), Readiness Planning (Stage 3), Staff Hiring and Training (Stage 4), Fidelity Monitoring System in Place (Stage 5), Services and Consultation Begin (Stage 6), Ongoing Service Delivery and Monitoring (Stage 7), and Development of Competency (Stage 8). Within each stage are implementation activities tailored to describe the implementation strategy for Camp Cope-A-Lot (e.g. meeting with schools, contacting principals). Pre-implementation (Phase 1) activities include Stages 1 through 3 (i.e. agreeing to implement the intervention, considering feasibility for implementing the intervention, and readiness planning for implementation). Implementation (Phase 2) activities include Stages 4 through 7 (i.e. hiring and training staff, monitoring fidelity, providing services and consultation, and tracking staff ongoing service delivery and fidelity). A total of 44 items spanning the phases of implementation populated the 8 Stages. The SIC is a date-driven measure whereby the date on which implementation activities are completed is recorded (e.g. Date of first school response to first planning contact; Date of teacher/therapist recruitment review; Date of first child treatment session). Examples of items from each of the 8 Stages are shown in Table 2. The SIC has demonstrated validity and reliability when used with other EBIs and analyzed using item response theory-based Rasch modeling (Linacre, 2009; Saldana, 2014).

Table 2.

Adapted Camp Cope-A-Lot SIC: example items for each stage.

Stage	Example question
Pre-implementation phase
Stage 1	Date of principal meeting to describe program
Stage 2	Date feasibility process initiated
Stage 3	Date of teacher/therapist recruitment review
Implementation phase
Stage 4	Date therapist identified
Stage 5	Date of first session report form per therapist
Stage 6	Date of first session per therapist
Stage 7	Date completed final consultation call
Sustainability phase
Stage 8	Date of first knowledge test

SIC scoring

The SIC yields three primary scores that can be calculated within stage, phase, or full implementation: Duration (the amount of time to complete implementation activities), Proportion (the proportion of implementation activities completed), and Final Stage (the furthest point in the implementation process achieved by a site). A total Duration score was calculated as the total number of days from the first to last activity across all phases. For each phase, duration scores were calculated as the difference between the first and last date of activities within that phase. Proportion scores also were calculated across each of the three phases using a ratio of the number of activities completed to the number of possible activities per phase. As defined by the developer (LS) and implementation team of the Camp Cope-A-Lot model, successful implementation was assessed by whether at least one provider at each site initiated treatment with a child using the evidence-based protocol.

Clinician Demographics and Attitudes Questionnaire (CDAQ)

The CDAQ is a 15-item self-report questionnaire that assesses providers’ demographic characteristics, CBT experiences, and opinions about EBIs for youth anxiety (Beidas et al., 2009; 2012). Used in related research (Beidas et al., 2009; 2012), the CDAQ demonstrated acceptable psychometrics with appropriate intraclass coefficient (ICC = 0.91) and Spearman Brown split-half reliability (0.85). The CDAQ was administered prior to training.

Measures of competence

Knowledge test

This 20-item (5 true/false, 15 multiple choice) test assesses knowledge of CBT for youth anxiety (Beidas et al., 2009; Walkup et al., 2008). Correct answers are summed to provide a total score. Three versions of comparable difficulty exist (Beidas et al., 2009). Different versions were administered pre and post-training and at the end of the school year following implementation of Camp Cope-A-Lot.

Performance-based role plays

Before and after Camp Cope-A-Lot training, as well as at the end of the first year of the study, providers completed role plays by phone. Study staff read the provider a vignette about an anxious child with whom they were to conduct an exposure task. The role play was then conducted with the study staff member, who had a detailed script for completing an eight-minute role play. Role-plays were recorded, transcribed, and coded by master’s level study staff for fidelity, as measured by adherence (e.g., obtaining ratings of children’s subjective anxiety) and skill. Approximately 20% of tapes were coded for reliability by multiple raters. Reliability was acceptable for both the Adherence Total Score (Krippendorff’s α = 0.80) and the Skill Total Score (Krippendorff’s α = 0.73).

Organizational social context (OSC)

The OSC quantitatively evaluates the social culture and climate of mental health and social services organizations. In this study, school staff were verbally instructed to consider their school as the organization they were rating. Responses were rated on a scale from 1 (Never) to 5 (Always). The OSC yields six subscales, which are grouped by the domains of culture (i.e. norms and values that drive behavior in the organization) and climate (i.e. the psychological impact of the work environment on the individual). The culture domain includes proficiency, rigidity, and resistance subscales, and the climate domain includes engagement, functionality, and stress subscales. Each subscale is reported using T-scores, which have a mean of 50 and a standard deviation of 10, and are based on a normative sample of 100 mental health organizations (Glisson et al., 2008). Reliability coefficients (alpha) for all subscales are greater than 0.75.

Procedures

All study procedures were approved by Temple University’s Institutional Review Board.

Treatment program

Camp Cope-A-Lot (Khanna & Kendall, 2008) is a 12-session computer-assisted CBT intervention for anxious youth that is based on the evidence-based Coping Cat program (Kendall & Hedtke, 2006). Prior to implementing Camp Cope-A-Lot with students, providers underwent training in the delivery of the program, which included reading the Camp Cope-A-Lot manual and attending a one-day workshop. The workshop included didactic training including role-plays, hands-on support on using the computer program, and other practical skills needed to deliver the intervention with fidelity.

Provider assessment

Prior to the workshop, providers completed measures asking about demographic information and CBT knowledge. Immediately following the workshop and at the end of the school year, they completed another test of CBT knowledge and completed performance-based role-plays that assessed adherence and skill with Camp Cope-A-Lot. After training, providers were assigned youth identified with anxiety who they treated at school.

Consultation

As part of standard Camp Cope-A-Lot implementation, providers were expected to attend eight 30-minute consultation calls across one semester. Masters- or doctoral-level consultants with specialized training and experience in child anxiety treatment and Camp Cope-A-Lot led the consultation calls. The calls incorporated role plays, didactic training (e.g. on how to use exposures effectively in a school settings), and case discussion. Providers could join by phone or online via a conferencing program.

Organizational data collection

Providers (M=4.3 staff per school) completed baseline reports of organizational health (i.e. the OSC) in the fall semester of the school year. The completion of the paper-and-pencil measure took approximately 20 minutes, and school staff members were ensured that responses remained confidential. A study staff member either remained present while the school staff member completed the measure or returned within one day to retrieve the completed measure.

Implementation behavior

Implementation behavior was measured by the SIC. Study staff were trained to document implementation activities for data collection (i.e. recording who was involved in activities and the date on which activities were completed for each school). Because the SIC was introduced after several schools had begun implementing, a small amount of initial data was collected retrospectively²; however, the majority of data were collected prospectively with data being recorded as it occurred. Regular calls took place between PCK and LS with updates on SIC data including new sites, dates of completed activities, people and time involved in each activity, and discontinued sites.

Data analytic plan

Inferential and descriptive analyses were conducted in SPSS. There were no missing data on the variables of interest for this study (i.e. SIC duration and proportion; OSC subscales).

First, descriptive analyses for the SIC were calculated, including mean duration and proportion scores. To supplement these descriptive analyses, authors HEF and LS reviewed SIC data for all schools and selected two representative schools to describe in more detail as case examples, including one school that demonstrated successful implementation and one school that did not achieve successful implementation. Next, Pearson correlations among study variables are presented in Table 3. Correlations were examined to identify which OSC subscales would be retained for additional analyses. Proficiency emerged as the only subscale significantly correlated with any SIC duration or proportion variables and was the only organizational variable retained for additional analyses. US and Canadian schools were compared using t-tests to examine measures of implementation behaviors (i.e. proportion and duration scores), organizational factors, and number of children treated. A chi square was used to compare the likelihood of whether a child was treated or not in US versus Canadian schools. Then, three separate linear regressions were conducted – one to predict pre-implementation duration (Phase 1), one to predict implementation duration (Phase 2), and one to predict SIC proportion scores. Finally, SIC variables and the OSC proficiency subscale were examined as possible predictors of implementation success and competence. A logistic regression was used to examine predictors of program startup (at least one child being treated). Hierarchical regressions were conducted to examine predictors of number of children treated and to examine predictors of competence (measured by role plays and knowledge scores).

Table 3.

Correlations between SIC duration/proportion scores and OSC subscales (N=20 schools).

	Correlation coefficients
	Duration of scored activities (Phase 1)	Duration of scored activities (Phase 2)	Proportion of scored activities (Phase 1)a	Proportion of scored activities (Phase 2)	OSC proficiency	OSC rigidity	OSC resistance	OSC engagement	OSC functionality	OSC stress
Duration of scored activities (Phase 1)	—
Duration of scored activities (Phase 2)	−0.55*	—
Proportion of Scored Activities (Phase 1)a	–	–	—
Proportion of scored activities (Phase 2)	0.23	0.18	—	—
OSC proficiency	0.65**	0.37	—	0.03	—
OSC rigidity	−0.03	0.02	—	0.07	0.43+	—
OSC resistance	−0.03	0.04	—	−0.15	0.13	0.61**	—
OSC engagement	0.39+	−0.24	—	0.17	0.25	−0.05	−0.04	—
OSC functionality	0.41+	−0.30	—	−0.01	0.34	−0.22	−0.28	0.68**	—
OSC stress	−0.16	−0.003	—	−0.13	0.24	0.36	0.20	−0.57**	−0.43+	—

⁺p<.10;

^*p<.05;

^**p<.01.

^aCells are blank for proportion in phase 1 (pre-implementation) because there was no variability due to all participating schools completing all activities in phase 1 (pre-implementation).

Results

Overall SIC outcomes

Duration and proportion scores for pre-implementation and implementation activities were examined. All schools (N = 20) completed all pre-implementation activities, suggesting a thorough assessment of feasibility (Stage 2) and readiness (Stage 3) by all participating schools, as guided by the study implementation supports provided. As a result, there was no variability in pre-implementation proportion scores. Although there was more variability between schools in the number of Phase 2 implementation activities completed, proportion scores remained high (M = 0.81, SD = 0.18). Across all schools, the duration for implementation activity completion was significantly longer than pre-implementation activities (Table 4). A total of 16 out of the 20 schools (80%) achieved program startup (i.e. treated at least one child).

Table 4.

Comparisons between US and Canadian sites on main variables of interest (N=20; US schools: n=7; Canadian schools: n=13).

	US		Canada			Overall
	Mean	SD	Mean	SD	T	Mean	SD
Implementation outcomes
Pre-implementation duration	219.57	52.08	147.46	0.52	3.66**	172.70	45.85
Implementation duration	198.29	40.74	226.62	22.27	−2.03+	216.70	32.09
Proportion of pre-implementation activities completeda	1.00	0	1.00	0	–	1.00	0
Proportion of implementation activities completed	0.88	0.12	0.78	0.19	1.26	0.81	0.18
OSC
OSC year 1 proficiency	67.69	7.57	58.93	6.43	2.74*	61.99	7.91
Participants
Number of service providers	3.57	1.40	1.00	0	4.87**	1.90	1.48
Number of children treated	7.00	1.00	1.15	0.55	14.33**	3.20	2.95

⁺p<.10;

^*p<.05;

^**p<.01.

Note: OSC: Organizational Social Context.

^aNo variability in proportion scores for pre-implementation because all schools completed all pre-implementation activities.

SIC case examples

To provide further context on differences between sites that successfully implemented Camp Cope-A-Lot and those that did not, we provide detailed descriptions of the implementation process for two sites that demonstrated common patterns of implementation. First, School A, located in the Northeastern United States, completed the implementation process successfully. School staff completed all implementation activities at a steady pace. The first child was treated within one month of training, and providers engaged in consultation within a week of the initial intake. Importantly, providers at this school consistently engaged in consultation through the final consultation session. Although providers at this school successfully passed all knowledge tests, the first role play demonstrating competency was not passed. However, given that this school moved at a relatively fast pace to implement all activities, the assessment of competency may have been premature; indeed, the second role play was passed with competency 6 months later.

In contrast, School B, located in Canada, demonstrated strong adherence to the pre-implementation phase (Phase 1) activities, but did not complete the implementation phase (Phase 2) activities with fidelity. The pace of pre-implementation was significantly faster than the average successful pre-implementation. Notably, Stage 3 readiness activities were completed within 2 days (compared to the mean of 12.67 days for schools with successful implementation), which was likely too fast for the school to adequately prepare for or develop the infrastructure necessary to support a successful implementation. Subsequently, although providers from this school participated in training, attended some clinical consultation, passed all knowledge tests, and even consented a child to participate, the school never launched their treatment services. This pattern of poor-quality pre-implementation as a predictor of successful program launch is consistent with previous SIC findings and results from the present study.

Comparisons between US and Canadian sites

Out of the 20 schools that had SIC data collected, seven were located in the US and 13 were located in Canada. The four schools that did not achieve program startup (i.e., treat at least one child) were all Canadian sites. As shown in Table 4, t-tests were used to compare US and Canadian sites on (a) SIC outcome variables (pre-implementation duration, implementation duration, proportion of pre-implementation activities completed, and proportion of implementation activities completed); (b) organizational measures of proficiency; and (c) participants (number of service providers and number of children treated).

In terms of SIC outcome variables, pre-implementation duration was significantly longer for US sites than Canadian sites (Table 4). Notably, the duration of pre-implementation activities for US sites (M = 219.57 days) was generally consistent with average durations found for successful implementation of other evidence-based mental health interventions, as measured by the SIC (Saldana et al., 2015)³. Canadian sites completed pre-implementation activities at a much faster pace (i.e. M = 147.46 days). Prediction models of success were calculated based on previous analyses of SIC data across a range of practices (Saldana et al., 2015). Although significant differences emerged in the pre-implementation phase (Phase 1), implementation duration and proportion scores did not differ significantly between US and Canadian sites.

The proficiency subscale of the OSC was the only OSC subscale significantly correlated with any SIC duration or proportion variables (r = 0.65, as shown in Table 3); this is a key organizational measure of the degree to which providers are competent and place the well-being of clients first. OSC proficiency scores were significantly higher at US sites, indicating increased clinical competency and prioritization of client well-being in US compared to Canadian sites (Table 4).

For tests of participant differences, there were no significant differences between US and Canadian sites regarding whether at least one child was treated with Camp Cope-A-Lot, χ² (2, N = 20)=2.69, p = .10, but significantly more children were treated at US schools, t(8.04)=14.33, p<.001.

Predictors of SIC outcomes

Pre-implementation (phase 1) duration

A hierarchical regression was conducted to examine predictors of pre-implementation SIC duration. Site location (i.e. Canadian versus US) was included as a covariate. As shown in Table 5, OSC proficiency was included in step 1 and accounted for significant variance in pre-implementation duration, such that higher proficiency was associated with longer pre-implementation duration. The addition of site location in step 2 accounted for significant additional variance. With site location added to the model, proficiency only trended toward significance. However, site location was a significant predictor of pre-implementation duration, such that Canadian sites had significantly shorter pre-implementation durations when proficiency was held constant.

Table 5.

Hierarchical regression predicting pre-implementation (phase 1) duration (N = 20).

	Step 1		Step 2
	Coefficient (SE)	p	Coefficient (SE)	p
OSC proficiency	3.76 (1.04)	.002	1.90 (0.96)
	.07
International (0=US; 1=Canada)			−55.51 (15.58)	.002
R 2	.42		.67
F (df1, df2)	13.03 (1, 18)**		17.09 (2, 17)**
ΔR2			.25
Fchange (df1, df2)			12.69 (1, 17)**

^*p<.05;

^**p<.01.

Implementation (phase 2) duration

Because site location and OSC proficiency together contributed significant variance to pre-implementation duration, they were examined individually as possible predictors of implementation duration. However, neither variable individually contributed significant variance. Pre-implementation duration was the only variable included in the final model predicting implementation duration; it was a significant predictor, such that a longer duration in pre-implementation predicted a shorter implementation duration, b= −0.38, t= −2.78, p=.01, R²=0.30.

Proportion

Given that there was no variability in pre-implementation proportion scores (i.e. all sites completed all activities), predictors of these scores were not examined. OSC proficiency, pre-implementation duration, and site location were examined as potential predictors of implementation proportion scores, but none of them were significant predictors.

Predictors of implementation success and competence

Successful program startup

The SIC defines successful program startup for a given school as treating at least one child with Camp Cope-A-Lot. A logistic regression was used to examine whether organizational factors (i.e. proficiency), site location, or duration of pre-implementation SIC activities predicted successful program startup; none of these predictors was significant. However, the same predictors were examined as predictors of number of children treated, and site location was a significant predictor, such that US sites treated significantly more children, b= −6.03, t= −10.49, p < 0.001, R²=0.93. The four sites that did not treat any children were all Canadian.

Predictors of competence

Competence was defined as passing the final knowledge test (M = 6.82 months after training, SD = 1.44) and the final role play (M = 5.94 months after training, SD = 0.35). Five sites out of 20 (four Canadian; one US) did not achieve competence; four of those sites treated at least one child with Camp Cope-A-Lot. Pre-implementation duration, implementation duration, site location, and proficiency were included as predictors of whether at least one provider at each site achieved competence. There were no significant predictors of achieving competence, but proficiency trended toward significance, b = 0.03, t = 2.08, p=.054, R²=0.27.

Discussion

The implementation of EBIs in schools is an important avenue for increasing access to psychosocial services for youth. The results of the present study suggest that the SIC was able to assess and detect variability in the implementation process in schools in both the US and Canada. In addition, this study underscores the importance of thorough completion of pre-implementation activities; all schools completed all pre-implementation activities and the majority of schools (80%) successfully treated at least one child with Camp Cope-A-Lot. Furthermore, the study provides some evidence to suggest that spending more time preparing for implementation may allow schools to be better prepared for the implementation phase. Results also support the notion that having proficient organizational cultures may affect the rate at which schools complete implementation activities, as measured by the SIC. This is consistent with a model posited by Williams and Beidas (2019) in which proficient organizational cultures are a catalyst to changing clinician behavior and, in turn, increasing EBI implementation.

Comparisons between US and Canadian sites

Comparisons between schools in the US and Canada indicated that results varied by location in terms of proficiency scores and pre-implementation duration, which were both higher at US schools. This difference might be driven in part by the individualized approach to conducting training in the US, where schools were able to train at their own pace, versus those in Canada where training dates were pre-determined and overseen by a district administrator. This is particularly notable given that more of the Canadian sites did not achieve program startup. As mentioned above, this could be due to the fact that Canadian sites spent less time completing pre-implementation activities (i.e. preparing for program startup). Relatedly, the uniform timeline for pre-implementation activities across Canadian sites may not have allowed for sufficient tailoring/planning for each individual school context or for time for all providers to achieve competence. Importantly, these differences may not be due to geography, but instead due to quality of pre-implementation behavior and differences in organizational structure.

Organizational factors: role of proficiency

Examination of the relationship between the SIC and organizational constructs indicated that proficiency was the most relevant organizational predictor of the duration of implementation outcomes for a computer-assisted SBI. Proficient organizational cultures place the well-being of the clients first and have competent and knowledgeable providers (Glisson et al., 2008). Consistent with this definition, proficiency trended toward significantly predicting competence, such that more proficient schools were more likely to achieve competence as measured by knowledge tests and role plays. Schools with higher proficiency scores also spent longer completing pre-implementation activities. Although this finding also only trended toward significance, given the small sample size and preliminary nature of the study, it is worth considering the potential role of proficiency in future research given its consistence emergence as a predictor. Previous studies have also identified the importance of proficient organizational cultures when implementing services for youth (Williams & Glisson, 2013). Proficiency appears to be highly relevant to several previously-identified organizational factors that affect implementation (Baweja et al., 2016; Beidas et al., 2012; Langley et al., 2010; Ringle et al., 2015). For example, schools with higher proficiency scores might focus more on ensuring that they have the resources necessary to administer the intervention. Such resources might include supplies, administrative support, and time reserved for training. If schools that are more proficient are in fact more invested in evaluating whether such resources are available, this may explain the finding that proficient schools spend more time in pre-implementation. Indeed, this approach appears advantageous, as implementation proceeds more quickly when more time is spent on pre-implementation activities.

Although proficiency was related to implementation duration, there were no other organizational factors associated with the proportion of SIC activities completed. Although it is possible that proficiency is the most relevant OSC construct in schools, it could also suggest that individual subscales of the OSC may not be sensitive to the school environment, and as indicated by Williams et al. (2019), may be better conceptualized as complete profiles. Recent work examining measures of implementation-specific organizational factors in the school setting has identified a need for adapting commonly-used measures for the school setting (Lyon et al., 2018) and an examination of organizational factors in conjunction with individual factors (Locke et al., 2019) Taken together, these findings suggest that a modified version and/or approach to interpreting the OSC may make it more suitable to the school setting.

Predictors of successful startup and competence

Successful program startup is a key SIC milestone and is defined as treating at least one child with Camp Cope-A-Lot. None of the variables examined (i.e. proficiency, site location, and SIC duration) differentially predicted whether program startup occurred. An impressive 16 out of 20 schools successfully achieved program-start up. This high proportion of schools that achieved startup limited the ability to statistically determine what factors differentiated the four schools that did not successfully use Camp Cope-A-Lot. Importantly, the fact that all sites completed all pre-implementation activities likely contributed to the high proportion of sites achieving program startup, and the lack of prediction made from the SIC duration score.

Another measure of implementation success was the number of children treated with the intervention at a given school. Neither organizational factors nor SIC duration predicted the number of children treated at each school. However, US sites treated a significantly higher number of children than did Canadian sites. This finding can likely be accounted for by the fact that US sites had more than one provider per school, whereas Canadian sites all had only one provider per school. This suggests that the size of the workforce may affect the degree to which use of the program is widespread throughout the school. At US sites, several providers who implemented Camp Cope-A-Lot with youth did not have a mental health background. This highlights the potential role of non-mental health school staff in increasing the availability of mental health prevention and intervention programs in schools.

Limitations and future directions

Findings from this study should be considered in the context of some limitations. The main limitation in the interpretation of inferential analyses is the small sample size. For example, there were several correlations that were approaching significance, but did not fall below the p=.05 threshold (Table 3). With a larger sample size, some of these correlations may have been significant and warranted further examination. In addition, organizational culture and climate subscales were analyzed independently in the current study, but a larger sample size would have allowed for a more thorough and holistic examination of organizational culture and climate, as conducted by Williams et al. (2019). Other more sophisticated statistical analyses (e.g. multilevel modeling to account for nesting of students within schools) and comparison of sites on a wider variety of organizational contextual variables should be conducted in future studies with a larger sample. Another limitation is that the absence of variability in pre-implementation activities prevented examination of relationships between specific pre-implementation activities and implementation success. Several schools (n = 9) that declined participation in the study did engage in some pre-implementation activities (SIC Stages 1 and 2); however, their lack of consent prevented the collection of data necessary to be included in the current analyses. Finally, although the SIC definition of program startup (i.e. delivering the intervention at least once) has been used by several studies (e.g. Nadeem et al., 2018), this is a necessary but insufficient condition for schools to achieve sustained use of CCAL. Future work should examine additional predictors of sustained EBI use after achieving initial program startup. Despite its limitations, this study provides insights about the implementation process for an SBI using a standardized measure (the SIC) and contributes to the limited literature on the role of organizational factors in the implementation of SBIs.

Implications

Results from this study have several implications for the implementation of SBIs. First, particular attention should be given to pre-implementation activities (i.e. initial engagement, readiness planning) to ensure that program startup goes smoothly. Carefully assessing readiness and feasibility of implementing the intervention might reduce the amount of time needed during the actual implementation of an intervention. Although measurement refinement is needed, there is evidence from this study and from other ongoing work (Lyon et al., 2018) that organizational factors are related to the implementation of EBIs in schools. In the planning phase, the role of proficiency within the organization should be considered. Objective assessments of competence in the intervention (e.g. role plays) are one indicator of proficient organizational cultures, which require sufficient staff knowledge of the intervention being delivered. Proficient organizational cultures also prioritize the well-being of the clients for whom the intervention is targeted. Thus, this should be assessed and addressed prior to commencing implementation efforts. Another consideration for future implementation efforts is the role of organizational structure in maximizing implementation outcomes. Differences between US and Canadian sites can at least partially be attributed to differences in organizational structure. For example, US sites had a larger number of staff members implementing the program and use of the program was more widespread throughout these schools. Together, findings from this study contribute to the growing literature emphasizing the importance of pre-implementation and organizational factors in the process of implementing new interventions in schools.