Are inner context factors related to implementation outcomes in cognitive-behavioral therapy for youth anxiety?

Abstract

Among the challenges facing the mental health field are the dissemination and implementation of evidence-based practices. The present study investigated the relationships between inner context variables (i.e., adopter characteristics and individual perceptions of intra-organizational factors) and two implementation outcomes – independently rated therapist fidelity on a performance-based role-play (i.e., adherence and skill) and self-reported penetration of cognitive behavioral therapy for youth anxiety following training. A significant relationship was found between inner context variables and fidelity. Specifically, adopter characteristics were associated with adherence and skill; individual perceptions of intra-organizational factors were associated with adherence. Inner context variables were not associated with penetration. Future directions are discussed.

The development, dissemination, and implementation of Evidence-Based Practice (EBP), defined as “the integration of the best available research with clinical expertise” (American Psychological Association, 2006), remains a focus in discourse concerning quality improvement in mental health care (Weisz, Sandler, Durlak, & Anton, 2005). Unfortunately, the promise of dissemination and implementation of EBPs has yet to be fully realized (President's New Freedom Commission on Mental Health, 2003). Various implementation strategies have been developed to improve the dissemination and implementation of EBPs in the community. An implementation strategy is a “systematic intervention process to adopt and integrate evidence-based health innovations into usual care” (Powell et al., 2011). Examples of implementation strategies include discrete strategies, such as audit and feedback, as well as more complex strategies, such as the use of community development teams to implement EBP (Saldana & Chamberlain, 2012) and the Availability, Responsiveness, and Continuity (ARC) organizational implementation strategy (Glisson et al., 2010). One frequently used discrete implementation strategy is training community therapists (Addis & Krasnow, 2000; Herschell, McNeil, & McNeil, 2004; Rakovshik & McManus, 2010; Williams, Martinez, Dafters, Ronald, & Garland, 2011).

Despite the frequent use of training as a discrete implementation strategy (Barwick et al., 2012), results of studies that have evaluated training as the sole implementation strategy have been disappointing (see review by Beidas & Kendall, 2010). Typical training efforts include printed education materials and one-time didactic workshops and are better than no training in improving therapist knowledge, but have no significant effect on client outcomes (Beidas & Kendall, 2010; Farmer et al., 2008; Fixsen et al., 2005; Shafran et al., 2009) or therapist behavior (Hawkins & Sinha, 1998; Sholomskas et al., 2005). Ongoing support, or consultation (Edmunds, Beidas, & Kendall, 2013; Nadeem, Gleacher, & Beidas, 2013), appears to be a critical discrete implementation strategy that needs to be included in the training process (Beidas, Edmunds, Marcus, & Kendall, 2012). An additional explanation for ineffectiveness of training alone is that it likely does not attend to contextual factors that may impact therapist behavior. Transferring innovation, such as EBP, to routine clinical use is a complex, multilevel process dependent upon more than therapists’ understanding of a treatment itself. Clinicians, their practice settings, and other factors can impact successful training (Beidas & Kendall, 2010) and implementation (Damschroder et al., 2009; Fixsen et al., 2005; Taxman & Belenko, 2012). Effective implementation may only occur when therapists are trained effectively and when the context supports behavior change (Beidas & Kendall, 2010; Sanders & Turner, 2005; Turner & Sanders, 2006). Understanding factors that impact therapist behavior and implementation outcomes will move the field forward in exploring other potential avenues for implementation interventions.

A variety of implementation outcomes are important to consider when transferring innovation to routine clinical use. Implementation outcomes are “the effects of deliberate and purposive actions to implement new treatments, practices and services” (Proctor et al., 2010, p. 65) and are distinct from service and/or client outcomes. They are most proximal to implementation strategies and are intermediate outcomes between implementation strategies and client outcomes (Proctor et al., 2010). Two implementation outcomes are of particular interest when evaluating training and consultation as discrete implementation strategies: fidelity and penetration. Fidelity refers to “the degree to which an intervention was implemented as it was prescribed in the original protocol” (Proctor et al., 2010, p. 69) whereas penetration refers to, “the integration of a practice within a service setting and its subsystems” (Proctor et al., 2010, p. 70). These outcomes are worthy of investigation given findings that fidelity decreases when EBPs are implemented in community settings (Henggeler, 2004; Henggeler, Melton, Brondino, Scherer, & Hanley, 1997) and that penetration is understudied as an implementation outcome (Proctor et al., 2010).

Fidelity is an important implementation outcome because it is the most proximal outcome to training and consultation; whereas penetration is important because it must occur for fidelity to matter (i.e., if therapists do not integrate an EBP within their practice, their fidelity is a moot point). Fidelity includes: 1) adherence (i.e., whether therapists implement prescribed treatment components), 2) competence (i.e., skill in implementing treatment), and 3) treatment differentiation (i.e., whether treatment differs from others along important dimensions; Perepletchikova, Treat, & Kazdin, 2007; Waltz, Addis, Koerner, & Jacobson, 1993). Assessment of fidelity typically includes qualitative measures of competence and quantitative measures of adherence, the latter of which sufficiently address both adherence and differentiation (Perepletchikova et al., 2007; Waltz et al., 1993). Although findings have been mixed regarding the relationship between fidelity and outcomes (Perepletchikova & Kazdin, 2005), within a randomized controlled trial, fidelity assessment helps determine treatment integrity and provides information on active treatment ingredients (i.e., which specific components are critical for improved outcomes). Within community settings, fidelity assessment can help determine what quantity and quality of active ingredients are being delivered (Schoenwald, Chapman, & Garland, in press). Both fidelity and penetration are important implementation outcomes to examine because they contribute to the research-practice gap. Poor fidelity contributes to the research-practice gap by limiting exposure to active ingredients. Similarly, lack of penetration (i.e., when an EBP does not become integrated into a service delivery setting) prevents the delivery of the gold-standard treatment to those individuals who could benefit from them. Given that research shows limited penetration following initial implementation (Stirman, Cook, Calloway, Castro, & Charns, 2012), a greater understanding of factors that impact fidelity and penetration over time will allow the field to explore potential avenues for maximizing and sustaining implementation outcomes.

The Exploration, Preparation, Implementation, and Sustainment framework (EPIS; Aarons, Hurlburt, & Horowitz, 2011) is a multi-level and multi-phase conceptual model that provides a lens through which to understand both the implementation process as well as important contextual factors that impact the implementation process. The process of implementing an innovation occurs in four distinct phases: exploration, preparation, implementation, and sustainment. Furthermore, a number of contextual factors that impact each phase uniquely are grouped into the following categories: intervention characteristics, outer context, and inner context. The outer context refers to system-level factors such as the service environment, inter-organizational environment, and consumer support/advocacy. The inner context refers to adopter characteristics and intra-organizational characteristics (Aarons et al., 2011). In this manuscript, we focus on inner context factors (i.e., adopter characteristics and individual perceptions of intra-organizational characteristics) during the implementation phase.

Adopter characteristics include demographics (e.g., clinical experience) and attitudes (e.g., openness) towards EBPs. The findings from preliminary research examining the impact of demographics and attitudes on implementation outcomes are mixed. For example, therapists with more clinical experience and a graduate degree prior to training were more likely to demonstrate superior posttraining fidelity in Motivational Interviewing, as assessed via observational coding of audiotapes by trained coders (Carpenter et al., 2012). This finding suggests that tailoring training based on level of previous training and experience may be a worthy pursuit in order to maximize implementation outcomes. However, another study demonstrated varied relationships over time between therapist demographics and observational ratings of competence in implementing an EBP for adolescent substance abuse (Garner, Hunter, Godley, & Godley, 2012). Those who reported having other licenses or certifications and those who had spent longer amounts of time at their current positions were less likely to be rated as competent nine-months after training.

Similarly mixed findings are found regarding the relationship between therapist attitudes and implementation outcomes. Higher initial appeal and openness toward EBPs by clinicians participating in training on EBPs for youth predicted decreases in knowledge score commission errors following training (Lim, Nakamura, Higa-McMillan, Shimabukuro, & Slavin, 2012). However, within the same study, greater initial appeal of EBPs was associated with decreases in selected knowledge scores pertaining to anxiety and disruptive behavior techniques following training. These findings reveal the complex relationship between adopter characteristics and training outcomes and suggest the need for further investigations that take into account the impact of inner context factors.

Intra-organizational factors (e.g., culture, support, climate) are specific to the setting in which a therapist practices. The management literature suggests that organizational variables, such as management support and financial resources, have an important impact on implementation of innovation (Klein, Conn, & Sorra, 2001). Within the mental health field, low workplace support, as reported by individuals who received training in the Triple P-Positive Parenting Program, was predictive of less self-reported implementation with clients 6 months following training (Sanders, Prinz, & Shapiro, 2009). Similarly, another study found that perceived setting barriers were negatively correlated with clinicians’ self-reported usage of cognitive-behavioral therapy (CBT) for depression (Lewis & Simons, 2011). More work is needed to determine whether these relationships remain when service delivery is measured objectively (e.g., via record or session tape review). Additional work has found that more engaged organizational climates, as rated by clinicians, have been associated with desirable treatment outcomes in youth, as assessed via parent report using standardized measures (Glisson & Green, 2011). Interestingly, the relationship between organizational climate and client outcomes was not mediated by the type or perceived quality of services delivered. These findings beg additional questions regarding how intra-organizational factors interplay with each other as well as how they interact with intervention characteristics and outer context factors to impact service delivery and client outcomes. In this study, we have explored individual perceptions of intra-organizational factors rather than aggregated organizational level constructs, but have drawn from this literature given the dearth of research on individual perceptions of intra-organizational factors.

Given the mixed findings to date and the importance of better understanding factors that contribute to implementation outcomes, the present study investigated the relationships between adopter characteristics (i.e., demographics; attitudes), individual perceptions of intra-organizational variables (e.g., organizational climate), and implementation outcomes (i.e., fidelity and penetration) for community therapists who received training and consultation in an EBP for youth anxiety. Specific implementation outcomes examined included independently rated fidelity as part of a performance based role-play (i.e., adherence and skill) and self-reported penetration (i.e., the number of anxious youth treated with the EBP divided by the number of anxious youth treated by trained providers).

METHOD

Participants

Demographics

Participants were 115 therapists from the northeast United States. Therapists were primarily female (90.4%) with a mean age of 35.93 (SD = 11.36). Ethnicity/race included: Caucasian (67.0%), African American (13.0%), Hispanic/Latino (5.2%), Asian (4.3%), Native American (.9%), and Other (5.2%). Twenty-nine percent of therapists reported being licensed. Therapists reported the following degrees: MA (38.3%), enrolled in graduate program (12.2%), MD (6.0%), PhD (5.2%), PsyD (4.3%), EdD (1.7%), and Other (24.3%).

Therapists reported having about 65.46 months of experience (SD = 82.38) and identifying with CBT moderately (M = 4.86, SD = 1.68; range = 1–7). Fifty percent of therapists had treated an anxious youth previously. Of those therapists, they reported treating 0–2 youth (M = .11, SD = .43), and receiving zero hours of supervision around the treatment of anxious youth. Therapists reported their current caseload at an average of 19.48 clients (SD = 23.72), and reported receiving 1.57 hours of supervision weekly (SD = 2.66).

Procedure

Recruitment and Screening

The Institutional Review Board at a large northeastern university approved all procedures. Participants were recruited from the community via professional listservs, directors of clinical training programs, and word of mouth. Two-hundred and four participants initially responded to the recruitment efforts; 115 enrolled. Eighty-nine therapists did not participate in the study for the following reasons: never responded to follow-up contact after they initially emailed the principal investigator (N = 37); not available for scheduled study dates (N = 25); did not confirm workshop attendance or complete pre-assessment (N = 15); not eligible due to exclusion criteria (N = 10); and no-show or cancelled last minute prior to training workshop (N = 2).

Participants were matched to one of six training dates based on their availability. Random assignment occurred at the level of training date (i.e., training dates were randomly assigned to condition) using equal allocation concealment. This randomization process has some limitations because randomization to training date is not the same as therapist randomization to condition. On the training day, all participants consented to participate.

Inclusion/exclusion criteria

Therapists had to (a) work in the community, (b) currently or in the future plan to work with children aged 8–17 with DSM-IV anxiety disorders, (c) identify with having received training within the mental health field, (d) volunteer to participate in the workshop and follow-up consultation, (e) read and speak English, and (f) have access to a computer or telephone for consultation. The exclusion criterion was prior training of >8 hours on CBT for child anxiety. We selected this criterion based on a methodological decision made in a similar trial (see Miller, Yahne, Moyers, Martinez, & Pirritano, 2004).

Participant Flow and Retention

Individuals who completed the pretraining assessment made up the intent-to-train (ITT) sample. Of the 115 participants in the ITT sample, 113 (98%) completed the post-training assessment and 100 (87%) completed the post-consultation assessment. Recruitment began in May 2009 and was completed by September 2009. Assessments for the study began in June 2009 and were completed by January 2010. Training occurred between June 2009 and October 2009.

Compensation

Participants received: (1) free lunch provided the day of the training, (2) a therapist manual, a child workbook, and therapy materials (e.g., therapy games) following completion of the posttraining assessment, and (3) a $15 dollar gift card following completion of the postconsultation assessment.

Measures

Inner Context Variables

Evidence-Based Practice Attitude Scale (EBPAS; Aarons, 2004)

The EBPAS is a 15-item tool that assesses participants’ attitudes towards the adoption and implementation of EBP. Four scales can be calculated: appeal, requirements, openness, and divergence (Aarons, 2004). Appeal (Chronbach’s α = .80) refers to the extent to which a therapist will adopt a new practice if it is intuitively appealing. Requirements (Chronbach’s α = .90) refer to the extent to which a therapist will adopt a new practice if required by the organization or legally mandated. Openness (Chronbach’s α =.78) is the extent to which a therapist is generally receptive to using new interventions. Divergence (Chronbach’s α = .59) is the extent to which a therapist perceives research-based treatments as not useful clinically (Aarons, 2004). For the purposes of our study, we recoded divergence so that higher scores on all subscales indicated highly rated appeal, requirements, openness, or divergence. The EBPAS demonstrates good internal consistency, subscale alphas range from .59 to .90 (Aarons & Sawitzky, 2006), and its validity is supported by its relationship with both therapist and organizational characteristics (Aarons, 2004).

Organizational Readiness for Change (ORC; Lehman, Greener, & Simpson, 2002)

The ORC contains 129 items which measure organizational characteristics on a Likert rating scale from 1 (strongly disagree) to 5 (strongly agree). The 18 scales comprise five major domains: (a) motivation, (b) resources, (c) staff attributes, (d) organizational climate, and (e) training climate. Psychometric properties for this instrument are strong (Lehman et al., 2002; Saldana, Chapman, Henggeler, & Rowland, 2007). There is evidence that the ORC is valid in indexing organizational level constructs when aggregating across therapist reports (Saldana et al., 2007). Note that in this investigation, we did not have enough therapists per agency to measure organizational level constructs, thus we use the ORC as a way to gauge individual perceptions of intra-organizational characteristics.

Clinician Demographics and Attitudes Questionnaire (CDAQ; Beidas, Barmish, & Kendall, 2009)

The CDAQ is a 15-item assessment tool that measures demographics, prior experience with CBT for youth anxiety and attitudes towards empirically supported treatments in general, as well as attitudes towards CBT for youth anxiety specifically. Psychometrics on the CDAQ indicate acceptable reliability with an intraclass coefficient (ICC) of .91 and Spearman Brown split-half reliability of .85 (Beidas et al., 2009).

Implementation Outcomes

Fidelity

Performance-Based Role Play (PBRP)

A structured PBRP (Dimeff et al., 2009) was used to assess participant’s fidelity (i.e., adherence and skill) in a simulated clinical setting. The PBRP consisted of a phone call with a standardized child client presenting for treatment for anxiety. Undergraduate research assistants blind to condition were trained to portray typical anxious youth (i.e., not overly compliant or defiant). Therapist participants were asked to take 8-minutes to prepare the client for an exposure task, a competency central to CBT. Three vignettes (used in a previous training study; Beidas et al., 2009) representative of anxious youth were created and rated by six child anxiety treatment experts to ensure comparability. Each participant was randomly assigned a vignette order for the three PBRPs. The role-plays were digitally recorded and later independently coded.

Adherence and Skill Checklist (ASCL; Beidas et al., 2009)

This measure was developed to measure both adherence to the content of CBT for youth anxiety and skill in delivery. Adherence was assessed by coding the presence of six core CBT competencies: (a) identification of somatic symptoms, (b) identification of anxious cognitions, (c) relaxation, (d) coping thoughts, (e) problem-solving, and (f) positive reinforcement. Skill was evaluated using a Likert scale from 1 (not well) to 7 (very well). Skill was rated as: “How skillful was the clinician’s performance in preparing the child for the exposure task using the cognitive-behavioral framework?” Note that this manner of rating skill is specific to skill in the application of specific CBT components; making it related to adherence. In other words, if a therapist did not adhere to the core CBT competencies, they would be unable to score highly on the skill variable.

Coders (one doctoral level psychology graduate student, two post-undergraduates, one honors undergraduate) were trained through readings, didactics, and supervised practice with feedback. Coders were blind to hypotheses, training condition, and time-point of the assessment. Rated adherence (ICC = .98) and skill (ICC = .92) demonstrated strong inter-rater reliability.

Penetration

Identification and Treatment of Anxious Youth (ITAY; Benjamin, Beidas, Edmunds, Cohen, & Kendall, 2010)

The ITAY is a self-report measure that assesses primary treatment setting, rates of treatment use since ending consultation, types of treatment modalities used, barriers of treatment use, and facilitators of treatment use. The measure involves both close-ended questions and 7-point Likert scales. In this study, consistent with the definition provided by Proctor and colleagues (2010), we defined penetration as the percentage of anxious youth treated with CBT over the past 3 months (i.e., anxious youth treated with CBT divided by anxious youth treated overall). Participants completed the ITAY 3 months following training (i.e., postconsultation assessment).

Assessment Procedure

At pretraining, participants completed measures of their demographics, attitudes, organizational characteristics, and the PBRP. Following training and 3 months of consultation, participants completed an assessment evaluating penetration of CBT for child anxiety and the PBRP.

Training and Consultation

Participants were randomized to training conditions: (1) routine training: a one-day workshop that covered the specific manual and procedures of CBT for child anxiety (i.e., training as usual), (2) computer training: computer training on CBT for child anxiety that was accomplished through a commercially-developed interactive DVD, and (3) augmented training: a one-day workshop that included a focus on principles of CBT and active learning (including behavioral role-play exercises). Given that no significant differences between the three conditions were identified on therapist fidelity, we chose to collapse across training conditions in our analyses (see Beidas et al., 2012).

Participants from all three conditions were provided weekly consultation via the WebEx virtual conferencing platform for three months following training. Participants could call in via telephone or computer to attend the 1-hour weekly virtual meeting. Those who used their computer were able to view a white-board and the individual leading the consultation via web camera. Consultation curriculum was designed with participant input and included case consultation, didactic topics (e.g., treating a client with comorbid depression), practice with concepts (e.g., relaxation), and assistance in implementation of the treatment within context (e.g., psychiatry clinic, school). Generally, consultation consisted of thirty minutes of client discussion and thirty minutes of didactics and behavioral rehearsal. Consultation included both instructor-led structured material and unstructured peer-guided material (Weingardt, Cucciare, Bellotti, & Lai, 2009). On average, the 108 consultation sessions lasted 52.57 minutes (SD = 10.79, range = 22–65) and had an average of 7.83 participants (SD = 4.52, range = 1–20). Number of cases discussed per call averaged 2.69 (SD = 1.90, range = 0–7). Participants attended an average of 7.15 consultations (SD = 3.17; range = 0–10) in the three-month period following their training (i.e., between the posttraining and postconsultation assessment). We refer the reader to Edmunds et al. (2013) for details regarding consultation.

Data analytic plan

The primary analyses were three multiple linear regressions. The three outcome variables included fidelity (i.e., adherence and skill) and penetration. The set of predictors included the following: adopter characteristics (i.e., experience, age, attitudes; as reported on the EBPAS and CDAQ) and individual perceptions of intra-organizational variables (i.e., motivation for change, resources, staff attributes, organizational climate, and training climate; as reported on the ORC).

All statistical assumptions were met, and no significant differences in inner context factors at pretraining were found when comparing conditions. ITT analyses were conducted and all participants (N = 115) were included, with the last observation carried forward for therapists who did not complete study assessment time-points (n = 13). For measures where there were missing data at pre-training, the mean of the condition was computed and imputed. Study completers were those who completed all three PBRPs. Of the 115 participants, 13 (11%) did not complete all PBRPs. There was no significant difference among conditions in study completion (χ² (2) = .11, p = .95).

RESULTS

Fidelity: Adherence and Skill

As reported previously (Beidas et al., 2012), the RM-ANOVA demonstrated that there was a main effect of time such that therapist adherence (F (2, 222) = 100.21, p< .001) and skill (F (2, 222) = 79.90, p <.001) improved after receiving the training and consultation package.

Penetration

Over the 3 months following training (i.e., the time period between the posttraining and postconsultation assessment), as measured by the ITAY¹, participants reported treating an average of 2.63 anxious youth (SD = 7.28, range = 0–65). Two hundred and forty two anxious youth were treated by the sample of clinicians in three months; participants reported providing CBT to 79% of these youth (N = 191). Other modalities included relaxation by itself (n=36, 62.1%), supportive therapy (n=29, 50%), family therapy (n=25, 43.1%), play therapy (n=18, 31%), peer support/group intervention (n=14, 24.1%), and other (n=11, 19%). Participants reported delivering about 8.11 (SD = 9.61) sessions per youth, and moderate involvement of parents in treatment (M = 3.48, SD = 1.28; 7 = highest involvement).

Relationship between adopter characteristics, individual perceptions of intra-organizational characteristics and adherence

A multiple regression examined the relationship between adherence and potential predictors, while controlling for baseline adherence and skill. A statistically significant relationship between the set of predictors and adherence was found (F (19, 108) = 2.29, p = .005. The R Square Change statistic for the increase in R² associated with the added predictors was 0.188. Using a proportional reduction in error interpretation for R², information provided by the added variables reduces error in predicting adherence by 18.8%. As in Table 1, one intra-organizational factor (i.e., organizational climate) had a significant positive regression weight, indicating that as individual perceptions of organizational climate increased, adherence also increased. Two individual adopter characteristics had significant negative weights, indicating that as experience in months and scores on the EBPAS requirements scales increased, adherence decreased. Other predictors were not significant.

Table 1.

Relationship between adopter characteristics, individual perceptions of intra-organizational characteristics and adherence

Predictor Variable ( N = 109)	Mean (SD)	b	β
Demographics
Experience (months)	65.14 (82.55)	−.006	−.304*
Age	35.95 (11.44)	.015	.110
EBPAS
EBPAS appeal	3.34 (.62)	.393	.157
EBPAS divergence^	.89 (.53)	−.078	−.027
EBPAS openness	3.15 (.58)	.203	.076
EBPAS requirement	2.75 (1.01)	−.465	−.302**
CDAQ
CDAQ opinion	5.98 (1.07)	.235	.162
CDAQ confidence	5.42 (1.49)	−.031	−.029
CDAQ motivation	6.56 (.76)	−.369	−.181
CDAQ usefulness	6.63 (.62)	−.025	−.010
ORC
Motivation for Change	33.31 (5.51)	−.007	−.024
Resources	33.62 (4.83)	−.018	−.057
Staff Attributes	36.18 (3.69)	−.101	−.240
Organizational Climate	33.33 (3.11)	.129	.259*
Training Climate	23.70 (6.19)	.050	.197

Note. EBPAS = Evidence Based Practice Attitude Scale (Aarons, 2004); CDAQ = Clinician Demographics and Attitudes Questionnaire (Beidas et al., 2009); ORC = Organizational Readiness for Change (Lehman et al., 2002). Note that higher scores on the EBPAS reflect more highly rated appeal, divergence, openness and requirements (range 0–4). Higher scores on the CDAQ reflect more positive opinions, confidence, motivation to learn, and belief in the utility of cognitive-behavioral therapy for child anxiety (range 1–7). Higher scores on the ORC reflect more highly rated motivation for change, resources, staff attributes, organizational climate, and training climate (range 10–50).

^{^}The mean of the divergence subcale is reverse scored.

^*≤ .05,

^**≤.01,

^***≤ .001. Note these analyses control for baseline skill and adherence.

Relationship between adopter characteristics, individual perceptions of intra-organizational characteristics and skill

A multiple regression examined the relationship between skill and potential predictors, while controlling for baseline adherence and skill. A statistically significant relationship between the set of predictors and skill was found (F (19, 108) = 1.879, p = .026. The R Square Change statistic for the increase in R² associated with the added predictors was 0.165. Using a proportional reduction in error interpretation for R², information provided by the added variables reduces error in predicting skill by 16.5%. As in Table 2, one individual adopter characteristic had a significant negative weight, indicating that as scores on the EBPAS requirements scales increased, skill decreased². Other predictors were not significant.

Table 2.

Relationship between adopter characteristics, individual perceptions of intra-organizational characteristics and skill

Predictor Variable (N = 109)	Mean (SD)	b	β
Demographics
Experience (months)	65.14 (82.55)	−.003	−.160
Age	35.95 (11.44)	.000	−.003
EBPAS
EBPAS appeal	3.34 (.62)	.419	.182
EBPAS divergence^	.89 (.53)	−.249	−.094
EBPAS openness	3.15 (.58)	.216	.088
EBPAS requirement	2.75 (1.01)	−.423	−.300**
CDAQ
CDAQ opinion	5.98 (1.07)	.337	.254
CDAQ confidence	5.42 (1.49)	−.057	-.059
CDAQ motivation	6.56 (.76)	-.591	-.317
CDAQ usefulness	6.63 (.62)	1.93	.084
ORC
Motivation for Change	33.31 (5.51)	−.001	−.004
Resources	33.62 (4.83)	−.016	−.055
Staff Attributes	36.18 (3.69)	−.046	−.120
Organizational Climate	33.33 (3.11)	.080	.175
Training Climate	23.70 (6.19)	.015	.064

Note. EBPAS = Evidence Based Practice Attitude Scale (Aarons, 2004); CDAQ = Clinician Demographics and Attitudes Questionnaire (Beidas et al., 2009); ORC = Organizational Readiness for Change (Lehman et al., 2002). Note that higher scores on the EBPAS reflect more highly rated appeal, divergence, openness and requirements (range 0–4). Higher scores on the CDAQ reflect more positive opinions, confidence, motivation to learn, and belief in the utility of cognitive-behavioral therapy for child anxiety (range 1–7). Higher scores on the ORC reflect more highly rated motivation for change, resources, staff attributes, organizational climate, and training climate (range 10– 50).

^{^}The mean of the divergence subcale is reverse scored.

^*≤ .05,

^**≤.01,

^***≤ .001. Note these analyses control for baseline skill and adherence.

Relationship between adopter characteristics, individual perceptions of intra-organizational characteristics and penetration

A multiple regression examined the relationship between penetration and potential predictors, while controlling for baseline adherence and skill. A statistically significant relationship between the set of predictors and penetration was not found (F (19, 88) = 1.205, p = .280. Means and regression weights are provided in Table 3.

Table 3.

Relationship between adopter characteristics, individual perceptions of intra-organizational characteristics and adoption

Predictor Variable (N = 89)	Mean (SD)	b	β
Demographics
Experience (months)	64.21 (81.30)	.001	.129
Age	35.54 (11.69)	.001	.030
EBPAS
EBPAS appeal	3.38 (.55)	−.136	−.161
EBPAS divergence^	.91 (.54)	.206	.239
EBPAS openness	3.16 (.57)	.060	.074
EBPAS requirement	2.72 (.97)	.069	.142
CDAQ
CDAQ opinion	6.02 (1.02)	−.101	−.221
CDAQ confidence	5.57 (1.35)	.013	.038
CDAQ motivation	6.57 (.71)	.130	.196
CDAQ usefulness	6.60 (.63)	−.036	−.049
ORC
Motivation for Change	33.32 (5.27)	−.09	−.098
Resources	33.59 (5.08)	.019	.203
Staff Attributes	36.12 (3.69)	−.001	−.009
Organizational Climate	33.39 (3.25)	−.021	−.148
Training Climate	23.68 (6.42)	.008	.111

Note. EBPAS = Evidence Based Practice Attitude Scale (Aarons, 2004); CDAQ = Clinician Demographics and Attitudes Questionnaire (Beidas et al., 2009); ORC = Organizational Readiness for Change (Lehman et al., 2002). Note that higher scores on the EBPAS reflect more highly rated appeal, divergence, openness and requirements (range 0–4). Higher scores on the CDAQ reflect more positive opinions, confidence, motivation to learn, and belief in the utility of cognitive-behavioral therapy for child anxiety (range 1–7). Higher scores on the ORC reflect more highly rated motivation for change, resources, staff attributes, organizational climate, and training climate (range 10–50).

^{^}The mean of the divergence subcale is reverse scored.

^*≤ .05,

^**≤.01,

^***≤ .001. Note these analyses control for baseline skill and adherence.

DISCUSSION

Therapist fidelity (i.e., adherence and skill), as rated by independent evaluators during a performance-based role-play, significantly improved after receiving the training and consultation package, and therapists reported high penetration (i.e., using CBT with a majority of their anxious youth patients) following training. We were interested in exploring if inner context variables as defined by the EPIS model, specifically adopter characteristics and individual perceptions of intra-organizational factors, related to implementation outcomes during the implementation phase of EPIS. Adopter characteristics were associated with fidelity (both adherence and skill) but not penetration. Individual perceptions of intra-organizational variables were related to fidelity (adherence specifically) but not skill or penetration. Similar to previous investigations (e.g., Garner et al., 2012), these results suggest a nuanced relationship between adopter characteristics and individual perceptions of intra-organizational variables and implementation outcomes. These results partially support the EPIS framework, which suggests a number of inner context factors which may be important at different stages of the implementation process (exploration, preparation, implementation, and sustainment). Additionally, the results link inner context factors to particular implementation outcomes from the Proctor et al. (2010) framework, which further advances the conceptual model.

Therapists reported high penetration – in other words, using CBT for child anxiety with the majority of anxious youth with whom they worked. However, they also reported using a variety of other modalities in the treatment of anxious youth, including non-evidence-based approaches (e.g., play therapy). These reports call into question the accuracy of the therapist report with regard to what they consider to CBT. It is possible that therapists rated themselves as using CBT if they used any components of the treatment rather than the whole package. These findings are consistent with previous studies, suggesting that therapists use a wide range of treatment strategies with youth (Garland et al., 2010). However, little is known about the effectiveness of mixing eclectic strategies with EBP on child outcomes. Therapists reported providing about 8 sessions in 3 months, which is a lower dosage than one might expect given that CBT for child anxiety is intended to be a weekly treatment (typically ranging from 12–16 sessions in 3 months; Kendall, Hudson, Gosch, Flannery-Schroeder, & Suveg, 2008; Walkup et al., 2008).

Adopter characteristics were associated with fidelity, including both adherence and skill. Specifically, certain therapist attitudes and demographics predicted adherence. One specific attitude, being more likely to adopt an EBP if required to do so by one’s organization, was related to lower adherence and skill scores after training and consultation. This finding may be explained by the fact that virtually none of the therapists who participated were mandated or required to use this treatment, rather they volunteered to learn this new treatment. Therefore, they may have been less motivated to apply the treatment in an adherent or skillful manner given that there were not organizational pressures to apply the treatment with adherence. Only one demographic variable was associated with adherence. As therapist experience increased, adherence to CBT for child anxiety decreased. This may be due to experience serving as a proxy for increased allegiance to other theoretical orientations. This finding is corroborated by other studies which showed that therapists with more clinical experience required more intensive training to become more adherent and competent (e.g., Martino et al., 2011). However, other studies have noted that having more clinical experience may be related to higher adherence and competence after training (Carpenter et al., 2012). It may be that other factors (e.g., what type of previous experience or what type of new skills are being taught) influence whether amount of previous experience improves or hinders implementation outcomes. Other important demographic variables noted to be related to adherence and skill in similar studies including longevity at organization, additional treatment certifications (Garner et al., 2012), and anxiety sensitivity (Harned, Dimeff, Woodcock, & Contreras, in press) were not measured in this study. It is important to note that the literature shows the least clarity with regard to which demographics consistently predict fidelity and this may be due to third variables that are unmeasured (i.e., inner- and outer-context variables). Future research that examines the interplay between and across inner- and outer-context variables on implementation outcomes is critical.

One individual perception of intra-organizational variables was associated with adherence. Interestingly, therapists who reported more positive organizational climates at their agency were more adherent to CBT for youth anxiety after training and consultation. This finding is consistent with previous research showing a relationship between therapist perceptions of certain aspects of organizational climate (e.g., job satisfaction, growth and advancement) and caregiver-reported therapist adherence in the delivery of MST (Schoenwald, Chapman, Sheidow, & Carter, 2009). One potential explanation is that therapists in more ideal organizational climates may be better able to focus on applying EBPs in an adherent matter versus therapists in chaotic and less ideal climates. Grounding this finding in other research that shows that more engaged organizational climates are associated with desirable treatment outcomes in youth (Glisson & Green, 2011) may suggest that the increased adherence we observed in this study could potentially lead to improved treatment outcome in the youths being treated. A meditational model was not supported in a study examining organizational climate, therapist adherence, and outcomes for youth treated with MST (Schoenwald et al., 2009). However, meditational models for other treatments, such as CBT for youth anxiety, may be an important area for future consideration.

Contrary to predictions, neither adopter characteristics nor individual perceptions of intra-organizational variables were associated with penetration. Although these findings suggest that clinicians can successfully use evidence-based treatments regardless of their demographics, attitudes, or organizational climate, they are at odds with an ecological perspective that hypothesizes the importance of inner context variables when implementing a treatment (Aarons et al., 2012). It is likely that these specific inner context variables are not predictive of penetration, and that other unmeasured inner context variables (e.g., innovation-values fit) would be more predictive. The lack of findings between inner context variables and penetration could also be due to one of the following issues. First, the study was underpowered to detect significant findings on the penetration variable because there were fewer participants to complete the penetration measure. Second, there may be a measurement problem, particularly with measurement of intra-organizational factors, given that these were individual perceptions of intra-organizational factors rather than aggregated organizational factors. Third, penetration was measured using self-report which is a known limitation. The current findings add to a decidedly mixed literature with regard to whether adopter characteristics and intra-organizational factors predict penetration and are somewhat inconsistent with previous studies. For example, studies have found that perceived setting barriers are negatively correlated with implementation outcomes (Harned et al., in press; Lewis & Simons, 2011). Discrepancies across studies call into question the adequacy of measurement as well as potential failure to investigate important third variables, such as client factors. Given that no clear answer has been reached regarding the interrelations among individual perceptions of intra-organizational variables and implementation outcomes and the importance of fostering long-term implementation of EBPs across a variety of organizational settings, further research in this area is important.

The present study has numerous strengths, including the randomization of training conditions, high completion rates of participants, and assessing inner context factors. Nonetheless, limitations should be noted. A first limitation has to do with the randomization process. We randomly assigned training dates to condition rather than randomly assigning therapists to condition for pragmatic reasons. However, it is possible that non-randomness could be introduced to the study design. For example, a therapist who signs up for the first date might be more motivated than a therapist who signs up for a later date. Also, because training occurred while recruitment was still ongoing, some therapists were prevented from signing up from earlier slots. Despite being typical in this literature, another weakness is that a number of measures were investigator-created (e.g., Dimeff et al., 2009). Although we used an innovative way to measure fidelity, it is not the traditional manner through which fidelity is measured, and did not include therapist fidelity observed during actual therapy sessions. To our knowledge, there is no information on the convergence between fidelity as rated through a behavioral role-play and actual in-session behavior (Beidas, Cross, & Dorsey, in press). Also, our penetration measure relied on self-report rather than objective measures. This is a limitation given research suggesting low concordance between self-report and observational data of session content (Hurlburt, Garland, Nguyen, & Brookman-Frazee, 2010).

Although the study included an organizational measure, organizational climate may not have been optimally assessed given that other measures exist to assess these constructs (Glisson, 2002). Aggregating across multiple ratings of organizational factors provides a better estimate of organizational culture and climate than individual accounts, yet this approach was not used in the present study because we did not have a large enough number of participants working in the same organizations. The current study may be more accurately described as an examination of individual therapists’ perceptions of their workplaces’ organizational climate and culture. Additional potential limitations concern the study’s generalizability. Its focus on CBT for youth anxiety may not generalize to training in other EBPs. Also, as previously mentioned, the participants were self-referred participants who were motivated to volunteer for training. Different findings may have emerged if participants included less motivated individuals who were mandated to receive training. Further, only 54 out of the 115 participants identified an anxious youth to treat in the consultation period following training. This is curious given that these individuals sought out this type of specialty training and reported that they either currently or planned to treat anxious youth in the future. This may be potentially due to difficulty identifying “pure” anxiety cases in community settings (Beidas et al., 2012; Ehrenreich-May et al., 2010; Southam-Gerow, Silverman, & Kendall, 2006).

Conclusion and Future Directions

The EPIS framework posits that inner context variables, such as adopter characteristics and intra-organizational variables, are likely to impact implementation outcomes (Aarons et al., 2012). The present results found partial support for this perspective. Future work is necessary to further delineate the relationships among inner context variables and implementation outcomes. Future studies would benefit from seeking a wider range of participants, including self-referred and mandated participants, in order to tease apart whether inner context variables differentially influence outcomes across these groups. Improved measurement of inner context variables and implementation outcomes across different stages of the implementation process would allow for greater clarity regarding the relationships among these factors. Specifically, outcome measures should be varied (Proctor et al., 2011) and should include objective ratings of fidelity and penetration in addition to self-report measures. Measurement of organizational variables should include multiple raters within an organization in order to determine whether aggregate measures differentially predict training outcomes as compared to individual perceptions of organizational variables. In addition to investigating whether inner context variables predict implementation outcomes, investigations can examine the relations between inner context variables and specific training techniques (e.g., whether learning style moderates the relation between training techniques and outcomes). Greater understanding of these relationships is needed to devise optimal training approaches that can be used to increase the chances youth receive optimal care regardless of the setting in which they are served.