Interactions between clinician and organizational characteristics to predict cognitive-behavioral and psychodynamic therapy use

Abstract

Conceptual models of implementation posit contextual factors and their associations with evidence-based practice (EBP) use at multiple levels and suggest these factors exhibit complex cross-level interactions. Little empirical work has examined these interactions, which is critical to advancing causal implementation theory and optimizing implementation strategy design. Mixed effects regression examined cross-level interactions between clinician (knowledge, attitudes) and organizational characteristics (culture, climate) to predict cognitive-behavioral and psychodynamic therapy use with youth (N = 247 clinicians across 28 agencies). Results indicated several interactions, highlighting the importance of attending to interactions between variables at multiple levels to advance multilevel implementation theory and strategies.

Interactions between clinician and organizational characteristics to predict cognitive-behavioral and psychodynamic therapy practice use

Identifying factors that predict clinicians’ use of evidence-based practices (EBPs) in mental health settings is important for developing effective implementation strategies to improve the quality of care for those seeking treatment (Proctor et al., 2011). Conceptual models of implementation posit factors influencing clinician practice use at multiple levels (e.g., individual, organizational). Consistent with this, there is an extensive literature examining predictors of clinician practice use documenting main effects of variables across these levels on implementation outcomes in mental health service systems (Becker, Smith, & Jensen-Doss, 2013; Beidas et al., 2015; Brookman-Frazee, Haine, Baker-Ericzén, Zoffness, & Garland, 2010; Czincz & Romano, 2013; Henggeler et al., 2008; Higa-McMillan, Nakamura, Morris, Jackson, & Slavin, 2015; Jensen-Doss, Hawley, Lopez, & Osterberg, 2009; Lim, Nakamura, Higa-McMillan, Shimabukuro, & Slavin, 2012; Nelson & Steele, 2007). However, how these multilevel factors interact with one another to predict practice use is poorly understood. While many conceptual models imply interactive or synergistic cross-level effects, few speculate the direction or strength of cross-level interactions and therefore lack causal specificity (Damschroder et al., 2009; Tabak, Khoong, Chambers, & Brownson, 2012; Wandersman et al., 2008). Some frameworks suggest that general and/or innovation-specific organizational context is a necessary foundation for successful implementation to occur (Durlak & DuPre, 2008; Powell et al., 2017) and emerging evidence has started to unpack the relationship between these contextual factors (e.g., the relationship between organizational and therapist characteristics; Powell et al., 2017).

While there is a burgeoning literature examining the mechanisms by which organizational processes influence EBP implementation (e.g., Williams, Glisson, Hemmelgarn, & Green, 2017a). to our knowledge, no empirical work has yet examined how these organizational processes interact with clinician characteristics to predict clinician practice use. Identification of potential mechanisms by which implementation strategies may influence EBP use has been identified as a key step toward advancing the science of implementation (Lewis et al., 2018). Empirical examination of how theorized implementation predictors interact can improve our understanding of the mechanistic processes through which implementation strategies affect practice use, guide selection of targeted implementation and de-implementation strategies, and identify strategic points of intervention. Prior work has established that both organizational and clinician factors influence individual clinician practice use (Beidas et al., 2015; Aarons, Ehrhart, Farahnak, & Hurlburt, 2015), although the mechanisms by which these levels interact with one another to influence practice use are poorly understood (Brookman-Frazee & Stahmer, 2018; Glisson & Williams, 2015; Lewis et al., 2018). Elucidating cross-level predictive pathways in the context of EBP implementation efforts can inform our understanding of these cross-level mechanisms that influence clinician uptake of EBPs, which in turn can inform the development of implementation strategies to target these causal pathways.

Building on work examining the relative contributions of clinician and organizational variables as predictors of clinician practice use in the publicly-funded Philadelphia mental health system (Beidas et al., 2015; Powell et al., 2017), we expand this prior work to examine the interactive processes across these factors that influence clinician practice. We aim to highlight the importance of specifying and testing cross-level interactions at the two levels most proximal to implementation - the clinician and the organization- (Damschroder et al., 2009). We then draw out implications of these interactions to begin to unpack how therapist-level variables moderate the influence of organizational culture and climate on practice use. Recent work has highlighted the importance of considering such boundary conditions of organizational influences (i.e., the “who, where, when” questions associated with the relationship between organizational characteristics and clinician behavior; Whetten, 1989) to better understand the relationship between organizational factors and outcomes of interest (Busse, Kach, & Wagner, 2016).

We explicitly selected variables for inclusion in this study that have shown both theoretical and empirical links with clinician practice use in mental health. Clinician variables of interest were attitudes toward and knowledge of EBPs, two key clinician-level constructs in implementation frameworks (Damschroder et al., 2009) that are considered proximal antecedents of EBP delivery (Brookman-Frazee & Stahmer, 2018; Lyon et al., 2013). For organizational variables, a distinction has been made between general and implementation specific organizational factors (Durlak & DuPre, 2008), both of which are linked to successful EBP implementation (Williams, Ehrhart, Aarons, Marcus & Beidas, 2018). Thus, we examined proficient organizational culture (norms and expectations that clinicians are competent in up-to-date practices and prioritize client well-being) and functional climate (perceptions that the work environment supports clinicians’ personal well-being) as general organizational factors associated with EBP use (Glisson, 2002). We looked at implementation climate (perceptions that the use of EBP is expected, rewarded, and supported by the organization; innovation specific; Ehrhart, Aarons, & Farahnak, 2014) as an implementation-specific factor. These specific constructs were selected as prior empirical work has highlighted these constructs as particularly important for optimizing EBP delivery (Aarons et al., 2012; Beidas et al., 2015; Ehrhart et al., 2016; Williams & Glisson, 2013; Williams & Glisson, 2014a; Olin et al., 2014) and are consistent with identified possible mechanisms of change in organizational interventions (e.g., the Availability Responsiveness, and Continuity [ARC] intervention; Williams, et al., 2017).

Based on work showing that organizational culture and climate are associated with therapists’ EBP attitudes, knowledge, and use (Glisson, Green, & Williams, 2012; Glisson, Williams, Hemmelgarn, Proctor, & Green, 2016; Powell et al., 2017; Williams et al., 2017b), we anticipated that interactions between organizational culture and climate with clinician EBP attitudes and knowledge would explain practice use in a more nuanced way than direct associations. Specifically, we hypothesized that there would be boundary conditions to the effect of organizational culture and climate, with organizational factors predictive of EBP use only within the context of optimal levels of the examined clinician-variables (i.e., in the presence of positive attitudes, high knowledge) and that organizational variables would be less and/or not associated with clinician EBP use when clinicians had negative attitudes and/or low knowledge. Based on findings suggesting the importance of organizational factors relative to clinician factors as main effects of EBP use (Beidas et al., 2015; Durlak & DuPre, 2008; Kam, Greenberg, & Walls, 2003; Powell et al., 2017), we expected that in the absence of interactions, organizational characteristics would predict EBP use more strongly than clinician characteristics.

To evaluate these hypotheses, we were interested in two types of therapy approaches that vary in the extent to which they have been tested and studied in the literature: both EBP and other practices. Prior work has shown that when EBPs are delivered, they are often delivered alongside a variety of other treatment strategies that vary in the extent to which they have empirical support for their efficacy and effectiveness (Beidas et al., 2017; Garland et al., 2010). As a result, in addition to examining factors associated with EBP use to inform implementation strategy development, there is a need to examine factors associated with use of other, less evidence-based treatment strategies. Such work could both: a) help inform an understanding of why clinicians continue to utilize treatment practices with less evidence within the context of EBP implementation efforts, and b) potentially inform the design of strategies to de-implement the use of such other treatment strategies and facilitate the delivery of the EBP as intended (Niven et al., 2015; Prasad & Ioannidis, 2014).

Our EBP of interest was cognitive behavioral therapy (CBT), which has the most empirical support for its efficacy across a range of youth mental health disorders (Dorsey et al., 2017; Higa-McMillan, Francis, Rith-Najarian, & Chorpita, 2016; Hofmann, Asnaani, Vonk, Sawyer, & Fang, 2012; McCart & Sheidow, 2016; Weisz et al., 2017; Zhou et al., 2015). CBT has been a target of implementation efforts across the country (Novins, Green, Legha, & Aarons, 2013), including Philadelphia (Powell et al., 2016), where the current study was conducted. As a result, we would expect that community clinics high in organizational proficiency, functionality, and implementation climate would place an emphasis on the use of CBT techniques amongst their clinicians compared with other treatment practices with less evidence. Thus, CBT is an ideal EBP for advancing our understanding of implementation determinants and cross-level processes.

In terms of other treatment strategies, we were particularly interested in clinician use of psychodynamic therapy. While small studies of psychodynamic techniques show modest success (Abbass, Rabung, Leichsenring, Refseth, & Midgley, 2013; Midgley, O’Keefe, French, & Kennedy, 2017), psychodynamic therapy is generally considered to not yet have sufficient empirical support for its use with youth (De Nadai & Storch, 2013; Midgley et al., 2017) to be considered an EBP. Despite this, use of these techniques often persists in community practice (Beidas et al., 2017) and the organizational influences on use of these less evidence-based practices is poorly understood. Importantly, psychodynamic therapies are generally thought to be less cost-effective than CBT (Egger et al., 2016). Thus, low-resourced community settings may wish to prioritize use of CBT over psychodynamic strateiges, making it an important intervention for studying possible mechanisms of de-implementation.

Methods

Participants and Procedures

All procedures were approved by the City of Philadelphia and the University of Pennsylvania institutional review boards. Cross-sectional surveys were collected from therapists as part of a larger longitudinal study examining the impact of a policy mandate on clinician use of EBPs for youth clients within the Philadelphia community mental health system (Beidas et al., 2013). The Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) in Philadelphia had supported four EBP implementation initiatives at the time of data collection (cognitive therapy, prolonged exposure, trauma-focused-CBT, and dialectical behavior therapy; see Powell et al., 2016 for an overview). Of note, all EBP initiatives placed emphasis on the use of CBT strategies, providing further support for studying CBT as the EBP of interest in this study. Clinicians attended a two-hour meeting during which research staff presented the study. Interested clinicians provided informed consent and completed questionnaires independently during this group meeting; they received $50 for participating. While clinicians completed measures, agency leadership was not present to facilitate clinician comfort in answering questions honestly.

Participants were 247 community mental health clinicians working in 28 youth-serving agencies in Philadelphia. Approximately 58% of clinicians employed by the 28 organizations participated in 2015. Participants averaged 38.7 years old (SD = 11.9), were largely female (n = 192, 77.7%), and of heterogeneous ethnic backgrounds (40.9% White, 16.2% Hispanic/Latino; 30.0% African American, 4.5% Asian, 4.0% Multiracial, 2% Other; 6 individuals did not report on their ethnicity). Average years of experience was 10.1 years (SD = 8.6). Just under half of clinicians (46.6%; n = 115) had participated in one of the EBP training initiatives sponsored by the Philadelphia DBHIDS (see Powell et al., 2016).

Measures

Clinician practice use.

Clinicians’ use of CBT and psychodynamic therapy was measured using the Therapy Procedures Checklist – Family Revised (Weersing, Weisz, & Donenberg, 2002), a well-validated self-report measure (Kolko, Cohen, Mannarino, Baumann, & Knudsen, 2009; Weersing et al., 2002). Consistent with the original developed measure (Weersing et al., 2002) and prior large-scale studies of clinician practice use in the public mental health system (e.g., Beidas et al., 2015; Beidas et al., 2017), clinicians were asked to select a representative client on their caseload and indicate the extent to which they used 62 therapeutic techniques with that client over the course of that client’s treatment. All items are rated on a 5-point Likert scale that ranges from 1 (Rarely Use) to 5 (Use Most of the Time). To our knowledge, the TPC-FR is the only published self-report instrument of therapist practices that spans multiple treatment protocols and theoretical orientations; techniques measured are drawn from family, CBT, and psychodynamic therapy strategies. The TPC-FR yields subscales with total scores for each of these three treatment families. Prior work suggests that the TPC-FR demonstrates adequate psychometric properties, including test-retest reliability and sensitivity to within-therapist changes in technique use (Kolko, et al., 2009; Weersing et al., 2002). Only the CBT and psychodynamic scales were used in this study, given our specific interest in clinician use of these therapeutic strategies. Example CBT items include “using time-out from reinforcement” and “training the child to recognize maladaptive thoughts.” Example psychodynamic items include “trying to understand the effects of early life experiences” and “analyzing the child’s dreams, fantasies, or other products (e.g., art).” Both subscales showed excellent internal consistency in this sample (CBT α = .92, Psychodynamic α = .87).

Clinician characteristics.

Clinicians’ attitudes towards EBP were assessed using the Evidence-Based Practice Attitude Scale (EBPAS), which yields four subscales: appeal (EBP is intuitively appealing), requirements (would use EBP if required), openness (general openness to innovation), and divergence (perceived divergence between EBP and current practices; Aarons, 2004). Higher scores indicate more favorable attitudes. The EBPAS has excellent psychometric properties (Aarons et al., 2010); in this sample, subscale alphas were >0.70 (α range= .72-.89), with the exception of the divergence subscale (α = .60), which showed internal consistency somewhat lower than those published in the national norms (Aarons et al., 2010). As such, we did not include the divergence subscale.

EBP knowledge was assessed with the Knowledge of Evidence-Based Services Questionnaire (KEBSQ; Stumpf, Higa-McMillan, & Chorpita, 2009), which is one of the only existing measures of youth mental health practice knowledge with psychometric support. The KEBSQ is a 40-item multiple true-false self-report measure that assesses knowledge of EBP techniques on a scale of 0–160; higher scores indicate more EBP knowledge. Prior analysis demonstrated that the KEBSQ demonstrates acceptable psychometric properties, including test-retest reliability, discriminative validity, and sensitivity to change over time (Stumpf et al., 2009).

Organizational characteristics.

Organizational proficiency culture and functional climate were assessed using the Organizational Social Context Measurement System (OSC), a well-validated measure developed specifically to assess organizational culture and climate in mental health and social service organizations (Glisson et al., 2012; Glisson et al., 2008). These factors are theorized to be key organizational factors linked to EBP use (Williams & Glisson, 2014b; Williams, Glisson, Hemmelgarn, & Green, 2017a).The proficiency scale assesses an organization’s proficiency culture, or the norms and behavioral expectations for clinicians to place the well-being of clients first, be responsive to client needs, and be competent in up-to-date treatment practices. The functionality scale assesses an organization’s functional climate, or the extent to which clinicians perceive they can get their job done effectively and have a well-defined understanding of how they fit within the organization. Scoring yields a T-score with a μ = 50 and σ = 10 (Glisson et al., 2008). OSC profiles were calculated by the OSC development team. Of note, OSC profiles for each organization are usually created by creating an average score by agency composed of responses from front line service providers. However, two of our organizations did not have enough front line providers to create the OSC profile without including agency leaders. As aggregate statistics (i.e., awg) were acceptable and empirical work suggests that agreement in small organizations between leaders and followers is high (Beidas et al., 2018); we included agency leaders in the total profile score for those two organizations only. Higher scores on the OSC subscales indicate more proficient and functional organizational cultures.

Implementation climate was assessed via the 18-item Implementation Climate Scale (ICS) total score (Ehrhart et al., 2014). The ICS assesses multiple factors that contribute to successful implementation including: organizational focus on EBP, educational support for EBP, recognition for using EBP; rewards for using EBP; selection of staff for EBP; and selection of staff for openness. The ICS has demonstrated good reliability and validity (Ehrhart et al., 2014). Higher scores on the ICS indicate higher levels of implementation climate. Organization-level scores were calculated by aggregating (i.e., averaging) items across clinicians within each organization based on evidence of adequate within-organization agreement (average within group agreement values = .73-.99; LeBreton & Senter, 2008).

Analysis Plan.

Preliminary analyses confirmed the need for a multilevel approach, as there was significant inter-organizational variance in CBT (ICC = .15) and psychodynamic therapy (ICC = .06) use. Of note, while the ICC for psychodynamic therapy is considered relatively small, it is consistent with prior studies in community mental health (Beidas et al., 2015) and is thought to be a meaningful proportion of variance to proceed with examining predictors (Zyzanski, Flocke, & Dickinson, 2004). We first examined bivariate relationships between each predictor of interest and clinicians’ EBP use via two-level mixed effects models with random organization intercepts to account for clinicians nested within agencies. Next, we ran two-level mixed-effects regression models to test the cross-level interactions between organizational (level 2) and clinician (level 1) characteristics in predicting clinicians’ CBT and psychodynamic use. Each model contained all clinician predictors, a single organizational characteristic, and their interactions. All variables were grand-mean centered prior to computing interaction terms. Significant interactions were probed via simple slopes and plotted at ±1.5 SD of the mean of each independent variable. Main effects were interpreted for variables without significant interactions. All analyses were conducted using SAS PROC MIXED and controlled for whether clinicians had participated in a system-sponsored CBT training (Powell et al., 2016) and years of clinical experience. All presented coefficients are unstandardized to facilitate interpretation; coefficients reflect estimated changes in the dependent variable as a result of one point increases in the independent variable, controlling for the effects of other variables in the model.

Results

Bivariate Analyses.

Table 1 shows clinician and organization means, standard deviations, and bivariate results illustrating the relationships between each variable and practice use. Greater CBT use was associated with higher EBPAS Openness scores, lower EBP knowledge, and more proficient organizational cultures. Higher psychodynamic therapy use was associated with higher EBPAS Openness scores and lower EBP knowledge.

Table 1.

Descriptive statistics of study variables and bivariate predictors of practice use

	Descriptive Statistics		Relationship to CBT Use1		Relationship to Psychodynamic Therapy Use1
	M	SD	B (SE)	p	B (SE)	p
Organizational Predictors
Proficient culture	53.86	9.30	.02 (.01)	.007	.01 (.01)	.17
Functional climate	62.52	9.84	.01 (.01)	.35	.01 (.01)	.42
Implementation climate	2.01	.45	.21 (.14)	.13	.02 (.14)	.91
Clinician predictors
EBP Knowledge	96.19	8.90	−.01 (.004)	.007	−.03 (.05)	<.001
EBPAS Requirements	2.86	1.00	.033 (.03)	.39	.05 (.04)	.28
EBPAS Appeal	3.17	.68	.002 (.06)	.97	−.04 (.06)	.51
EBPAS Openness	3.08	.71	.19 (.06)	.001	.13 (.06)	.04

¹Results reflect two-level mixed effects analysis.

Interactions Predicting CBT.

Table 2 shows results of the mixed effects models examining the cross-level interactions between clinician and organization characteristics in predicting CBT use; Figure 1 illustrates the patterns of significant interactions. There was a significant interaction between the EBPAS Appeal subscale and both proficiency culture and functional climate (proficiency b = .02, p = .003; functionality b = .02, p = .004) scales, such that more proficient cultures and more functional climates predicted greater CBT use when clinicians viewed EBPs as intuitively appealing (proficiency b = .04, p < .001; functionality b = .02, p = .06), but was not associated with CBT use when clinicians viewed EBPs as unappealing (proficiency b = −.002, p = .82; functionality b = −.01, p = .51). There was also a significant interaction between EBP knowledge and proficiency culture (b = −.01, p = .03), such that higher proficiency culture was associated with higher CBT use when clinicians had low EBP knowledge (b = .03, p = .001) but was not related at high levels of EBP knowledge (b < -.001, p = .99; Figure 2). Implementation climate did not interact with any clinician variables to predict CBT use.

Table 2.

Cross-level interactions and main effects predicting practice use

	CBT Use		Psychodynamic Use
	Main Effect Coefficient	Interaction Coefficient	Main Effect Coefficient	Interaction Coefficient
	b (SE)	b (SE)
Proficient Culture Model
Proficient culture (OSC)	.01 (.01)*	--	<.01 (.01)	--
EBPAS Requirements	<.01 (.04)	<-.01 (.01)	.07 (.05)	−.01 (.01)
EBPAS Appeal	<.01 (.06)	.02 (.01)**	−.05 (.07)	.01 (.01)
EBPAS Openness	.17 (.06)**	<-.01 (.01)	.13 (.17)	<.01 (<.01)
EBP Knowledge	−.01 (<.01)	<-.01 (<.01)*	−.03 (.01)**	<-.01 (<.01)
Functional Climate Model
Functional climate (OSC)	<.01 (.01)	--	<-.01 (.01)	--
EBPAS Requirements	.02 (.04)	−.01 (<.01)	.08 (.05)	−.01 (.01)
EBPAS Appeal	−.03 (.07)	.02 (.01)**	−.10 (.07)	.01 (.01)
EBPAS Openness	.21 (.06)**	−.01 (.01)	.15 (.07)*	<-.01 (.01)
EBP Knowledge	−.01 (<.01)	<-.01 (<.01)	−.03 (.01)**	<.01 (.01)
Implementation Climate Model
Implementation climate (ICS)	.08 (.05)	--	−.09 (.10)	--
EBPAS Requirements	−.01 (.12)	−.05 (09)	.06 (.05)	−.03 (.10)
EBPAS Appeal	−.02 (.06)	.12 (.13)	−.09 (.07)	.05 (.15)
EBPAS Openness	.17 (.06)**	−.21 (.14)	.15 (.07)*	.02 (.15)
EBP Knowledge	−.01 (<.01)	<-.01 (.01)	−.03 (.01)**	−.02 (.01)*

Note.

^*p < .05.

^**p < .01. ICS = Implementation Climate Scale. OCS = Organizational Social Context. EBPAS = Evidence-Based Practices Attitudes Scale. Interaction coefficients were of primary interest and main effects were interpreted in the absence of significant interactions.

Figure 1.

Interactions predicting CBT Use

Figure 2.

Interactions predicting Psychodynamic Therapy Use

Main Effects of CBT.

As not all predictors showed interactions, a follow-up mixed-effects regression model containing only the predictors (i.e., no interaction terms) indicated that higher clinician openness to new practices was a main effect of more CBT use (EBPAS Openness b = .17), controlling for all other predictors (ps < .01). Implementation climate and the EBPAS Requirements subscale were not main effects of CBT use.

Interactions Predicting Psychodynamic Therapy.

Table 2 shows the results for psychodynamic therapy; Figure 2 illustrates the pattern for significant interactions. There was a significant interaction between EBP knowledge and implementation climate (b = −.02, p = .04), such that lower EBP knowledge was marginally associated with higher use of psychodynamic strategies at low levels of implementation climate (b = −.02, p = .05) but was more strongly associated with lower psychodynamic strategy use in the presence of high implementation climate (b = −.04, p < .001). Proficiency culture and functional climate did not interact with clinician variables to predict psychodynamic strategy use.

Main Effects of Psychodynamic Therapy.

A mixed-effects model containing no interactions indicated that higher clinician openness to new practices was a main effect of more psychodynamic therapy use (EBPAS Openness b = .14), controlling for all other predictors in the model (p < .05). Proficiency culture, functional climate, and the EBPAS Appeal and Requirements subscales were not main effects of psychodynamic therapy use.

Discussion

Although conceptual models of implementation suggest that clinician and organization factors interact to influence practice use, few studies have explored this empirically. Empirical work in this vein is essential to furthering causal theory in implementation science. Overall, our findings suggested that the extent to which organizational factors influence a clinician’s practice may be dependent on clinician attitudes toward and knowledge of EBPs. This is consistent with conceptual theory suggesting the presence of cross-level interactions. Specifically, findings provide preliminary support for a “necessary but not sufficient” conceptualization of the role of organizational culture and climate. In other words, results supported the assertion that optimizing organizational culture and climate is important for successful EBP use, but also suggested that there may be boundaries to the effect of organizational context as a function of clinician characteristics.

Findings from this study suggest that optimal outcomes (highest CBT use, lowest psychodynamic technique use) may occur when there is synergy between organizational and clinician constructs. This is consistent with prior work that suggests the need for multi-level implementation and de-implementation strategies. However, our findings extend prior literature by highlighting that fostering an optimal organizational culture and climate to facilitate EBP use be an insufficient implementation strategy when clinicians in an organization hold negative attitudes and lower EBP knowledge. Thus, our data suggest that it is possible that organizational-level implementation strategies may be sufficient when clinicians are relatively positive to and generally knowledgeable about EBPs. However, a multi-level approach targeting both clinicians and organizations is likely to be particularly important when implementing EBPs for clinicians with lower knowledge and more negative toward EBPs.

Importantly, our results demonstrated that while multiple clinician and organization variables were predictive of clinician practices in this study, relying on main effects analyses alone (as is typical in the extant literature) was misleading. For example, while bivariate associations indicated that clinicians working within more proficient cultures (i.e., those that place emphasis on being up to date with the latest evidence and improving client well-being) reported higher CBT use on average, interactions indicated proficient culture was associated with higher CBT use primarily for those clinicians who viewed EBPs as appealing; there was no effect of proficient culture on CBT use for clinicians with more negative attitudes. Similarly, neither functional organizational climate nor implementation climate were associated with clinician practice use in the bivariate analyses but both variables showed conditional effects on outcomes. Specifically, higher functional climates predicted increased CBT use only for those clinicians who view EBPs as appealing and higher implementation climates predicted lower psychodynamic therapy use predominantly only when clinician knowledge of EBP was high. This highlights the importance for future empirical work to move beyond testing the main effects of predictors, as examining main effects alone obscures a more complex picture of interacting variables.

All significant interactions in this study provided support for a synergistic effect between therapist characteristics and organizational culture and climate. Hypothesizing specific explanations for these interaction results is critical to building casual theory and understanding potential causal pathways influencing clinician EBP use. Identifying these causal pathways will allow identification of possible mechanisms that can be targeted in future implementation studies. With respect to interactions predicting CBT use, examination of items on the EBPAS appeal subscale suggest that certain items assess how likely clinicians are to use EBP if it is used by colleagues who are happy with it (Aarons, 2005). A respondent’s attitudes on the appeal subscale may therefore be more malleable to the organizational context as a function of the extent to which an individual looks to and connects with others in their organization to inform their practice. EBP knowledge also interacted with proficiency culture, suggesting that higher proficiency culture positively impacted CBT use primarily for clinicians with lower EBP knowledge, with less impact for those with high knowledge. One possible explanation for this is that organizational cultures in which clinicians are expected to be up to date on evidence-based practices effectively pull clinicians with lower EBP knowledge toward the mean with respect to CBT use.

With respect to potential explanations for interactions predicting psychodynamic technique use, a combination of high EBP knowledge and high implementation climate led to the lowest levels of psychodynamic technique use. This suggests that fostering a strong implementation climate may have an impact on reducing use of, or de-implementing, psychodynamic interventions, but only when clinicians have a strong knowledge base of EBPs. It is also worth noting that there was relatively less variance in psychodynamic practice use attributable to the organization than for CBT (psychodynamic ICC = .06, CBT ICC = .15). Given that data were collected within a system actively training and supporting clinician use of CBT across organizations (and not psychodynamic techniques), it is possible that higher use of psychodynamic practices use may represent a more “independent” decision and thus primarily driven by clinician-level factors. This would also be consistent with prior work suggesting that organizational factors may function differently as a function of the extent to which an intervention requires more interdependence within an organization (Jacobs et al., 2014).

Contrary to our hypothesis that in the absence of interactions organizational factors would more strongly relate to clinician practice use, the only main effect was a clinician-level variable. Clinician openness to new practices related to higher CBT and psychodynamic technique use regardless of organizational context. The EBPAS openness items may tap into a more general openness to innovation, rather than EBP specifically (Aarons, 2005), and clinician openness may be a primary driver of adopting a variety of innovations into practice (i.e., eclecticism). Given that openness can be construed as a personality construct (McCrae & Costa, 1987), it may be less malleable to organizational influences. This further supports the importance of multilevel implementation and de-implementation strategies. Taken together, results point to several hypotheses that can be tested in future work to further understanding of the causal mechanisms at play within the context of implementation efforts.

Several limitations are worth noting. While we attempted to select variables consistent with prior theory and empirical work to enhance parsimony, it is possible other organizational factors interact with clinician variables to predict practice use or that there may be interactions within levels (e.g., organizational culture with climate; (Williams & Glisson, 2014b)). Additionally, data were cross-sectional, and we were unable to test mediation relationships, (e.g., whether organizational context influences EBP use through changes in therapists’ attitudes); the presence of interactions in this study highlights the importance of examining the mechanisms underlying these interactions in prospective work. We also looked at implementation climate for EBPs broadly, consistent with published psychometrics of the ICS (Ehrhart et al., 2014; Ehrhart, Torres, Wright, Martinez, & Aarons, 2016); other work has found it to be beneficial to ask about innovations at a more specific level (e.g., CBT rather than EBP; Jacobs, Weiner, & Bunger, 2014). Results may have differed had we asked about implementation climate for CBT specifically, rather than EBP broadly.

In addition, while a major strength of this paper is our ability to extend prior work examining relative contributions of clinician and organizational factors to EBP use that used similar, self-reported EBP use to reach conclusions, self-reported EBP use has limitations. Self-report does not always correlate with observational data (Hurlburt, Garland, Nguyen, & Brookman-Frazee, 2010) and clinicians may over-report the amount of EBP in their practice (Creed, Wolk, Feinberg, Evans, & Beck, 2016). We also asked clinicians to report their use of therapy techniques with a representative client on their caseload, rather than a comprehensive examination of therapist practices. It is not known the extent to which reports about a representative client will generalize across a full clinician caseload.

Finally, some of our coefficients were relatively small; there is likely variation in clinician practices unexplained by our predictors and interactions. One important variable that likely contributes to variation is clinician practices is client characteristics (Benjamin Wolk et al., 2016) and we did not assess whether the specific CBT strategies clinicians endorsed were clinically appropriate for their client’s unique presentations. While the examination of three-way interactions (organization by therapist by client characteristics) would stretch the bounds of our sample beyond appropriate, future work with larger datasets should explore these complex interactive processes to more fully explain variation in clinician practice use. Furthermore, while self-report represents the most feasible method of assessing clinician practice (Schoenwald & Garland, 2013), future work would benefit from replicating findings with observational data and with multiple identified clients on clinicians’ caseloads. In addition, it will be important for all interactions to be replicated in additional samples.

Conclusions

Findings support frameworks suggesting a complex interplay between organizational and individual implementation determinants as they relate to practice use and extends theory by empirically demonstrating there may be boundary conditions to the impact of organizational culture and climate for EBP implementation in mental health. Results underscore the importance of future empirical work studying interaction effects within the context of implementation efforts and suggest the potential value of attending to clinician characteristics in tandem with organizational culture and climate in implementation and de-implementation efforts.