Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Jan 1.
Published in final edited form as: Behav Res Ther. 2015 Nov 30;76:40–46. doi: 10.1016/j.brat.2015.11.008

Increasing Clinicians’ EBT Exploration and Preparation Behavior in Youth Mental Health Services by Changing Organizational Culture with ARC

Charles Glisson 1, Nathaniel J Williams 2, Anthony Hemmelgarn 3, Enola Proctor 4, Philip Green 5
PMCID: PMC4706472  NIHMSID: NIHMS742031  PMID: 26649464

Abstract

Objective

Clinician EBT exploration and preparation behavior is essential to the ongoing implementation of new EBTs. This study examined the effect of the ARC organizational intervention on clinician EBT exploration and preparation behavior and assessed the mediating role of organizational culture as a linking mechanism.

Method

Fourteen community mental health agencies that serve youth in a major Midwestern metropolis along with 475 clinicians who worked in those agencies, were randomly assigned to either the three-year ARC intervention or control condition. Organizational culture was assessed with the Organizational Social Context (OSC) measure at baseline and follow-up. EBT exploration and preparation behavior was measured as clinician participation in nine separate community EBT workshops held over a three-year period.

Results

There was 69 percent greater odds (OR = 1.69, p < .003) of clinicians in the ARC condition (versus control) attending each subsequent workshop. Workshop attendance in the control group remained under two percent and declined over three years while attendance in the ARC condition grew from 3.6 percent in the first workshop to 12 percent in the ninth and final workshop. Improvement in proficient organizational culture mediated the positive effect of the ARC intervention on clinicians’ workshop attendance (a×b = .21; 95% CI: LL = .05, UL = .40).

Conclusions

Organizational interventions that create proficient mental health agency cultures can increase clinician EBT exploration and preparation behavior that is essential to the ongoing implementation of new EBTs in community youth mental health settings.

Keywords: ARC, organizational intervention, OSC, EBT implementation, organizational culture

A number of evidence based treatments (EBT’s) are effective for the most common children’s mental health problems (Chorpita et al., 2011; Weisz & Gray, 2008; Weisz et al., 2013), but few of these EBTs are used in community mental health agencies that serve youth (Evans, Koch, Brady, Meszaros, & Sadler, 2013; Garland et al., 2010; Hoagwood, Kelleher, Feil, & Comer, 2000; Sheehan, Walrath, & Holden, 2007; Wonderlich et al, 2011; Zima et al., 2005). Addressing this discrepancy requires a better understanding of the use of EBTs as a phased process that begins with the service provider’s exploration and preparation to use EBTs and ends with EBT implementation and sustainment (Aarons, Hurlburt, & Horwitz, 2011; Chorpita & Regan, 2009). Research on strategies for increasing the use of EBTs has focused on the implementation phase with very little research focused on exploration and preparation (Novins et al, 2013). The lack of information about exploration and preparation represents a significant knowledge gap because these initial phases are essential to the ongoing adoption and implementation of new EBTs. Moreover, because new EBTs continue to be developed, the process of EBT exploration and preparation is never complete for a clinician who intends to use the most effective treatments.

Strategies that promote the use of a single EBT have been the focus of much of the implementation research to date (McHugh & Barlow, 2010). In the single EBT implementation approach, agencies select a specific EBT, require clinicians to use that specific EBT, and provide training (e.g., Lopez et al., 2011). Although agency support for the use of EBTs is important, there is a need for agency strategies that encourage individual clinicians to explore and prepare to use multiple new EBTs according to their own interests and skills and the unique needs of their particular clients. Clinician discretion in exploring and selecting EBTs is essential to the clinician’s commitment to fidelity and sustainment, as well as to ensure that the clinician will continue to seek out new EBTs as they become available (Cadwallader, Jarvis, Bitner, & Ostrom, 2010; Deci & Ryan, 1987; Wanberg & Banas, 2000). Similar to the adage of “giving an individual a fish feeds the person for a day but teaching an individual to fish feeds the person for a lifetime,” there is a need for strategies that encourage continuous and ongoing EBT exploration and preparation behavior among mental health clinicians.

We take an organizational view of the challenge of promoting and sustaining EBT exploration and preparation behavior at the individual clinician level. This view is based on the idea that organizations’ social contexts differ in the extent to which their norms and expectations (i.e., culture) encourage members to use new state of the art techniques and that such cultures are essential to EBT exploration and preparation among clinicians (Glisson & Williams, 2015; Jacobs et al., 2015; Rogers, 2003; Williams & Glisson, 2014b). Furthermore, we know from research on the diffusion of innovation that social context represents a key driver of individuals’ innovation behavior and argue that an organization’s social context is as important to its members’ use of innovations such as EBTs as the technical knowledge and skills associated with the innovation (Aarons & Sommerfeld, 2012; Klein, Conn, & Sorra, 2001, Rogers, 2003).

There is empirical evidence that the use of evidence-based practices in mental health services is associated with organizational social context (Aarons et al, 2009; Novins et al., 2013; Rohrbach Graham, & Hansen, 1993). We know, for example, that clinicians in mental health agencies with proficient organizational cultures have more positive views of EBTs, are more likely to adopt EBTs, and that new treatment programs are more likely to be sustained in those agencies (Aarons et al., 2012; Aarons, Sommerfeld, & Walrath-Greene, 2009; Baer et al., 2009; Glisson, Schoenwald et al., 2008). Mental health agencies with proficient organizational cultures expect clinicians to use state-of-the-art treatment techniques and place a high priority on improving the wellbeing of clients (Glisson, Landsverk et al., 2008). Clinicians who work in proficient cultures report that they are expected to be effective and there is evidence that mental health organizations with proficient cultures provide higher quality service and better outcomes (Glisson, Hemmelgarn, Green, & Williams, 2013; Olin et al., 2014; Williams & Glisson, 2013; Williams & Glisson, 2014a). These preliminary studies suggest that EBT exploration and preparation behavior among mental health clinicians can be increased with a planned organizational intervention that creates proficient organizational cultures.

The Availability, Responsiveness and Continuity (ARC) Organizational Intervention

The ARC organizational intervention has been shown in randomized trials to improve organizational cultures, climates, work attitudes and service outcomes in mental health agencies that serve youth (Glisson, Hemmelgarn et al., 2012; Glisson, Hemmelgarn et al., 2013; Glisson, Schoenwald et al., 2010). The ARC intervention strategies depend on trained specialists who work at all levels of an organization to: (a) embed guiding principles for improving services, (b) develop shared mental models among organizational members to support the improvement effort, and (c) enact organizational tools for identifying and addressing service barriers (Glisson, Dukes, & Green, 2006; Glisson & Schoenwald, 2005). The ARC specialists use The ARC Team Leader Guide and ARC Team Member Handbook to facilitate an agency’s use of the three strategies (i.e., guiding principles, shared mental models, and organizational tools) to improve services.

The ARC Guide and Handbook are used by line level ARC teams that are composed of clinicians and led by clinical supervisors who are trained by an ARC specialist in the three strategies using a phased, multistage approach. Guided by the ARC principles and mental models with a focus on improving service quality and outcomes, the ARC teams meet weekly to identify barriers to service improvement within their organization and develop recommendations for removing those barriers. The recommendations developed by the ARC teams are reviewed by an Organization Action Team (OAT) composed of members from each level of the organization from clinicians to middle managers to top executives. The OAT meets monthly to make organizational policy and procedural changes necessary for addressing the service barriers identified by the ARC teams. The principles, mental models and organizational tools used by the ARC teams and OAT are designed to develop the organizational norms and values associated with a proficient culture while providing the organizational capacity for clinician driven service improvement.

Although preliminary studies show that ARC improves mental health service quality and outcomes with and without the implementation of a specific EBT, there has been no effort to determine the effect of ARC on individual clinicians’ EBT exploration and preparation behavior. This is important because ongoing exploration and preparation behavior for using new EBTs in community mental health agencies requires considerable effort at the clinician level. First, clinicians must identify EBTs that they believe will be helpful with their particular clients and will complement their own personal history of training and experience. Second and most importantly, clinicians must enroll in and attend EBT training opportunities (e.g., EBT workshops) when those opportunities become available in their community. ARC is designed to develop organizational cultures within community mental health agencies that will encourage efforts by individual clinicians to identify and seek training in EBTs that will improve their proficiency. ARC materials and specialists provide as part of the ARC intervention regular reminders of the importance of evidence-based treatments and clinician training, including workshop opportunities, in promoting the norms and expectations associated with proficient organizational cultures. We therefore hypothesize that ARC’s success at developing proficient organizational cultures within community mental health agencies will encourage EBT exploration and preparation by the individual clinicians working in those agencies. Specifically, we assess whether agencies that participate in the ARC intervention create more proficient cultures and whether clinicians in those agencies are more likely to attend a series of different EBT training workshops offered in their community over an extended period of time.

The study examines clinicians’ attendance at nine community-based training workshops, each addressing a different EBT, over a three year period in one large Midwestern city. Each of the nine workshops was conducted by a nationally known expert or experts as part of a major university affiliated continuing education program. Following the randomization of agencies to ARC and control conditions, the study tests two hypotheses: (1) Clinicians in agencies that participate in the ARC organizational intervention are more likely to attend a series of different EBT training workshops that are offered to clinicians in the community; and (2) the effect of ARC on clinician workshop attendance over time is mediated by ARC’s success in creating proficient organizational cultures in the clinicians’ agencies during that same period.

Method

Participants

Community mental health agencies

A sample of 14 outpatient nonprofit mental health agencies that serve youth were identified in 2009 in a large, Midwestern metropolis. The agencies were selected to reflect characteristics of a national representative sample of mental health agencies that serve youth (e.g., Schoenwald et al., 2008) and each agency had one or more units that delivered mental health treatment to youth with 15 or more staff. Agencies were excluded from the study if they had initiated EBT adoption in the prior 12 months or if they were part of a federally-funded mental health services research network. Similar to national samples, the agencies delivered a variety of mental health services to youth (e.g., pharmacotherapy, individual psychotherapy, family therapy, skills-training, therapeutic groups) in a range of settings with a variety of theoretical orientations and approaches. The agencies were matched on size (number of staff) and randomly assigned to either the ARC intervention or control condition.

Clinicians

The study included 475 participating clinicians (n = 259 in ARC agencies, n = 216 in control agencies), representing 86% of the clinicians in the participating agencies’ programs that provided direct clinical services to youth and their families (M = 34 clinicians per agency, SD = 23.88, min = 8, max = 96). The majority of clinicians were female (82.1%, n = 390), white (82.5%, n = 392) and experienced with an average of 9.14 years (SD = 8.87) working in mental health settings. The most common educational backgrounds included master’s degrees (73.7%, n = 350) or bachelor’s degrees (19.2%, n = 91) in social work (38.3%, n = 182) or an allied health field (29.9%, n = 142). Clinicians in the ARC and control agencies did not differ on education, years of experience, gender, age, race, or turnover (all p’s > .05). Clinicians provided informed consent following protocols approved by the Institutional Review Boards of the University of Tennessee, Knoxville, and Washington University in St. Louis.

Procedures

Intervention condition

The ARC organizational intervention occurred over a 36-month period from 2010 to 2013. The procedures used to train the ARC specialists and the ARC manuals used to guide the intervention process were tested in three prior randomized controlled trials (Glisson et al., 2006; Glisson et al., 2010; Glisson et al., 2013). ARC specialists facilitated the intervention using separate ARC manuals for team leaders and team members, respectively, which were designed to guide teams through the completion of the ARC components, embed guiding principles, and develop shared mental models related to building an organizational work environment for identifying and addressing barriers to effective mental health services. ARC specialists have advanced graduate degrees and five or more years of experience working with human service organizations. Training for ARC specialists includes reading ARC research articles, ARC manuals and facilitation guides, and other selected research articles. ARC specialists also participate in didactic training sessions and weekly consultation and supervision with developers of the ARC intervention. Additional information about the ARC intervention is available in previously published articles, the manuals published by the University of Tennessee Children’s Mental Health Services Research Center, and at the Center website (cmhsrc.utk.edu).

EBT workshops

Nine EBT workshops were presented over a three year period through a major university continuing education program as part of its usual community continuing education program. The nine workshops were selected to include established EBTs for the most common childhood disorders including disruptive behavior disorders, anxiety disorders, depression, and trauma. Enrollment was offered to clinicians in the agencies in the two study conditions prior to each workshop before enrollment was opened to the entire community so that no clinicians who tried to register for any workshop in either condition were denied admission. No clinicians in either condition were required to attend the workshops by their agency and no information was provided to the participating agencies on who attended. Workshops were offered sequentially at intervals of two to six months apart with two workshops offered in the first year, four workshops during the second year, and three workshops during the third year.

Clinician surveys

Clinicians completed the Organizational Social Context (OSC) measure of organizational culture and climate during regularly scheduled staff meetings at each agency at baseline and following the completion of the intervention. Research assistants administered and collected the surveys directly from clinicians during staff meetings in which supervisors and managers were not present. Clinicians were assured of the confidentiality of their responses and informed that their responses would only be reported in an aggregate form that did not identify the agency or respondent. The surveys took approximately 20 minutes to complete. The clinician response rate averaged 86% across baseline and follow-up.

Measures

EBT Workshop Attendance

Clinicians’ attendance at EBT workshops was monitored based on information collected from attendees at each workshop by continuing education program staff. The continuing education staff members were blind to clinicians’ treatment condition and uninformed of the study hypotheses. At each EBT workshop, clinician attendance was recorded at check-in at the beginning of the workshop. Participating clinicians whose names and affiliations were on the check-in log at each workshop were identified as having attended that specific workshop. Participating clinicians who were working in the participating agencies at the time of the workshop but whose names were not on the attendance log were identified as not attending that specific workshop. Participating clinicians were coded as present or absent from a workshop only if they were employed in a participating agency at the time of the workshop.

Organizational social context

Each agency’s organizational culture and climate were assessed at baseline with the Organizational Social Context (OSC) measure (Glisson, Landsverk et al., 2008). Baseline measures were included in the analyses as OSC profile scores that incorporated six dimensions of organizational culture (proficiency, rigidity, and resistance) and climate (engagement, functionality, and stress). Clinicians’ responses to the six culture and climate scales were aggregated to the agency level (Chan, 1998) following confirmation of within-agency agreement of responses using the rwg(j) statistic (James, Demaree, & Wolf, 1993). Values of rwg(j) for all agencies and scale dimensions exceeded the recommended cutoff of .70 (M rwg(j) = .96, sd = .03, min = .80, max = .99) (LeBreton & Senter, 2008). The alpha reliabilities for the six OSC scales averaged .85 (sd = .09) and ranged between .70 and .94.

OSC profile scores represent a configured approach (Schulte, Ostroff, Shmulyian, & Kinicki, 2009) to characterizing organizational social context in reference to three latent sub-population culture and climate profiles derived empirically from a national sample of children’s mental health service agencies. The criterion validity and predictive validity of OSC profile scores were confirmed in two samples of children’s mental health agencies on three separate outcome criteria including clinicians’ work attitudes (Glisson et al., 2014), mental health service quality (Olin et al., 2014), and clinical outcomes for youth (Glisson et al., 2013). OSC profile scores range from 1.00 to 3.00 with higher scores representing more positive culture and climate profiles (i.e., associated with higher service quality, clinician job satisfaction and commitment, and improved youth outcomes). In the present study, agencies’ baseline OSC profile scores ranged from 1.00 to 3.00 with a mean of 1.70 (SD = .72). The agencies in the two conditions did not differ on their baseline OSC profile scores (ARC = 1.72, control = 1.68, t = .11, p< .912).

Proficient culture was assessed following the ARC intervention with the 15 item proficiency subscale of the OSC (Glisson, Green et al., 2012; Glisson, Landsverk et al., 2008). The OSC’s factor validity, including proficiency, was confirmed in two national studies of children’s mental health clinics and child welfare agencies and subsequent studies confirmed its predictive validity for clinician turnover (Glisson, Schoenwald et al., 2008), new program sustainability (Glisson, Schoenwald et al., 2008), service quality (Olin et al. 2014), and youth mental health services outcomes (Glisson et al., 2013; Glisson & Green, 2011; Williams & Glisson, 2013; Williams & Glisson, 2014a). Proficient organizational cultures are characterized by shared norms and expectations that clinicians place the well-being of each client first, are skilled service providers, and have up-to-date knowledge of effective practices. Items are completed by clinicians using a 5-point rating scale ranging from 1 (never) to 5 (always). Alpha reliability for the proficient culture scale was .89. Aggregate agency-level proficiency scores were converted to T-scores with a μ = 50 and σ = 10 using national norms from a representative sample of children’s mental health agencies (Glisson, Landsverk et al., 2008).

Data Analysis

Hypothesis 1 was tested using a fully random, 3-level hierarchical linear model (HLM) with a Bernoulli distribution and a logit link function for a dichotomous outcome (clinician attendance at each of nine workshops) and the nesting of repeated observations within clinicians and clinicians within agencies (Hedeker & Gibbons, 2006; Raudenbush & Bryk, 2002). Using HLM 6 software with maximum likelihood estimation, the log of the clinicians’ odds of workshop attendance was modeled in a linear growth model as a function of workshop (i.e., nine workshops were offered sequentially over three years) at level 1, clinician at level 2, and agency at level 3. The workshop variable was represented by nine ordinal values beginning with the first workshop and ending with the ninth workshop. Level 3 included predictors representing random assignment to the ARC versus control conditions and agencies’ baseline OSC.

The HLM model for Hypothesis 1 estimates the effect of ARC on the trend in the log of the clinicians’ odds of attending the nine workshops over three years adjusted for baseline OSC, as well as clinician-level and agency-level clustering. Preliminary analyses confirmed that a large and statistically significant proportion of the variance in clinicians’ growth in odds of workshop attendance was at the agency level (90%, χ2 = 27.01, p < .01) and that clinicians in the ARC and control agencies did not differ on the odds of attending the first workshop (ARC = .036, control = .011, t = 1.40, p < .187).

Hypothesis 2 was tested using the product of coefficients approach for multilevel mediation analysis described by Krull and MacKinnon (2001), Zhang, Zyphur, and Preacher (2009), and Pituch, Murphy, & Tate (2010). The mediated effect is estimated as the product of coefficients from simultaneous equations representing the effect (labeled a) of ARC on the agency’s proficient culture and the effect (labeled b) of proficient culture on clinicians’ growth in probability of workshop attendance controlling for ARC. The product (a × b) estimates the cross-level mediated effect of ARC on clinicians’ growth in probability of workshop attendance due to proficient culture. A significant mediation effect coupled with a non-significant direct effect of ARC (labeled c′) in the second equation (that includes follow-up proficient culture) support mediation. Such a result suggests ARC’s effect on clinicians’ trends in their probability of workshop attendance is explained by ARC’s effect on proficient culture.

We employed two complementary approaches to test the statistical significance of the mediation effect (Biesanz et al., 2010; Hayes & Scharkow, 2013; MacKinnon et al., 2004). First, the joint significance test examined the joint null hypothesis for the two coefficients that comprise the mediation effect, H0: a = 0 and b = 0 (Cohen & Cohen, 1983, p. 366). A significant mediation effect is supported if the null hypothesis for both coefficients is rejected (MacKinnon et al., 2002). Second, asymmetric 95% confidence limits were constructed around the point estimate of the mediated effect using computationally intensive Monte Carlo methods (MacKinnon, Lockwood, & Williams, 2004; Preacher & Selig, 2012). Confidence limits based on Monte Carlo methods are more accurate, have more reliable Type I error rates, and exhibit higher statistical power than normal theory approaches (e.g., the Sobel test) because the sampling distributions of indirect effects rarely conform to normality (MacKinnon et al., 2004). The online R utility developed by Selig and Preacher (2008) was used to construct the Monte Carlo CIs for the indirect effects in this study.

Results

Table 1 presents results from the HLM analysis of Model 1 examining growth in the clinicians’ logged odds of workshop attendance in ARC versus control agencies. Consistent with Hypothesis 1, clinicians in ARC agencies exhibited significantly steeper increases in their logged odds of workshop attendance (ARC = .525, p < .003), while clinicians in the control agencies exhibited decreases in their logged odds of attendance (control = −.347, p < .033). The corresponding odds ratio for the ARC effect (OR = 1.69, p < .003) indicates there was 69 percent greater odds of clinicians in the ARC condition than in the control attending each subsequent EBT workshop. As illustrated in Figure 1, ARC increased clinicians’ EBT exploration and preparation behavior over time as measured by their probability of attendance at each of the nine EBT workshops offered over the three year period. The attendance gap between the two conditions increased sharply through the last workshop which was attended by twelve percent of the clinicians in the ARC condition and less than one percent of the clinicians in the control condition.

Table 1.

Three-level logistic hierarchical linear model analyses of clinicians’ EBT workshop attendance

Model 1 Model 2
Parameter Coeff. SE Coeff. SE
Fixed Effects
 Intercept −3.644*** .399 −4.058*** .510
 Slope (control) −.347* .142 −.174 .152
 Slope x baseline OSC −.051 .083 −.166 .093
 Slope x ARC .525** .136 .305 .152
 Slope x proficient culture .022* .008
Variance Components
 Clinician intercepts .823 .815
 Clinician slopes .004 .004
 Agency intercepts .603** 1.527**
 Agency slopes .015** .005
Open in a new tab
*

p < .05,

**

p < .01,

***

p < .001

Note: OSC = Organizational Social Context; ARC = Availability, Responsiveness, and Continuity; EBT = evidence-based practice.

Figure 1.

Figure 1

Open in a new tab

ARC effect on the trend in the probability of attending nine different EBT workshops

Hypothesis 2 stated that an improvement in proficient culture would mediate ARC’s effects on clinicians’ EBT workshop attendance. This hypothesis requires ARC to increase proficient organizational culture. Results from an ordinary least squares regression analysis indicated that ARC agencies exhibited greater improvement in proficient culture at follow-up relative to control agencies (B = 9.34, SE = 2.24, p = .002, ΔR2 = .39) after controlling for baseline organizational social context (ARC also contributed to a higher level of proficient culture without the baseline OSC included as a covariate). Hypothesis 2 also requires that the improvement in proficient culture predict an increase in the clinicians’ logged odds of workshop attendance after controlling for the effect of ARC. Results from the HLM analysis testing this effect (Model 2) indicated that improvements in proficient culture predicted significantly steeper increases in the log of the clinicians’ odds of attending an EBT workshop during the three year period (γ = .022, SE = .008, p = .027). Based on these analyses, the cross-level indirect effect of ARC on clinicians’ EBT attendance was statistically significant (a×b = .21; 95% CI: LL = .05, UL = .40). Moreover, the direct effect of ARC was not significant in the second model (γ = .305, SE = .152, p = .07), indicating that ARC’s effect on clinicians’ EBT exploration and preparation behavior was mediated by improvements in proficient organizational culture.

Discussion

Previous research on strategies for increasing the use of EBTs in youth mental health services has focused in large part on the implementation phase while very little research has focused on the initial phases of EBT exploration and preparation. This is an important knowledge gap because exploration and preparation by individual clinicians are essential to ensuring that newly developed EBTs will be adopted as they become available. Also, individual clinician exploration and preparation behavior engenders the level of commitment to the use of EBTs that is essential to successful implementation and sustainment. The current study addresses this knowledge gap with evidence from a randomized trial showing that clinician exploration and preparation behavior, represented by EBT workshop attendance, can be improved with an organizational intervention that creates proficient organizational cultures. The finding that the ARC organizational intervention increases self-selected clinician attendance at community EBT training workshops and that the effect is mediated by ARC’s success in creating a proficient organizational culture, underscores the role played by organizational norms and expectations in EBT exploration and preparation behavior.

We need a better understanding of strategies to increase EBT exploration and preparation behavior among clinicians who provide mental health treatment to youth because the behavior is essential to improving and sustaining the effectiveness of community mental health services. The present results show that proficient organizational cultures can be developed in mental health agencies that serve youth and that the success in building a proficient culture increases and sustains clinician exploration and preparation behavior for using EBTs over a three year period. It is important that the differences in the probability of clinician EBT workshop attendance between the ARC and control condition became steadily larger in each successive workshop over three years, showing that attendance increased in the series of workshops as the proficient organizational cultures became more pronounced and participation in clinician-selected EBT training became the norm.

Although the odds of attending each subsequent EBT workshop increased significantly over time in the ARC condition, the proportion of clinicians attending each individual workshop is not high. Twelve percent of the clinicians in the ARC condition attended the ninth and final workshop in comparison to less than one percent of the clinicians in the control condition. The proportion may seem particularly low in comparison to that sought by agency mandated training in which one EBT is selected by the agency and all clinicians in the agency are required to attend. However, we know that agency mandated training has had limited success in EBT implementation and sustainment. Moreover, it does not encourage individual clinicians to explore new EBTs that address their own unique needs as new training becomes available. The approach taken here of creating an organizational culture that encourages clinicians to individually and independently self-select the EBT workshops that meet their unique needs reflects the role of social context described in innovation diffusion studies (Rogers, 2003). We know from these studies that innovations diffuse slowly in the initial stage of the well-known S curve with the trend becoming increasingly steeper over time (which parallels the results reported here). These studies also show that innovations are more likely to be implemented successfully and sustained when the innovations are selected independently by the individual and supported by their social context, both of which are characteristics of the approach taken here.

Preliminary studies show that organizational social contexts can be improved with the ARC intervention and that improved social contexts are associated with lower staff turnover, more positive work attitudes, higher service quality and better service outcomes (Glisson, Dukes & Green, 2006; Glisson & Green, 2011; Glisson, Hemmelgarn et al., 2012; Glisson, Hemmelgarn et al., 2013; Glisson, Schoenwald et al., 2008; Glisson, Schoenwald et al., 2010; Olin et al., 2014; Williams & Glisson, 2013; Williams & Glisson, 2014a). The present findings provide evidence that proficient organizational cultures can be created with the ARC intervention to increase and sustain self-selected clinician attendance at a series of EBT workshops offered in the community over an extended period of time. This is the first randomized trial to confirm that such clinician EBT exploration and preparation behavior at the individual level can be increased with an organizational intervention, adding to preliminary findings that organizational interventions and associated improvements in organizational social context are essential to the ongoing use of EBTs in community-based youth mental health services.

Caveats

Two caveats must be considered in interpreting these results. First, agencies were randomly assigned to ARC and control conditions after matching on size so that clinicians were cluster randomized by the agencies that employed them. Cluster randomization is well accepted, but unmeasured clinician variables could have influenced attendance. This concern is reduced by the finding that there were no baseline differences between the two conditions in clinician education, years of experience, gender, age, race, or probability of turnover. Moreover, there was no significant difference in attendance between ARC and control conditions at the first workshop, but the difference grew steadily and significantly larger over the three year period. In addition, the effect of ARC on the increase in workshop attendance over time was mediated by the increase in proficient culture during the same period. Collectively, these findings suggest that the threat of unmeasured variables confounding the results is minimized.

A second caveat is that EBT workshop attendance does not fully capture all exploration and preparation behaviors, including unmeasured attendance at unknown workshops, although random assignment is expected to control the effects of unmeasured variables across the two conditions. Moreover, the act of attending a workshop is perhaps a necessary but not necessarily sufficient aspect of a range of behaviors that represent a clinician’s preparation to use an EBT. At the same time, self-selected workshop attendance does represent EBT exploration and indicates a clinician’s personal investment in preparing to use a new EBT. EBT workshop attendance does not guarantee EBT use, but there are few behaviors that are better indicators of a clinician’s intent to use an EBT than the clinician attending a particular EBT workshop that was selected by the clinician. This and the finding that clinician EBT workshop attendance continued to increase over the three year period among clinicians in the agencies assigned to the ARC intervention, while it decreased in the control condition, suggests a sustained effect on exploration and preparation behavior across a variety of youth EBTs.

  • EBT exploration and preparation behavior includes clinician participation in EBT workshops that they select based on their own interests and training needs

  • A randomized study of 14 organizations found that the ARC organizational intervention increased clinician participation in nine community EBT workshops over three years

  • There was 69 percent greater odds of clinicians in the ARC condition versus control attending each subsequent EBT workshop.

  • Improvements in proficient organizational culture mediated the positive effect of the ARC intervention on clinicians’ workshop attendance

Acknowledgments

This research was supported by NIH R01-MH084855 (PI:CG) and F31-MH099846 (PI:NJW).

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Charles Glisson, University of Tennessee

Nathaniel J. Williams, Boise State University

Anthony Hemmelgarn, University of Tennessee

Enola Proctor, Washington University St. Louis

Philip Green, University of Tennessee

References

  1. Aarons GA, Fettes DL, Flores LE, Sommerfeld DH. Evidence-based practice implementation and staff emotional exhaustion in children’s services. Behaviour Research and Therapy. 2009;47(11):954–960. doi: 10.1016/j.brat.2009.07.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Aarons GA, Glisson C, Green PD, Hoagwood K, Kelleher KJ, Landsverk JA The Research Network on Youth Mental Health. The organizational social context of mental health services and clinician attitudes toward evidence-based practice: A United States national study. Implementation Science. 2012;7:56. doi: 10.1186/1748-5908-7-56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public sectors. Administration and Policy in Mental Health. 2011;38:4–23. doi: 10.1007/s10488-010-0327-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Aarons GA, Sommerfeld DH. Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. Journal of the American Academy of Child and Adolescent Psychiatry. 2012;51:423–431. doi: 10.1016/j.jaac.2012.01.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Aarons GA, Sommerfeld DH, Walrath-Greene CM. Evidence-based practice implementation: The impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implementation Science. 2009;4(83) doi: 10.1186/1748-5908-4-83. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Baer JS, Wells EA, Rosengren DB, Hartzler B, Beadnell B, Dunn C. Agency context and tailored training in technology transfer: A pilot evaluation of motivational interviewing training for community counselors. Journal of Substance Abuse Treatment. 2009;37:191–202. doi: 10.1016/j.jsat.2009.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Biesanz JC, Falk CF, Savalei V. Assessing mediational models: Testing and interval estimation for indirect effects. Multivariate Behavioral Research. 2010;45:661–701. doi: 10.1080/00273171.2010.498292. [DOI] [PubMed] [Google Scholar]
  8. Cadwallader S, Jarvis CB, Bitner MJ, Ostrom AL. Frontline employee motivation to participate in service innovation implementation. Journal of the Academy of Marketing Science. 2010;38:219–239. [Google Scholar]
  9. Chan D. Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology. 1998;83:234–246. [Google Scholar]
  10. Chorpita BF, Daleiden EL, Ebesutani C, Young J, Becker KD, Nakamura BJ, Phillips L, Starace N. Evidence-based treatments for children and adolescents: An updated review of indicators of efficacy and effectiveness. Clinical Psychology: Science & Practice. 2011;18:154–172. [Google Scholar]
  11. Chorpita BF, Regan J. Dissemination of effective mental health treatment procedures: Maximizing the return on a significant investment. Behaviour Research and Therapy. 2009;47(11):990–993. doi: 10.1016/j.brat.2009.07.002. [DOI] [PubMed] [Google Scholar]
  12. Cohen J, Cohen P. Applied multiple regression/ correlation analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum; 1983. [Google Scholar]
  13. Deci EL, Ryan RM. The support of autonomy and control of behavior. Journal of Personality and Social Psychology. 1987;53:1024–1037. doi: 10.1037//0022-3514.53.6.1024. [DOI] [PubMed] [Google Scholar]
  14. Evans SW, Koch JR, Brady C, Meszaros P, Sadler J. Community and school mental health professionals’ knowledge and use of evidence based substance use prevention programs. Administration and Policy in Mental Health. 2013;40:319–330. doi: 10.1007/s10488-012-0422-z. [DOI] [PubMed] [Google Scholar]
  15. Garland AF, Brookman-Frazee L, Hurlburt MS, Accurso EC, Zoffness R, Haine-Schlagel RA, et al. Mental health care for children with disruptive behavior problems: A view inside therapists’ offices. Psychiatric Services. 2010;61:788–795. doi: 10.1176/appi.ps.61.8.788. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Glisson C, Dukes D, Green P. The effects of the ARC organizational intervention on caseworker turnover, climate, and culture in children’s service systems. Child Abuse and Neglect. 2006;30:855–880. doi: 10.1016/j.chiabu.2005.12.010. [DOI] [PubMed] [Google Scholar]
  17. Glisson C, Hemmelgarn A, Green P, Dukes D, Atkinson S, Williams NJ. Randomized trial of the availability, responsiveness, and continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth. Journal of the American Academy of Child and Adolescent Psychiatry. 2012;51:780–787. doi: 10.1016/j.jaac.2012.05.010. [DOI] [PubMed] [Google Scholar]
  18. Glisson C, Hemmelgarn A, Green P, Williams NJ. Randomized trial of the availability, responsiveness and continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. Journal of the American Academy of Child and Adolescent Psychiatry. 2013;52:493–500. doi: 10.1016/j.jaac.2013.02.005. [DOI] [PubMed] [Google Scholar]
  19. Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, Green P The Research Network on Youth Mental Health. Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health. 2008;35:98–113. doi: 10.1007/s10488-007-0148-5. [DOI] [PubMed] [Google Scholar]
  20. Glisson C, Schoenwald SK. The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research. 2005;7:243–259. doi: 10.1007/s11020-005-7456-1. [DOI] [PubMed] [Google Scholar]
  21. Glisson C, Schoenwald SK, Hemmelgarn A, Green P, Dukes D, Armstrong KS, Chapman JE. Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology. 2010;78:537–550. doi: 10.1037/a0019160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Glisson C, Schoenwald SK, Kelleher K, Landsverk J, Hoagwood KE, Mayberg S, Green P The Research Network on Youth Mental Health. Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Administration and Policy in Mental Health. 2008;35:124–133. doi: 10.1007/s10488-007-0152-9. [DOI] [PubMed] [Google Scholar]
  23. Glisson C, Williams NJ. Assessing and changing organizational social contexts for effective mental health services. Annual Review of Public Health. 2015;36:507–523. doi: 10.1146/annurev-publhealth-031914-122435. [DOI] [PubMed] [Google Scholar]
  24. Glisson C, Williams NJ, Green P, Hemmelgarn A, Hoagwood K. The organizational social context of mental health Medicaid waiver programs with family support services: Implications for research and practice. Administration and Policy in Mental Health. 2014;41:32–42. doi: 10.1007/s10488-013-0517-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Hayes AF, Scharkow M. The relative trustworthiness of inferential tests of the indirect effect in statistical mediation analysis: Does method really matter? Psychological Science. 2013;24:1918–1927. doi: 10.1177/0956797613480187. [DOI] [PubMed] [Google Scholar]
  26. Hedeker D, Gibbons RD. Longitudinal data analysis. Hoboken, NJ: John Wiley & Sons; 2006. [Google Scholar]
  27. Hoagwood K, Kelleher KJ, Feil M, Comer DM. Treatment services for children with ADHD: A national perspective. Journal of the American Academy of Child and Adolescent Psychiatry. 2000;39:198–206. doi: 10.1097/00004583-200002000-00020. [DOI] [PubMed] [Google Scholar]
  28. Jacobs SR, Weiner BJ, Reeve BB, Hofmann DA, Christian M, Weinberger M. Determining the predictors of innovation implementation in healthcare: A quantitative analysis of implementation effectiveness. BMC Health Services Research. 2015;15:1–13. doi: 10.1186/s12913-014-0657-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. James LR, Demaree RG, Wolf G. Rwg: An assessment of within-group agreement. Journal of Applied Psychology. 1993;78:306–309. [Google Scholar]
  30. Klein KJ, Conn AB, Sorra JS. Implementing computerized technology: An organizational analysis. Journal of Applied Psychology. 2001;86:811–824. doi: 10.1037/0021-9010.86.5.811. [DOI] [PubMed] [Google Scholar]
  31. Krull JL, MacKinnon DP. Multilevel modeling of individual and group level mediated effects. Multivariate Behavioral Research. 2001;36:249–277. doi: 10.1207/S15327906MBR3602_06. [DOI] [PubMed] [Google Scholar]
  32. Landsverk J, Brown CH, Reutz JR, Palinkas L, Horwitz SM. Design elements in implementation research: A structural review of child welfare and child mental health studies. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38:54–63. doi: 10.1007/s10488-010-0315-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. LeBreton JM, Senter JL. Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods. 2008;11:815–852. [Google Scholar]
  34. Lopez MA, Osterberg LD, Jensen-Doss A, Rae WA. Effects of workshop training for providers under mandated use of an evidence-based practice. Administration and Policy in Mental Health. 2011;38:301–312. doi: 10.1007/s10488-010-0326-8. [DOI] [PubMed] [Google Scholar]
  35. MacKinnon DP. Introduction to statistical mediation analysis. New York: Lawrence Erlbaum Associates; 2008. [Google Scholar]
  36. MacKinnon DP, Lockwood CM, Hoffman JM, West SG, Sheets V. A comparison of methods to test mediation and other intervening variable effects. Psychological Methods. 2002;7:83–104. doi: 10.1037/1082-989x.7.1.83. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. MacKinnon DP, Lockwood CM, Williams J. Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research. 2004;39:99–128. doi: 10.1207/s15327906mbr3901_4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. McHugh RK, Barlow DH. The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist. 2010;65:73–84. doi: 10.1037/a0018121. [DOI] [PubMed] [Google Scholar]
  39. Novins DK, Green AE, Legha RK, Aarons GA. Dissemination and implementation of evidence-based practices for child and adolescent mental health: A systematic review. Journal of the American Academy of Child and Adolescent Psychiatry. 2013;52:1009–1025. doi: 10.1016/j.jaac.2013.07.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Olin SS, Williams N, Pollock M, Armusewicz K, Kutash K, Glisson C, Hoagwood KE. Quality indicators for family support services and their relationship to organizational social context. Administration and Policy in Mental Health and Mental Health Services Research. 2014;41:43–54. doi: 10.1007/s10488-013-0499-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Pituch KA, Murphy DL, Tate RL. Three-level models for indirect effects in school- and class-randomized experiments in education. Journal of Experimental Education. 2010;78:60–95. [Google Scholar]
  42. Preacher KJ, Hayes AF. Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods. 2004;40:879–891. doi: 10.3758/brm.40.3.879. [DOI] [PubMed] [Google Scholar]
  43. Preacher KJ, Selig JP. Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures. 2012;6:77–98. [Google Scholar]
  44. Raudenbush SW, Bryk AS. Hierarchical linear models: Applications and data analysis. 2. Thousand Oaks, CA: Sage publications; 2002. [Google Scholar]
  45. Rogers EM. Diffusion of innovations. 5. New York: Free Press; 2003. [Google Scholar]
  46. Rohrbach LA, Graham JW, Hansen WB. Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine. 1993;22:237–260. doi: 10.1006/pmed.1993.1020. [DOI] [PubMed] [Google Scholar]
  47. Schoenwald SK, Chapman JE, Kelleher K, Hoagwood KE, Landsverk J, Stevens J, Glisson C, Rolls-Reutz J The Research Network on Youth Mental Health. A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs) Administration and Policy in Mental Health. 2008;35:84–97. doi: 10.1007/s10488-007-0147-6. [DOI] [PubMed] [Google Scholar]
  48. Schulte M, Ostroff C, Shmulyian S, Kinicki A. Organizational climate configurations: Relationships to collective attitudes, customer satisfaction, and financial performance. Journal of Applied Psychology. 2009;94:618–634. doi: 10.1037/a0014365. [DOI] [PubMed] [Google Scholar]
  49. Selig JP, Preacher KJ. Monte Carlo method for assessing mediation: An interactive tool for creating confidence intervals for indirect effects [Computer software] 2008 Jun; Available from http://quantpsy.org/
  50. Sheehan AK, Walrath CM, Holden EW. Evidence-based practice use, training and implementation in community-based service settings: A survey of children’s mental health providers. Journal of Child and Family Studies. 2007;16:169–182. [Google Scholar]
  51. Tyson EH, Glisson C. A cross-ethnic validity study of the short-form assessment for children (SAC) Research on Social Work Practice. 2005;15:97–109. [Google Scholar]
  52. Vermunt JK, Magidson J. Latent class cluster analysis. In: Hagenaars JA, McCutcheon AL, editors. Applied latent class analysis. Cambridge: Cambridge University Press; 2002. pp. 89–106. [Google Scholar]
  53. Wanberg CR, Banas JT. Predictors and outcomes of openness to changes in a reorganizing workplace. Journal of Applied Psychology. 2000;85:132–142. doi: 10.1037/0021-9010.85.1.132. [DOI] [PubMed] [Google Scholar]
  54. Weisz JR, Gray JS. Evidence-based psychotherapy for children and adolescents: Data from the present and a model for the future. Child and Adolescent Mental health. 2008;13:54–65. doi: 10.1111/j.1475-3588.2007.00475.x. [DOI] [PubMed] [Google Scholar]
  55. Weisz JR, Kuppens S, Eckshtain D, Ugueto AM, Hawley KM, Jensen-Doss A. Performance of evidence-based youth psychotherapies compared with usual clinical care: A multilevel meta-analysis. JAMA Psychiatry. 2013;70:750–761. doi: 10.1001/jamapsychiatry.2013.1176. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Williams NJ, Glisson C. Reducing turnover is not enough: The need for proficient organizational cultures to support positive youth outcomes in child welfare. Children and Youth Services Review. 2013;35:1871–1877. doi: 10.1016/j.childyouth.2013.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Williams NJ, Glisson C. Testing a theory of organizational culture, climate, and youth outcomes in child welfare systems: A United States national study. Child Abuse & Neglect. 2014a;38:757–767. doi: 10.1016/j.chiabu.2013.09.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Williams NJ, Glisson C. The role of organizational culture and climate in the dissemination and implementation of empirically-supported treatments for youth. In: Beidas RS, Kendall PC, editors. Dissemination and Implementation of Evidence Based Practices in Child and Adolescent Mental Health. New York: Oxford University Press; 2014b. pp. 61–81. [Google Scholar]
  59. Wonderlich SA, Simonich HK, Myers TC, LaMontagne W, Hoesle J, Erickson AL, Korbel M, Crosby RD. Evidence-based mental health interventions for traumatized youth: A statewide dissemination project. Behaviour Research and Therapy. 2011;49(10):579–587. doi: 10.1016/j.brat.2011.07.003. [DOI] [PubMed] [Google Scholar]
  60. Zhang Z, Zyphur MJ, Preacher KJ. Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods. 2009;12:695–719. [Google Scholar]
  61. Zima BT, Hurlburt MS, Knapp P, Ladd H, Tang L, Duan N, et al. Quality of publicly-funded outpatient specialty mental health care for common childhood psychiatric disorders in California. Journal of the American Academy of Child and Adolescent Psychiatry. 2005;44:130–144. doi: 10.1097/00004583-200502000-00005. [DOI] [PubMed] [Google Scholar]

RESOURCES