Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Mar 1.
Published in final edited form as: Adm Policy Ment Health. 2017 Mar;44(2):269–283. doi: 10.1007/s10488-016-0742-5

Mechanisms of change in the ARC organizational strategy: Increasing mental health clinicians’ EBP adoption through improved organizational culture and capacity

Nathaniel J Williams 1, Charles Glisson 2, Anthony Hemmelgarn 2, Philip Green 2
PMCID: PMC5125909  NIHMSID: NIHMS792680  PMID: 27236457

Abstract

Background

The development of efficient and scalable implementation strategies in mental health is restricted by poor understanding of the change mechanisms that increase clinicians’ evidence-based practice (EBP) adoption. This study tests the cross-level change mechanisms that link an empirically-supported organizational strategy for supporting implementation (labeled ARC for Availability, Responsiveness, and Continuity) to mental health clinicians’ EBP adoption and use.

Method

Four hundred seventy five mental health clinicians in 14 children’s mental health agencies were randomly assigned to the ARC intervention or a control condition. Measures of organizational culture, clinicians’ intentions to adopt EBPs, and job-related EBP barriers were collected before, during, and upon completion of the three-year ARC intervention. EBP adoption and use were assessed at 12-month follow-up. Multilevel mediation analyses tested changes in organizational culture, clinicians’ intentions to adopt EBPs, and job-related EBP barriers as linking mechanisms explaining the effects of ARC on clinicians’ EBP adoption and use.

Results

ARC increased clinicians’ EBP adoption (OR = 3.19, p = .003) and use (81% vs. 56%, d = .79, p = .003) at 12-month follow-up. These effects were mediated by improvement in organizational proficiency culture leading to increased clinician intentions to adopt EBPs and by reduced job-related EBP barriers. A combined mediation analysis indicated the organizational culture-EBP intentions mechanism was the primary carrier of ARC’s effects on clinicians’ EBP adoption and use.

Conclusions

ARC increases clinicians’ EBP adoption and use by creating proficient organizational cultures that increase clinicians’ intentions to adopt EBPs.

Keywords: mechanism, mediation, organizational implementation strategy, organizational culture, intentions, mental health, evidence-based practice, ARC

Significant deficits remain in the development of implementation strategies that increase the use of evidence-based practices (EBPs) in routine mental health care (Collins et al., 2011; Insel, 2009; NIMH, 2008). Nearly half of all implementation strategies tested in mental health services fail to influence the targeted implementation, services, or clinical outcomes (Powell, Proctor, & Glass, 2014) and the most promising strategies are the least economically feasible (Novins et al., 2013). We argue that developing more effective and efficient implementation strategies requires a better understanding of the mechanisms that explain how these strategies influence implementation outcomes (Williams & Glisson, 2014b). However, almost no studies test the mechanisms that link implementation strategies to outcomes and no studies have supported a hypothesized linking mechanism in mental health settings (Williams, 2015).

The present study tests the mechanisms that link an empirically-supported organizational strategy for supporting implementation (labeled ARC for Availability, Responsiveness, and Continuity) to mental health clinicians’ EBP adoption and use in a four-year randomized controlled trial. ARC is an organizational intervention that facilitates the adoption and integration of empirically-supported clinical interventions (i.e., EBPs) and other innovations into routine practice by changing organizational cultures and reducing job-related service barriers (Glisson, Schoenwald et al., 2010; Powell et al., 2012). ARC has demonstrated positive main effects on clinician turnover, organizational culture, clinicians’ work attitudes, and clinical outcomes for youth (Glisson, Dukes, & Green, 2006; Glisson, Hemmelgarn, Green, & Williams, 2013; Glisson, Schoenwald et al., 2010). However, no studies have tested ARC’s effects on clinicians’ EBP adoption or the cross-level change mechanisms that explain how ARC influences clinicians’ practice behaviors. In the present RCT, we examine the cross-level contextual, motivational, and barrier-related mechanisms that explain ARC’s effects on clinicians’ EBP adoption and use. Our outcome criteria distinguish between clinicians including specific EBPs in their treatment repertoires (i.e., adoption) and the extent to which they incorporate EBPs in their work with clients (i.e., use).

Mechanisms of Change in ARC

The ARC organizational intervention is designed to improve an organization’s effectiveness and support clinicians’ EBP adoption by addressing two theoretically- and empirically-supported predictors of clinicians’ practice behaviors: (a) organizational culture, and (b) organizational capacity to reduce job-related service barriers (Williams & Glisson, 2014b). According to organizational culture theory, organizations that deliver mental health services are characterized by norms and behavioral expectations (i.e., cultures) that explain organizational differences in service quality and outcomes (Cooke & Rousseau, 1988; Hartnell, Ou, & Kinicki, 2011; Hofstede, Neuijen, Ohayv, & Sanders, 1990; Olin et al., 2014). The norms and expectations that characterize an organization’s culture influence clinicians’ practice behaviors by providing cues regarding the behaviors that are most valued and likely to be rewarded and by providing social and material contingencies that direct clinicians’ work priorities (Schein, 2004; Trice & Beyer, 1993).

According to behavioral science theories such as the Theory of Planned Behavior (Ajzen, 1991), culture-based social processes create shared outcome expectancies, social norms, and beliefs that influence individuals’ intentions to engage in select behaviors (Jaccard, Litardo, & Wan, 1999). Intentions represent self-instructions to perform a behavior or to obtain a specific outcome and describe a person’s motivation to perform a targeted behavior (Sheeran, 2002). The Theory of Planned Behavior construes intentions as the primary cross-level mechanism through which culturally-defined norms and expectations in a work environment influence individual practice behaviors (Figure 1). The ARC strategy builds on this logic to argue that changes in an organization’s culture lead to changes in clinicians’ intentions to engage in innovative behavior such as the adoption and use of EBPs (Williams & Glisson, 2014b).

Figure 1.

Figure 1

Open in a new tab

Hypothesized cross-level mechanisms of the ARC organizational implementation strategy

A large body of correlational research, summarized by a recent meta-analysis of 84 studies (Hartnell et al., 2011) supports the association between organizational culture and innovation behavior in the workplace. Hartnell and colleagues (2011) documented a population average correlation of ρ = .59 between organizational culture and work-related innovation behavior across a range of business, industrial, and healthcare settings. In mental health settings, several studies have demonstrated significant variability in mental health agencies’ cultural norms and expectations (Glisson, Landsverk et al., 2008; Glisson, Green, & Williams, 2012) and strong associations between organizational culture and clinicians’ practice attitudes, behaviors, and clinical outcomes (Aarons et al., 2012; Glisson, Schoenwald et al., 2008; Olin et al., 2014; Williams & Glisson, 2013; Williams & Glisson, 2014a).

Building on this research, the ARC strategy is designed to create proficient organizational cultures as a primary mechanism for supporting mental health service innovations such as the adoption and use of EBPs. Clinicians who work in agencies characterized by proficient cultures report they are expected to prioritize improvement in client well-being and to be competent in the use of up-to-date treatment practices such as EBPs (Glisson, Landverk et al., 2008). Proficient organizational cultures have been linked to more positive clinician attitudes toward EBPs, higher quality mental health services, and superior client outcomes in mental health and social service settings (Aarons et al., 2012; Olin et al., 2014; Glisson et al., 2013; Williams & Glisson, 2013; Williams & Glisson, 2014a). In the present study, organizational norms and expectations that emphasize proficiency are expected to increase clinicians’ intentions to adopt EBPs and to explain ARC’s effects on clinicians’ EBP intentions, adoption, and use. The study provides an initial test of ARC’s cross-level effects on clinicians’ EBP adoption and use through this hypothesized contextual-motivational mechanism (Figure 1).

A second mechanism targeted by ARC involves the development of an organization’s capacity to identify and reduce job-related service barriers. Numerous studies of EBP adoption in mental health settings indicate job-related barriers are among the most widely cited impediments to clinicians’ EBP adoption even when EBP use is mandated by governmental authorities and clinicians have ready access to EBP training and consultation (Aarons et al., 2009; Baer et al., 2009;; Gioia & Dziadosz, 2008; Jensen-Doss et al., 2009;). Examples of job-related barriers to EBP adoption and use include: an agency’s refusal to modify policies or workflow to accommodate EBPs, failure to provide materials necessary for EBPs, and failure to provide clinical supervision that supports EBPs (Bartholomew, Joe, Rowan-Szal, & Simpson, 2007; Raghavan, Inkelas, Franke, & Halfon, 2007). These data are consistent with implementation theories such as the Capacity, Opportunity, and Motivation (COM-B) model (Michie, van Stralen, & West, 2011) which predict that job-related barriers limit individuals’ enactment of EBP behaviors. The ARC strategy addresses this hypothesized determinant through a structured process that develops the organizational infrastructure and capacity to identify and remove job-related barriers to service innovation and effectiveness. The present study provides an initial test of this second ARC mechanism which is referred to as a barrier reduction mechanism.

The change mechanisms described above are depicted in the conceptual model shown in Figure 1. As shown in the model, ARC is expected to increase clinicians’ EBP adoption and use through two experimentally-manipulated mechanisms: (a) a contextually-induced increase in clinicians’ EBP intentions (i.e., contextual-motivational mechanism), and (b) the reduction of job-related EBP barriers (i.e., barrier reduction mechanism). Based on the model, we tested the following hypotheses: (1) ARC will increase clinicians’ EBP adoption and use, (2) ARC’s effects on EBP adoption and use will be mediated by improved proficiency culture leading to increased EBP intentions, (3) ARC’s effects on EBP adoption and use will be mediated by a reduction in job-related EBP barriers, and (4) the contextual-motivational mechanism and the barrier reduction mechanism will explain unique variance in clinicians’ EBP adoption and use. The study tests these hypotheses in a randomized controlled trial using longitudinal data sequenced over a four year period.

Method

Participants

The study included fourteen non-profit specialty mental health agencies purposefully selected from a large mid-Western metropolitan area in the United States. Agencies were recruited by inviting their chief executive officers (CEOs) to attend a conference at a local university in order to learn about the project. CEOs who were interested in the study met individually with the study investigators to obtain additional information about the study and to provide information necessary to assess their agency’s eligibility. Agencies were chosen to reflect the structural and workforce characteristics typical of children’s mental health service organizations in the United States (Schoenwald et al., 2008). Each agency employed 10 or more clinicians that served youth referred for emotional or behavioral disorders. In order to avoid selection of “early adopters” (Rogers, 2003), agencies were excluded from the study if they had implemented strategic initiatives related to EBP adoption in the 12 months prior to recruitment or if they were affiliated with a mental health services research network. A total of 16 agencies that met criteria were selected for the study based on this process; however, two agencies declined, resulting in a sample size of 14. Following approval by the CEO, clinicians were recruited for participation during regularly scheduled staff meetings at which they provided written informed consent. Similar to national samples (Schoenwald et al., 2008), clinicians working in the agencies delivered a variety of mental health services to youth (e.g., individual psychotherapy, family therapy, skill-training, groups) with the specific theoretical orientation and approach for each youth determined by individual clinicians (e.g., cognitive behavioral, integrative-eclectic, family systems, behavioral).

A total of 475 clinicians participated in the study (n = 259 clinicians in ARC agencies, n = 216 clinicians in control agencies), representing 86% of clinicians in agency programs that provided clinical services to youth and their families. On average, 34 clinicians per agency participated in the study (SD = 23.88, min = 8, max = 96). Participating clinicians exhibited a variety of levels of training and educational backgrounds with the most common terminal degrees at the bachelor’s (19.2%, n = 91) and master’s levels (73.7%, n = 350) and the most common majors in social work (38.3%, n = 182) or an allied health field such as counseling (29.9%, n = 142). The majority of clinicians were highly experienced, working in mental health settings for an average of 9.14 years (SD = 8.87), female (82.1%, n = 390), and white (82.5%, n = 392) with an average age of 36.42 years (SD = 11.65). Clinicians in the ARC and control conditions did not differ on gender, age, race, education, years of experience, or turnover (all p’s > .05).

Procedure

The study was designed as a randomized controlled trial in which agencies were matched on size (i.e., number of clinicians and budget) and one agency from each matched pair was randomly assigned to participate in the ARC intervention for 36 months. Agencies in the control condition received no intervention, although clinicians in these agencies were contacted at equal intervals using identical protocols to collect identical information regarding organizational culture, EBP intentions, job-related EBP barriers, and EBP adoption and use. Following completion of data collection for the study, CEOs in the control group were offered a 3-day workshop in which they received feedback on their organizations’ culture profiles along with information on the use of ARC to improve services in their organizations.

Clinicians in the ARC and control groups completed study questionnaires on-site at their agencies at baseline (Time 1 or 6–8 weeks prior to the intervention phase), two-thirds of the way through the ARC intervention period (Time 2 or 24-months post-baseline), immediately upon completion of the ARC intervention period (Time 3 or 36-months post-baseline), and at 12-month follow-up (Time 4 or 48-months post-baseline). Questionnaires were sequenced and timed based on theory-derived expectations regarding the hypothesized mechanisms. Questionnaires for both groups included scales assessing organizational culture (Time 1 & 2), intentions to adopt EBPs (Time 3), job-related EBP barriers (Time 4), and EBP adoption and use (Time 4).

Participating clinicians provided written informed consent and completed questionnaires in face-to-face meetings scheduled during regular work hours without supervisors or managers present. To ensure confidentiality, research staff unaffiliated with the agencies administered the questionnaires and collected them from clinicians immediately upon completion. Clinicians in all agencies were notified at the beginning of the project that participation in the study was voluntary and were able to opt out by avoiding scheduled meetings or by returning unfilled questionnaires with no penalty or knowledge of their participation by supervisors. Clinicians were not compensated for participation in the study. Average clinician turnover was 28 percent annually at baseline and because of turnover and new hires, sample sizes varied from a high of 207 clinicians at baseline to a low of 191 clinicians at 12-month follow-up. The overall response rate among participating clinicians across four waves of data collection was 86%.

ARC Intervention

The procedures used to train the ARC specialists and the ARC manuals used to guide the implementation strategy were validated in three prior randomized controlled trials (Glisson et al., 2006; Glisson et al., 2010; Glisson et al., 2013). Detailed specification of the ARC intervention and a description of the ARC procedures used in the present study are available in manuals published by the University of Tennessee Children’s Mental Health Services Research Center (cmhsrc.utk.edu). Briefly, ARC employs external facilitators (i.e., ARC Specialists) who work with agency CEOs and administrators, an internal ARC liaison, and front-line clinical teams to: (a) embed five principles of service effectiveness into the agency’s operating procedures, (b) develop an organizational infrastructure (e.g., vertically integrated organizational action team and clinical level treatment teams) and tools (e.g., continuous quality improvement) necessary to identify and remove barriers to service effectiveness, and (c) generate staff and leadership mental models that support innovation and service improvement efforts. These strategies are enacted during a three-year process and are designed to create proficient organizational cultures that support the use of effective practices (e.g., EBPs) and an organization’s capacity to eliminate barriers that hinder service effectiveness.

In the present study, ARC was facilitated by external ARC specialists during a 36-month period from 2010 to 2013. ARC specialists had advanced graduate degrees and five or more years of experience working with mental health and social service agencies. Training for specialists included reading ARC manuals and facilitation guides, didactic sessions, and weekly consultation and supervision for the duration of the intervention period. Specialists facilitated the ARC intervention using separate manuals for team leaders and team members.

Measures

ARC Fidelity

Fidelity to the ARC implementation strategy was assessed as a manipulation check at the conclusion of the ARC intervention period using seven items from the ARC principles questionnaire (APQ). The APQ was validated in two prior RCTs of ARC in which it demonstrated reliability, sensitivity to change, and discriminant validity with respect to agencies in ARC and control conditions (Glisson, Schoenwald et al., 2010; Glisson, Hemmelgarn et al., 2013). Clinicians complete scale items that refer to the unit-level enactment of ARC principles and to the completion of planned ARC activities within the previous month. Sample items include “Our program makes changes to be more effective in serving clients” and “All service team members participate in decisions to improve services.” Items were accompanied by a 5-point rating scale ranging from 1 (never) to 5 (always). Alpha reliability of this scale was 0.75.

Organizational Proficiency Culture

Organizational proficiency culture was assessed using the 15-item proficiency scale from the Organizational Social Context (OSC) measure (Glisson et al., 2012; Glisson, Landsverk et al., 2008). The factor validity of the OSC (including the proficiency scale) was confirmed in two national studies of children’s mental health clinics and child welfare agencies, respectively, and subsequent studies confirmed its predictive validity for clinicians’ EBP attitudes (Aarons et al., 2012), program sustainability (Glisson, Schoenwald et al., 2008), clinician turnover (Glisson, Schoenwald et al., 2008), service quality (Olin et al. 2014), and youth mental health outcomes (Glisson et al., 2013; Williams & Glisson, 2013; Williams & Glisson, 2014a). Items assessing proficiency culture refer to shared norms and behavioral expectations that clinicians will place the well-being of each client first and will be competent and have up-to-date knowledge of effective treatments. Items were accompanied by a 5-point rating scale ranging from 1 (never) to 5 (always). Alpha reliability for the proficiency scale was .89.

Clinicians’ responses to the proficiency culture items were aggregated to the agency level based on a referent-shift consensus composition model (Chan, 1998) and evidence of adequate within-agency agreement using the rwg(j) index (James, Demaree, & Wolf, 1993). A cutoff of rwg(j) = .7 is recommended to support aggregation to the unit level (LeBreton & Senter, 2008). Examination of the rwg(j) values for all 14 agencies indicated an average rwg(j) of .97 with a range from .91 to .99. Given this evidence of adequate within-agency agreement, aggregate agency-level scores were used to characterize the organizational cultures of participating agencies. Agency scores were converted to T-scores with a μ = 50 and σ = 10 using national norms from the children’s mental health agency sample (Glisson, Landsverk et al., 2008).

EBP Intentions

Clinicians’ intentions to adopt EBPs (hereafter EBP intentions) were measured using five items developed following well-established procedures for constructing behavioral intention scales based on the theory of planned behavior (Fishbein et al., 2001). Items referred to clinicians’ intentions to adopt EBPs in their work with clients and to behaviors indicative of intentions to adopt EBPs. Following prior research on EBP adoption in children’s service systems (Aarons et al., 2009), the instructions defined an “evidence-based practice” as “a specific treatment protocol that has been developed through research and is supported by the results of controlled treatment studies.” Sample items include “I intend to use an EBP in each treatment session” and “Out of the next 10 new clients you see, how many would you expect to treat using an EBP?” Items were accompanied by a 7-point scale ranging from 1 (strongly disagree) to 7 (strongly agree) or an 11-point scale ranging from 0 to 10, as appropriate. Alpha reliability of this scale was 0.80.

Job-Related EBP Barriers

Job-related EBP barriers were assessed with five items rated on a scale from 1 (strongly disagree) to 7 (strongly agree). Items referred to job-related EBP barriers reported in the literature on EBP adoption in mental health settings with an emphasis on policies and procedures that support or inhibit EBP adoption (Aarons et al., 2009; Bartholomew et al., 2007; Gioia & Dziadosz, 2008; Raghavan et al., 2007). Clinicians’ indicated the extent to which they experienced EBP barriers within the last six months. Sample items include “My agency does not provide materials—such as workbooks or technology—that are necessary to use EBPs,” and “My agency would not change paperwork or forms to accommodate the use of an EBP.” Alpha reliability for this scale was 0.76.

EBP Adoption and Use

Guided by the taxonomy of implementation outcomes proposed by Proctor et al. (2010), EBP adoption was conceptualized in this study as any action to try or employ an EBP including a clinician’s initial implementation of an EBP. This definition supports two related but distinct components of assessing EBP adoption—an intervention-centered approach that assesses whether a clinician has adopted a specific EBP and a client-centered approach that assesses the extent to which a clinician uses or tries to use EBPs with clients on her or his caseload. Both facets of adoption were assessed in the present study.

First, clinicians’ EBP adoption was measured with a condensed version of the EBP checklist developed by Walrath et al. (2006) and validated in prior research on EBP adoption in children’s mental health settings (Aarons et al., 2009). Clinicians completed nine items selected a priori based on the client populations served by clinicians in this study and the most up-to-date evidence regarding the treatment of emotional and behavioral disorders of youth. Selected EBPs included those addressing both externalizing (e.g., parent-child interaction therapy) and internalizing (e.g., exposure/ systematic desensitization) problems. Items referred to whether or not clinicians had used each specific EBP with any client in the last six months. Clinicians’ responses were dichotomized to indicate whether or not clinicians had adopted any one or more of the specific EBPs within the last 6 months (no = 0, yes = 1).

Second, clinicians’ EBP use was assessed as the percentage of clients currently treated using EBPs. After reading a definition of EBP as the use of specific protocols supported by research, clinicians completed a broadband measure indicating the percentage of clients they currently treat using an EBP (0–100%). To contextualize clinicians’ responses and to ensure continuity with the EBP intentions measure, the same definition of evidence-based practice was provided. A broadband item was selected to complement the EBP intentions measure and capture clinicians’ EBP use with their entire range of clients (Fishbein et al., 2001). The point biserial correlation between EBP adoption and EBP use was r = .36 (p < .001) suggesting these measures captured related but distinct facets of clinicians’ EBP adoption behavior.

Control Variables

In order to optimize statistical power and adjust for differences in baseline organizational social context between the treatment and control agencies, baseline Organizational Social Context (OSC) profile scores were included as covariates in all analyses. OSC profile scores represent a configural approach to characterizing organizational social context (Schulte, Ostroff, Shmulyian, & Kinicki, 2009) that incorporates all six dimensions of culture and climate assessed by the OSC (i.e., proficiency, rigidity, resistance, engagement, functionality, stress). The predictive validity of OSC profile scores was confirmed in two separate samples of children’s mental health agencies on three separate outcome indicators including work attitudes, service quality, and clinical outcomes (Glisson et al., 2013; Glisson et al., 2014; Olin et al., 2014). OSC profile scores range from 1.00 to 3.00 with higher values indicating more positive organizational social contexts that are associated with more positive clinician attitudes toward EBPs, higher service quality, and better client outcomes. In the present study, agencies’ OSC profile scores at baseline ranged from 1.00 to 3.00 with a mean of 1.70 (SD = .72).

Prior research has produced mixed results regarding the association between clinician characteristics and EBP adoption (Aarons et al., 2012). In the present study, clinicians’ characteristics were unrelated to any of the mediators or EBP outcomes with the exception of clinicians’ years of experience in mental health settings (r = .16 with EBP adoption, p = .035). Inclusion of this variable as a covariate did not change the results of the analyses and as a result the more parsimonious models are presented.

Data Analysis

Given the hierarchical structure of the data (i.e., clinicians nested within agencies), analyses incorporated two-level mixed-effects regression models with random agency intercepts, also known as hierarchical linear models (Raudenbush & Bryk, 2002). Analyses were implemented using the TWOLEVEL procedure in Mplus software (Version 7). Models with a dichotomous outcome (i.e., EBP adoption) incorporated a logit link function (i.e., two-level mixed effects logistic regression). Cases with missing values were excluded listwise for each analysis based on the missing at random (MAR) assumption. This approach maximized statistical power and used all available information. Results of Little’s missing completely at random (MCAR) test supported the MCAR assumption for the study variables, χ2 = 24.08, df = 27, p = .626, suggesting that missing responses were not systematic and therefore were unlikely to bias the study results. Preliminary analyses confirmed the presence of significant between-agency variance on each of the clinician-level criterion variables (i.e., EBP adoption and use) and clinician-level mediator variables (i.e., EBP intentions, job-related EBP barriers) with ICC(1)’s ranging from .10 to .30.

Tests of mediation were conducted using the product of coefficients approach for multilevel mediation analysis described by Zhang, Zyphur, and Preacher (2009). Under this approach, simultaneous equations are fit to the data which parse the total effect of the independent variable on the dependent variable into direct and indirect effects based on the hypothesized mediation model. Given that all of the indirect effects in this study crossed from the organizational level to the clinician level (e.g., ARC → proficiency culture → EBP intentions → EBP adoption), the analyses incorporated the centered within context with means reintroduced (CWCM) procedures described by Zhang et al. (2009) in order to obtain unbiased estimates of the mediated effects and correct statistical tests in the lower-level portion of the models. The models test the extent to which ARC’s effects on EBP adoption and use are mediated through two mechanisms: (a) a contextual-motivational mechanism in which ARC creates proficient cultures that increase clinicians’ EBP intentions, and (b) a barrier-reduction mechanism in which ARC reduces job-related EBP barriers.

The statistical significance of the mediated effects was tested using two complementary approaches (Hayes & Scharkow, 2013; MacKinnon et al., 2002; MacKinnon, Lockwood, & Williams, 2004). First, the joint significance test provided a null hypothesis significance testing approach (Cohen & Cohen, 1983, p. 366; MacKinnon et al., 2002). This approach tests the null hypothesis for each leg of the mediated effect (i.e., H0: a = 0 and b = 0), and rejects H0 in the event that the null hypotheses for both the a and b coefficients are rejected in their respective models. In the case of serial mediation (e.g., ARC → proficiency culture → EBP intentions → EBP adoption), this logic is extended to the a, b, and c coefficients (see Figure 1). MacKinnon et al. (2002) demonstrated the superiority of the joint significance test over other null hypothesis significance tests for simple mediation models and Taylor and colleagues (2008) extended this finding to three-path serial mediation models such as those tested here.

Second, asymmetric 95% confidence limits were developed for each mediated effect based on computationally intensive Monte Carlo methods using 100,000 replications (Hayes & Scharkow, 2013; Preacher & Selig, 2012). Under this approach, the mediated effect is statistically significant if the 95% confidence intervals do not span zero. Monte Carlo methods have been recommended by methodologists because they provide more accurate confidence limits and exhibit higher statistical power (while maintaining Type I error rates) than normal theory approaches such as the Sobel test (Hayes & Scharkow, 2013; MacKinnon et al., 2004). Monte Carlo CIs were constructed using an R utility developed by Preacher & Selig (2012).

Results

Table 2 presents descriptive statistics and bivariate correlations for the agency- and clinician-level variables in the study. The small and non-significant correlation between ARC and baseline OSC profile (r = .03, p = .912) suggests the randomization procedure was successful in balancing the treatment and control groups on organizational social context at baseline. The pattern of correlations among the study variables provides discriminant and convergent validity evidence and suggests the indicators are not simply capturing common method variance. For example, the absolute value of the correlations ranged from .00 to .69 and variables such as EBP use at Time 4 related to other constructs such as OSC at Time 1 (r = .00), Proficiency culture at Time 2 (r = .25) and EBP intentions at Time 3 (r = .60) in differential and theoretically expected ways. Significant variation was observed on all of the clinician- and agency-level variables indicating they captured meaningful variation across clinicians and sites. For example, clinicians’ EBP use at 12-month follow-up ranged from 0% to 100%, with 14% of the sample reporting they used EBPs with 25% or less of their clients and agency’s proficiency scores ranged from .76 SDs below the national mean (22nd percentile) to 1.8 SDs above the national mean (96th percentile).

Table 2.

Main effects of ARC on clinician EBP intentions, job-related EBP barriers, EBP adoption and use

EBP intentions
(T3)		 Job-related EBP
barriers (T4)		 EBP adoption (T4)
(yes/ no)a 		EBP use (T4)
(% clients)
γ SE p γ SE p γ SE p γ SE p
Fixed Effects
  Intercept 4.99 .22 .000 3.06 .21 .000 1.10 .37 .003 55.94 7.51 .000
  OSC Profile (T1) .16 .27 .561 −.38 .21 .062 −.15 .47 .756 −4.22 5.67 .456
  ARC .64 .30 .030 −.70 .31 .026 1.16 .39 .003 25.13 8.40 .003
Random Effects
  Agency intercepts .13 .19 .09 168.63
  Residual variance 1.96 1.08 — 749.42
Pseudo-R2 .39 .52 .80 .79
ICC(1) .10 .27 — .30
Open in a new tab

Note: These are two-level mixed effects regression models with random agency intercepts; ARC = Availability, Responsiveness, and Continuity organizational implementation strategy; OSC = organizational social context; EBP = evidence-based practice; T1 = Time 1; T3 = Time 3; T4 = Time 4.

a

This is a 2-level mixed effects logistic regression model with random agency intercepts.

Manipulation Check

Examination of clinicians’ responses to the ARC fidelity measure at the conclusion of the ARC intervention period confirmed (a) clinicians in the ARC condition reported significantly higher fidelity to the ARC principles than clinicians in control agencies (γ = .26, p = .007, d = .48), and (b) agencies in the ARC condition did not differ in their fidelity to the ARC principles (τ = .02, SE = .02, p = .422). The average APQ score in the ARC group was 4.07 out of 5, corresponding to a rating of “almost always” and representing 81% fidelity as a percentage of the total possible score. These results support the experimental manipulation and indicate the ARC agencies successfully executed the ARC implementation strategy.

ARC Main Effects on EBP Adoption and Use

Hypothesis 1 stated ARC would increase clinicians’ self-reported EBP adoption and use. Results of the two-level mixed effects regression analyses supported this hypothesis (see Table 2). At 12-month follow-up, clinicians in ARC agencies exhibited significantly higher odds of adopting a well-established EBP (γ = 1.16, SE = .39, p =.003, OR = 3.19) representing a 219% increase in their odds of EBP adoption relative to control. This translates into an increase of 21% in the probability of EBP adoption for clinicians in ARC agencies.

Similarly, at 12-month follow-up, clinicians in ARC agencies reported significantly greater use of EBPs with clients (γ = 25.13, SE = 8.40, p =.003) representing a standardized mean difference of d = .79 and a 45% increase in EBP use relative to control. These results support ARC’s main effects on EBP adoption and use.

Contextual-Motivational Mechanism

Hypothesis 2 indicated ARC’s effects on clinicians’ EBP adoption and use would be mediated by improved proficiency culture followed by increased clinician EBP intentions. Three steps were required to test this cross-level, serial mediation hypothesis. First, ARC improved proficiency culture at Time 2 relative to control. As shown in Table 3, results of an ordinary least squares regression analysis indicated ARC increased proficiency culture at Time 2 (B = 6.91, SE = 2.49, p = .020) accounting for 28% of the variance in proficiency culture.

Table 3.

ARC effects on clinician EBP adoption and use through proficiency culture and EBP intentions

Consequent
Proficiency culture
(T2)a 		EBP intentions
(T3)b 		EBP adoption
(yes/no) (T4)b 		EBP use
(% clients) (T4)b
Antecedent B SE p γ SE p γ SE p γ SE p
Intercept 41.97 3.79 <.001 5.22 .19 <.001 1.66 .36 <.001 60.82 4.15 <.001
OSC profile (T1) 7.19 2.03 .005 −.36 .21 .087 .13 .63 .834 −6.04 7.44 .417
ARC 6.91 2.49 .020 .06 .63 .928 .92 .61 .127 13.90 8.73 .111
Proficiency culture (T2) .08 .04 .045 −.05 .07 .451 −.65 .91 .478
EBP intentions-WG (T3)c .25 .20 .217 10.69 1.47 <.001
EBP intentions-BG (T3)c 1.75 .60 .003 24.32 5.36 <.001
Open in a new tab

Note: ARC = Availability, Responsiveness, and Continuity organizational intervention; BG = between group; EBP = evidence-based practice; OSC = organizational social context; T1 = Time 1; T2 = Time 2; T3 = Time 3, T4 = Time 4; WG = within group.

a

This is an ordinary least squares multiple regression model; F(2, 10) = 8.97, p = .006, R2 = .64.

b

This is a two-level mixed effects regression model with random agency intercepts.

c

This variable is centered within context with means reintroduced (CWCM).

Second, a mixed effects regression analysis indicated ARC increased clinicians’ EBP intentions at Time 3 (γ = .64, SE = .30, p =.030), representing a standardized mean difference of d = .44 and accounting for 39% of the between-agency variance (see Table 2). Results from a second mixed effects regression analysis supported mediation by showing that ARC’s effect on EBP intentions was reduced when proficiency culture was added to the model (γ = .06, SE = .63, p =.928) and higher proficiency culture predicted higher clinician EBP intentions (γ = .08, SE = .04, p = .045), controlling for ARC and baseline OSC profile (see Table 3). The joint significance test and the 95% Monte Carlo CIs (ab = .56, 95% CI [.01 to 1.38]) confirmed ARC’s effect on clinicians’ EBP intentions was significantly mediated by improved proficiency culture (see Table 5). The ratio of ARC’s indirect effect to its total effect was PM = .88, suggesting improved proficiency culture accounted for 88% of ARC’s total effect on clinicians’ EBP intentions (Preacher & Kelley, 2011).

Table 5.

Indirect effects of ARC on clinicians’ EBP adoption and use

95% Monte
Carlo CIs
ARC indirect effects Coeff. Lower
limit		 Upper
limit		 Joint sig.
test
Contextual Motivational Mechanism
  ARC → Proficiency → EBP intentions (ab) .56* .01 1.38 Sig.
  ARC → Proficiency → EBP intentions → EBP adoption (abc) 1.12* .17 2.74 Sig.
  ARC → Proficiency → EBP intentions → EBP use (abd) 15.63* 3.71 33.31 Sig.
Barrier Reduction Mechanism
  ARC → Job-related EBP barriers → EBP adoption (ij) .69* .06 1.60 Sig.
  ARC → Job-related EBP barriers → EBP use (ik) 12.47* 1.52 27.67 Sig.
Open in a new tab
*

p< .05

Note: ARC = Availability, Responsiveness, and Continuity; EBP = evidence-based practice.

Third, ARC’s effects on EBP adoption and use were serially transmitted through improved proficiency culture and subsequently increased EBP intentions (see Figure 1). ARC’s effect on EBP adoption at Time 4 was reduced once proficiency culture and EBP intentions were added to the model (γ = .92, SE = .61, p =.127) and increased clinician EBP intentions was the only significant predictor of EBP adoption (γ = 1.75, SE = .60, p = .003) after controlling for ARC, proficiency culture, and baseline OSC profile (see Table 3). As is shown in Table 5, results of the joint significance test of the a, b, and c coefficients for the hypothesized serial mediation effect (see Figure 1) as well as the 95% Monte Carlo CIs supported the hypothesized model (a × b × c = 1.12, 95% CI [.17 to 2.74]). Results from the analysis indicate ARC exerted a significant indirect effect through the serial influence of improved proficiency culture and increased EBP intentions accounting for 96% of ARC’s total effect on clinicians’ EBP adoption (i.e., PM = .96).

Results of the serial mediation analysis for clinicians’ EBP use mirrored the results for EBP adoption. ARC’s effects on EBP use were decreased once proficiency culture and EBP intentions were added to the model (γ = 13.90, SE = 8.73, p =.111) and increased clinician EBP intentions was the only significant predictor of EBP use (γ = 24.32, SE = 5.36, p < .001) after controlling for ARC, proficiency culture, and baseline OSC profile (see Table 3). Results of the joint significance test of the a, b, and d coefficients for this serial mediation effect (see Figure 1), as well as the 95% Monte Carlo CIs (abd = 15.36, 95% CI [3.71 to 33.31]), indicated ARC’s effect on EBP use was serially mediated by improved proficiency culture and increased EBP intentions (see Table 5). This two-step, contextual-motivational mechanism accounted for 61% of ARC’s total effect on EBP use.

Barrier Reduction Mechanism

Hypothesis 3 tested an alternative cross-level mechanism, namely that reduced job-related EBP barriers would mediate ARC’s effects on clinicians’ EBP adoption and use. This hypothesis required two tests. First, ARC reduced job-related EBP barriers (see Table 2). Clinicians in ARC agencies reported significantly fewer job-related EBP barriers at 12-month follow-up relative to control agencies (γ = −.70, SE = .31, p = .026) representing a standardized mean difference of d = −.59 and accounting for 52% of the between-agency variance.

Second, decreased job-related EBP barriers predicted EBP adoption and use controlling for ARC and baseline OSC profile (see Table 4). ARC’s effect on clinicians’ EBP adoption was diminished when job-related EBP barriers was added to the model (γ = .63, SE = .36, p =.075) and decreased job-related EBP barriers predicted increased EBP adoption (γ = −.90, SE = .30, p = .003), controlling for ARC and OSC profile. The joint significance test and the Monte Carlo 95% CIs for this mediated effect confirmed ARC’s effect was mediated by reduced job-related EBP barriers (ij = .69, 95% CI [.06 to 1.60]), accounting for 59% of ARC’s total effect on clinicians’ EBP adoption (Table 5).

Table 4.

ARC effects on clinician EBP adoption and use through job-related EBP barriers

Consequent
EBP adoption
(yes/no) (T4)a 		EBP use
(% clients) (T4)b
Antecedent γ SE p γ SE p
Intercept 1.39 .26 <.001 62.60 6.29 <.001
OSC profile (T1) −.49 .32 .124 −8.89 5.25 .090
ARC .63 .36 .075 12.54 6.95 .071
Job-related EBP barriers-WG (T4)c −.37 .22 .099 −9.40 2.18 <.001
Job-related EBP barriers-BG (T4)c −.90 .30 .003 −17.46 5.35 .001
Open in a new tab

Note: ARC = Availability, Responsiveness, and Continuity organizational intervention; BG = between group; EBP = evidence-based practice; OSC = organizational social context; T1 = Time 1; T4 = Time 4; WG = within group.

a

This is a two-level mixed effects logistic regression model with random agency intercepts.

b

This is a two-level mixed effects regression model with random agency intercepts.

c

This variable is centered within context with means reintroduced (CWCM).

Results from the analysis of EBP use were substantively similar to the results for EBP adoption (see Table 5). ARC’s effect on EBP use was diminished once job-related EBP barriers was added to the model (γ = 12.54, SE = 6.95, p =.071) and decreased job-related EBP barriers predicted increased EBP use (γ = −17.46, SE = 5.35, p = .001) in the same model (see Table 4). Results from the joint significance test and the Monte Carlo 95% CIs (ik = 12.47, 95% CI [1.52 to 27.67]) confirmed ARC’s effect was mediated by reduced job-related EBP barriers accounting for 49% of ARC’s total effect on clinicians’ EBP use.

Unique Contributions of the Contextual-Motivational and Barrier Reduction Mechanisms

The analyses reported above indicate ARC’s effects on clinicians’ EBP adoption and use were explained in part by improved proficiency culture which increased clinicians’ EBP intentions and by reduced job-related EBP barriers. Hypothesis 4 tested these two mechanisms simultaneously to assess the unique variance explained by each. This hypothesis was tested by estimating two-level mixed effects regression models in which proficiency culture, EBP intentions, and job-related EBP barriers simultaneously predicted clinicians’ EBP adoption (or use), controlling for ARC and OSC profile. Results from these analyses were substantively identical. In both analyses, EBP intention was the only significant predictor of EBP adoption (γ = 1.68, SE = .79, p = .032) and EBP use (γ = 21.91, SE = 7.10, p = .002) once all covariates were included in the model. These findings indicate clinicians’ increased EBP intentions rather than reduced job-related EBP barriers played the larger unique role in explaining ARC’s effects on clinicians’ EBP adoption and use.

Discussion

Results from this four year randomized controlled trial suggest mental health service systems can increase clinicians’ EBP adoption and use by creating proficient organizational cultures that prioritize client well-being and expect clinician competence in up-to-date treatment practices and by reducing job-related EBP barriers. To our knowledge, this is the first experimental study to support the mediational role of planned organizational culture change in shaping mental health clinicians’ practice behaviors (Parmelli et al., 2011). These results are important given that changes in clinicians’ self-reported behaviors were confirmed 12 months after the ARC intervention was concluded and were explained by sequentially measured effects on organizational proficiency culture and clinicians’ EBP intentions.

Results from this study support organizational culture theory in showing that purposeful change of an agency’s culture can contribute to targeted changes in mental health clinicians’ practice behaviors (Schein, 2004). A growing body of intervention studies in a variety of settings have demonstrated the effects of context-focused organizational interventions on organizational social contexts (Glisson, Hemmelgarn, et al., 2012), employees’ work behaviors (Larson et al., 2000; Zohar & Polachek, 2014), and clinical outcomes (Glisson et al., 2010; Glisson et al., 2013). The present study extends this research by demonstrating that the effects of the ARC organizational strategy on EBP adoption and use are mediated by improvements in organizational culture.

Findings from this study show that organizational proficiency culture and clinicians’ EBP intentions are sequential components of an effective mechanism for increasing clinicians’ EBP adoption. Moreover, results suggest these components may be more important determinants of clinicians’ EBP adoption behavior than job-related EBP barriers. Although prior correlational and qualitative studies have associated these variables with clinicians’ EBP adoption (Chaudoir, Dugan, & Barr, 2013), this is the first study to experimentally test the variance explained by each of these hypothesized mechanisms. The results support a social contextual perspective on EBP adoption in which prevailing cultural norms and subsequent socially-influenced intentions play a larger role than job-related barriers in shaping clinicians’ innovation behavior. This finding is consistent with diffusion of innovations theory (Rogers, 2003) which has argued for many decades that social processes are more important to innovation adoption and implementation than the technical requirements of the innovations themselves. The implication is that more attention should be devoted to developing efficient strategies for creating organizational social contexts in community-based mental health services that support targeted innovations such as EBPs. The role of social context in these findings provide a counter-balance and complement to the predominant focus on simply increasing technical competence in implementation studies, suggesting that both the technical and social aspects of implementation must be addressed to improve implementation outcomes (Novins et al., 2013).

The range of outcomes that occur during a phased implementation process provides an important lens for interpreting these results (Proctor et al., 2010). For example, implementation models such as the Exploration, Preparation/ Adoption, Implementation, and Sustainment model proposed by Aarons et al. (2011) suggest adoption occurs relatively early in the implementation process following an exploration phase and preceding active implementation and sustainment phases. Our findings indicate organizational proficiency culture and clinicians’ EBP intentions are more important than job-related barriers for increasing EBP adoption. However, different mechanisms could be more important for achieving other implementation outcomes in other phases. Although there is evidence from other studies that the advantages of a proficient organizational culture include positive effects on other implementation outcomes (e.g., Aarons et al., 2012), further research is needed to test these hypotheses and link the phases empirically.

An important challenge for future studies is addressing the different ways in which implementation strategies focus on organizational social context in shaping implementation outcomes. Whereas the present study addresses proficiency culture, other models of EBP implementation focus on different contextual constructs (Aarons et al., 2014; Ehrhart, Aarons, & Farahnak, 2014). For example, the EBP implementation climate construct focuses more narrowly on the implementation of EBPs (Ehrhart et al., 2014). Important questions for the field include (a) the extent to which proficiency culture and other contextual constructs such as EBP implementation climate have differential effects on the range of implementation outcomes (e.g., acceptability, adoption, fidelity, sustainment) with different EBPs (e.g., modular cognitive behavioral therapy, measurement-based care), and (b) which dimensions of organizational social context should be targeted to optimize ongoing EBP exploration, adoption, implementation, and sustainment as new EBPs are developed? The norms and expectations that characterize a proficient organizational culture may have advantages over other constructs in terms of generalizing positive effects across EBPs, time, and the range of implementation outcomes representing each phase.

Limitations

Four caveats must be considered in interpreting these results. First, agencies were randomly assigned to ARC and control conditions after matching on size so that clinicians were cluster randomized by the agencies that employed them. Cluster randomization is a well-accepted design strategy, particularly in implementation trials where randomization of organizations often results in relatively small intervention sample sizes. However, unmeasured clinician variables could have influenced the results. This concern is mitigated by the finding that there were no differences between the two conditions in clinician education, years of experience, gender, age, race, or turnover during the study period. Moreover, there was no significant difference in organizational social context between ARC and control agencies at baseline but differences emerged over the course of the study in organizational culture, clinicians’ EBP intentions, job-related EBP barriers, and EBP adoption and use. In addition, the effect of ARC on EBP adoption and use was fully mediated by the sequential improvement in proficiency culture and increased EBP intentions during the study period. Collectively, these findings suggest the threat of unmeasured variables confounding the results is minimized.

A second caveat is clinicians’ self-reported EBP adoption and use may not fully capture outcomes associated with each of the phases of EBP exploration, preparation, adoption, implementation, and sustainment (Aarons et al., 2011). Implementation research focuses on a variety of implementation outcomes and the variables incorporated into this study readily map onto the implementation outcome of adoption as described by Proctor and colleagues (Proctor et al., 2010). EBP adoption and use with clients represent necessary but not sufficient conditions for effective integration of EBPs into routine care. Clinicians are uniquely aware of the practice behaviors they attempt to use in sessions with clients and their reports of efforts to use EBPs represent but one important dimension in the effective integration of EBPs into community-based mental health services.

A third caveat concerns the reliability of the self-report measures of EBP adoption and use given evidence that observational and self-report measures of clinicians’ EBP adoption do not always correspond (Hogue, Dauber, Henderson, & Liddle, 2014; Hurlburt et al., 2010). There were several findings that reduce but do not eliminate these concerns. First, the wide range of values observed on the EBP outcome measures and the high percentage of clinicians’ who reported low EBP use increases confidence in the reliability of the findings by reducing concerns about social desirability biases. Also, the fact that ARC’s effects were observed across multiple indicators over four years provides convergent evidence that supports the reliability of the observed changes and indicates they were sustained over time. The 12 month intervals between measurement occasions reduces concerns about common method error variance because the long time interval reduced the salience of clinicians’ earlier responses and impeded the retrieval of prior responses from memory (Podsakoff et al., 2003). Finally, the statistically and substantively large effects of ARC on the EBP outcomes (e.g., 50% use vs. 81% use) suggests ARC’s effects were practically meaningful.

A fourth caveat pertains to additional mediating variables that may help explain ARC’s effects on clinicians’ EBP adoption and use. Although this study focused on contextual and motivational mechanisms because of their theoretical and empirical support, other intervening variables could be introduced into the model to help explain ARC’s effects. For example, increased leader receptivity to providing agency-sponsored EBP training may mediate the relationship between proficient organizational culture and increased clinician EBP adoption. Elaborating these linkages is an important area for further research.

Conclusion

Findings from this randomized controlled trial provide evidence that the ARC organizational intervention can increase clinicians’ EBP adoption and use through the transformation of organizational culture and the activation of clinicians’ EBP intentions. These findings highlight directions for future research on the role of organizational social context in shaping a variety of implementation outcomes in mental health services. Results also suggest organizational social context and clinicians’ behavioral intentions represent salient targets for increasing EBP adoption and effectiveness in mental health systems. Additional studies that extend our knowledge of how to efficiently marshal these and other mechanisms are essential to advancing implementation science and practice in the complex arena of mental health services.

Table 1.

Descriptive Statistics and Correlations for Study Variables

Variable Mean SD Min. Max. 1 2 3 4 5 6
Agency-Level
  1. ARC .50 .52 0 1
  2. OSC Profile (T1) 1.70 .72 1.00 3.00 .03
  3. Proficiency culture (T2) 57.48 7.18 42.45 68.27 .50 .69**
Clinician-Level
  4. EBP intentions (T3) 5.24 1.47 1.20 7.60 .17* .06 .25**
  5. Job-related EBP barriers (T4) 2.71 1.18 1.00 6.00 −.23* −.27** −.30** −.43**
  6. EBP adoption (yes/no) (T4) .82 .39 0 1 .20* −.01 .17* .29* −.24*
  7. EBP use (% clients) (T4) 70.69 31.92 0 100 .34** .00 .25** .60** −.45** .36**
Open in a new tab

Note: ARC = Availability, Responsiveness, and Continuity organizational intervention; EBP = evidence-based practice, OSC = Organizational Social Context; T1 = Time 1; T2 = Time 2; T3 = Time 3; T4 = Time 4.

*

p < .05,

**

p < .01

Acknowledgments

Funding: This study was funded by grants from the National Institute of Mental Health to NJW (F31MH099846) and CG (R01MH084855). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health.

Footnotes

Compliance with Ethical Standards:

Ethical approval: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Conflict of interest: Nathaniel Williams declares that he has no conflict of interest. Charles Glisson declares that he has no conflict of interest. Anthony Hemmelgarn declares that he has no conflict of interest. Philip Green declares that he has no conflict of interest.

Informed consent: Informed consent was obtained from all individual participants included in the study.

References

  1. Aarons GA, Ehrhart MG, Farahnak LR, Sklar M. Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annual Review of Public Health. 2014;35:255–274. doi: 10.1146/annurev-publhealth-032013-182447. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Aarons GA, Glisson C, Green PD, Hoagwood K, Kelleher KJ, Landsverk JA The Research Network on Youth Mental Health. The organizational social context of mental health services and clinician attitudes toward evidence-based practice: A United States national study. Implementation Science. 2012;7(56) doi: 10.1186/1748-5908-7-56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public sectors. Administration and Policy in Mental Health. 2011;38:4–23. doi: 10.1007/s10488-010-0327-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Aarons GA, Sommerfeld DH, Walrath-Greene CM. Evidence-based practice implementation: The impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implementation Science. 2009;4(83) doi: 10.1186/1748-5908-4-83. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Ajzen I. The theory of planned behavior. Organizational Behavior and Human Decision Processes. 1991;50:179–211. [Google Scholar]
  6. Baer JS, Wells EA, Rosengren DB, Hartzler B, Beadnell B, Dunn C. Agency context and tailored training in technology transfer: A pilot evaluation of motivational interviewing training for community counselors. Journal of Substance Abuse Treatment. 2009;37:191–202. doi: 10.1016/j.jsat.2009.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bartholomew NG, Joe GW, Rowan-Szal GA, Simpson DD. Counselor assessments of training and adoption barriers. Journal of Substance Abuse Treatment. 2007;33:193–199. doi: 10.1016/j.jsat.2007.01.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Chan D. Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology. 1998;83:234–246. [Google Scholar]
  9. Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation Science. 2013;8(22) doi: 10.1186/1748-5908-8-22. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cohen J, Cohen P. Applied multiple regression/ correlation analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum; 1983. [Google Scholar]
  11. Collins PY, Patel V, Joestl SS, March D, Insel TR, Daar AS, Bordin IA, Walport M. Grand challenges in global mental health. Nature. 2011;475:27–30. doi: 10.1038/475027a. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Cooke RA, Rousseau DM. Behavioral norms and expectations: A quantitative approach to the assessment of organizational culture. Group & Organization Studies. 1988;13:245–273. [Google Scholar]
  13. Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS) Implementation Science. 2014;9:157. doi: 10.1186/s13012-014-0157-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Fishbein M, Triandis HC, Kanfer FH, Becker M, Middlestadt SE, Eichler A. Factors influencing behavior and behavior change. In: Baum A, Revenson T, Singer J, editors. Handbook of Health Psychology. Mahwah, NJ: Lawrence Erlbaum Associates; 2001. pp. 3–17. [Google Scholar]
  15. Gioia D, Dziadosz G. Adoption of evidence-based practices in community mental health: A mixed methods study of practitioner experience. Community Mental Health Journal. 2008;44:347–357. doi: 10.1007/s10597-008-9136-9. [DOI] [PubMed] [Google Scholar]
  16. Glisson C, Dukes D, Green P. The effects of the ARC organizational intervention on caseworker turnover, climate, and culture in children’s service systems. Child Abuse and Neglect. 2006;30:855–880. doi: 10.1016/j.chiabu.2005.12.010. [DOI] [PubMed] [Google Scholar]
  17. Glisson C, Green P, Williams NJ. Assessing the organizational social context (OSC) of child welfare systems: Implications for research and practice. Child Abuse and Neglect. 2012;36:621–632. doi: 10.1016/j.chiabu.2012.06.002. [DOI] [PubMed] [Google Scholar]
  18. Glisson C, Hemmelgarn A, Green P, Dukes D, Atkinson S, Williams NJ. Randomized trial of the availability, responsiveness, and continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth. Journal of the American Academy of Child and Adolescent Psychiatry. 2012;51:780–787. doi: 10.1016/j.jaac.2012.05.010. [DOI] [PubMed] [Google Scholar]
  19. Glisson C, Hemmelgarn A, Green P, Williams NJ. Randomized trial of the availability, responsiveness and continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. Journal of the American Academy of Child and Adolescent Psychiatry. 2013;52:493–500. doi: 10.1016/j.jaac.2013.02.005. [DOI] [PubMed] [Google Scholar]
  20. Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, Green P The Research Network on Youth Mental Health. Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health. 2008;35:98–113. doi: 10.1007/s10488-007-0148-5. [DOI] [PubMed] [Google Scholar]
  21. Glisson C, Schoenwald SK, Hemmelgarn A, Green P, Dukes D, Armstrong KS, Chapman JE. Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology. 2010;78:537–550. doi: 10.1037/a0019160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Glisson C, Schoenwald SK, Kelleher K, Landsverk J, Hoagwood KE, Mayberg S, Green P The Research Network on Youth Mental Health. Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Administration and Policy in Mental Health. 2008;35:124–133. doi: 10.1007/s10488-007-0152-9. [DOI] [PubMed] [Google Scholar]
  23. Glisson C, Williams NJ, Green P, Hemmelgarn A, Hoagwood K. The organizational social context of mental health Medicaid waiver programs with family support services: Implications for research and practice. Administration and Policy in Mental Health. 2014;41:32–42. doi: 10.1007/s10488-013-0517-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hartnell CA, Ou AY, Kinicki A. Organizational culture and organizational effectiveness: A meta-analytic investigation of the competing values framework’s theoretical suppositions. Journal of Applied Psychology. 2011;96:677–694. doi: 10.1037/a0021987. [DOI] [PubMed] [Google Scholar]
  25. Hayes AF, Scharkow M. The relative trustworthiness of inferential tests of the indirect effect in statistical mediation analysis: Does method really matter? Psychological Science. 2013;24:1918–1927. doi: 10.1177/0956797613480187. [DOI] [PubMed] [Google Scholar]
  26. Hofstede G, Neuijen B, Ohayv DD, Sanders G. Measuring organizational cultures: A qualitative and quantitative study across twenty cases. Administrative Science Quarterly. 1990;35:286–316. [Google Scholar]
  27. Hogue A, Dauber S, Henderson CE, Liddle HA. Reliability of therapist self-report on treatment targets and focus in family-based intervention. Administration and Policy in Mental Health. 2014;41:697–705. doi: 10.1007/s10488-013-0520-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Hurlburt MS, Garland AF, Nguyen K, Brookman-Frazee L. Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health. 2010;37:230–244. doi: 10.1007/s10488-009-0251-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Insel TR. Translating scientific opportunity into public health impact. Archives of General Psychiatry. 2009;66:128–133. doi: 10.1001/archgenpsychiatry.2008.540. [DOI] [PubMed] [Google Scholar]
  30. Jaccard J, Litardo HA, Wan CK. Subjective culture and social behavior. In: Adamopoulos J, Kashima Y, editors. Social psychology and cultural context. Thousand Oaks, CA: Sage publications; 1999. pp. 95–106. [Google Scholar]
  31. James LR, Demaree RG, Wolf G. Rwg: An assessment of within-group agreement. Journal of Applied Psychology. 1993;78:306–309. [Google Scholar]
  32. Jensen-Doss A, Hawley KM, Lopez M, Osterberg LD. Using evidence-based treatments: The experiences of youth providers working under a mandate. Professional Psychology: Research and Practice. 2009;40:417–424. [Google Scholar]
  33. Larson EL, Early E, Cloonan P, Sugrue S, Parides M. An organizational climate intervention associated with increased handwashing and decreased nosocomial infections. Behavioral Medicine. 2000;26:14–22. doi: 10.1080/08964280009595749. [DOI] [PubMed] [Google Scholar]
  34. LeBreton JM, Senter JL. Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods. 2008;11:815–852. [Google Scholar]
  35. MacKinnon DP, Lockwood CM, Hoffman JM, West SG, Sheets V. A comparison of methods to test mediation and other intervening variable effects. Psychological Methods. 2002;7:83–104. doi: 10.1037/1082-989x.7.1.83. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. MacKinnon DP, Lockwood CM, Williams J. Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research. 2004;39:99–128. doi: 10.1207/s15327906mbr3901_4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Michie S, van Stralen MM, West R. The behavior change wheel: A new method for characterizing and designing behavior change interventions. Implementation Science. 2011;6(42) doi: 10.1186/1748-5908-6-42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. National Institute of Mental Health. National Institute of Mental Health strategic plan. Washington, DC: U. S. Department of Health and Human Services; 2008. [Google Scholar]
  39. Novins DK, Green AE, Legha RK, Aarons GA. Dissemination and implementation of evidence-based practices for child and adolescent mental health: A systematic review. Journal of the American Academy of Child and Adolescent Psychiatry. 2013;52:1009–1025. doi: 10.1016/j.jaac.2013.07.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Olin SS, Williams N, Pollock M, Armusewicz K, Kutash K, Glisson C, Hoagwood KE. Quality indicators for family support services and their relationship to organizational social context. Administration and Policy in Mental Health and Mental Health Services Research. 2014;41:43–54. doi: 10.1007/s10488-013-0499-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Parmelli E, Flodgren G, Beyer F, Baillie N, Schaafsma ME, Eccles MP. The effectiveness of strategies to change organizational culture to improve healthcare performance: A systematic review. Implementation Science. 2011;6(33) doi: 10.1186/1748-5908-6-33. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, Glass JE, York JL. A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review. 2012;69:123–157. doi: 10.1177/1077558711430690. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Powell BJ, Proctor EK, Glass JE. A systematic review of strategies for implementing empirically supported mental health interventions. Research on Social Work Practice. 2014;24:192–212. doi: 10.1177/1049731513505778. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Preacher KJ, Kelley K. Effect size measures for mediation models: Quantitative strategies for communicating indirect effects. Psychological Methods. 2011;16:93–115. doi: 10.1037/a0022658. [DOI] [PubMed] [Google Scholar]
  45. Preacher KJ, Selig JP. Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures. 2012;6:77–98. [Google Scholar]
  46. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Hensley M. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health. 2010;38:65–76. doi: 10.1007/s10488-010-0319-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Raghavan R, Inkelas M, Franke T, Halfon N. Administrative barriers to the adoption of high-quality mental health services for children in foster care: A national study. Administration and Policy in Mental Health and Mental Health Services Research. 2007;34:191–201. doi: 10.1007/s10488-006-0095-6. [DOI] [PubMed] [Google Scholar]
  48. Raudenbush SW, Bryk AS. Hierarchical linear models: Applications and data analysis. 2nd. Thousand Oaks, CA: Sage publications; 2002. [Google Scholar]
  49. Rogers EM. Diffusion of innovations. 5th. New York: Free Press; 2003. [Google Scholar]
  50. Schein EH. Organizational culture and leadership. San Francisco, CA: Jossey-Bass; 2004. [Google Scholar]
  51. Schneider B, Brief AP, Guzzo RA. Creating a climate and culture for sustainable organizational change. Organizational dynamics. 1996;24:7–19. [Google Scholar]
  52. Schoenwald SK, Chapman JE, Kelleher K, Hoagwood KE, Landsverk J, Stevens J, Glisson C, Rolls-Reutz J The Research Network on Youth Mental Health. A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs) Administration and Policy in Mental Health. 2008;35:84–97. doi: 10.1007/s10488-007-0147-6. [DOI] [PubMed] [Google Scholar]
  53. Schulte M, Ostroff C, Shmulyian S, Kinicki A. Organizational climate configurations: Relationships to collective attitudes, customer satisfaction, and financial performance. Journal of Applied Psychology. 2009;94:618–634. doi: 10.1037/a0014365. [DOI] [PubMed] [Google Scholar]
  54. Sheeran P. Intention-behavior relations: A conceptual and empirical review. In: Stroebe W, Hewstone M, editors. European Review of Social Psychology, Vol. Vol. 12. London: Wiley; 2002. p. 136. [Google Scholar]
  55. Taylor AB, MacKinnon DP, Tein JY. Tests of the three-path mediated effect. Organizational Research Methods. 2008;11:241–269. [Google Scholar]
  56. Trice HM, Beyer JM. The cultures of work organizations. Englewood Cliffs, NJ: Prentice Hall; 1993. [Google Scholar]
  57. Walrath CM, Sheehan AK, Holden EW, Hernandez M, Blau G. Evidence-based treatments in the field: A brief report on provider knowledge, implementation, and practice. Journal of Behavioral Health Services Research. 2006;33:244–253. doi: 10.1007/s11414-005-9008-9. [DOI] [PubMed] [Google Scholar]
  58. Williams NJ. Multilevel mechanisms of implementation strategies in mental health: Integrating theory, research, and practice. Administration and Policy in Mental Health and Mental Health Services Research. 2015 doi: 10.1007/s10488-015-0693-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Williams NJ, Glisson C. Reducing turnover is not enough: The need for proficient organizational cultures to support positive youth outcomes in child welfare. Children and Youth Services Review. 2013;35:1871–1877. doi: 10.1016/j.childyouth.2013.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Williams NJ, Glisson C. Testing a theory of organizational culture, climate, and youth outcomes in child welfare systems: A United States national study. Child Abuse & Neglect. 2014a;38:757–767. doi: 10.1016/j.chiabu.2013.09.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Williams NJ, Glisson C. The role of organizational culture and climate in the dissemination and implementation of empirically-supported treatments for youth. In: Beidas RS, Kendall PC, editors. Dissemination and Implementation of Evidence Based Practices in Child and Adolescent Mental Health. New York: Oxford University Press; 2014b. pp. 61–81. [Google Scholar]
  62. Zhang Z, Zyphur MJ, Preacher KJ. Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods. 2009;12:695–719. [Google Scholar]
  63. Zohar D, Polachek T. Discourse-based intervention for modifying supervisory communication as leverage for safety climate and performance improvement: A randomized field study. Journal of Applied Psychology. 2014;99:113–124. doi: 10.1037/a0034096. [DOI] [PubMed] [Google Scholar]

RESOURCES