A Multi-Level Examination of Stakeholder Perspectives of Implementation of Evidence-Based Practices in a Large Urban Publicly-Funded Mental Health System

Abstract

Our goal was to identify barriers and facilitators to the implementation of evidence-based practices from the perspectives of multiple stakeholders in a large publicly funded mental health system. We completed 56 interviews with three stakeholder groups: treatment developers (n = 7), agency administrators (n = 33), and system leadership (n = 16). The three stakeholder groups converged on the importance of inner (e.g., agency competing resources and demands, therapist educational background) and outer context (e.g., funding) factors as barriers to implementation. Potential threats to implementation and sustainability included the fiscal landscape of community mental health clinics and an evolving workforce. Intervention characteristics were rarely endorsed as barriers. Inner context, outer context, and intervention characteristics were all seen as important facilitators. All stakeholders endorsed the importance of coordinated collaboration across stakeholder groups within the system to successfully implement evidence-based practices.

Evidence-based practices (EBPs; American Psychological Association Presidential Task Force on Evidence-Based Practice, 2006; Institute of Medicine Committee on the Quality of Health Care in America, 2001; Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996) for individuals with psychiatric disorders are not widely available in community settings (American Psychological Association, 2009). Implementation of EBPs in the community poses a major challenge for the behavioral health field and results of efforts to implement EBPs have had some disappointing results (McHugh & Barlow, 2010). Understanding the perspectives of stakeholders involved in the implementation process can provide a richer and more nuanced understanding of how best to implement EBPs in future efforts.

Implementation science frameworks provide important conceptual guidance around contextual factors that influence implementation. Typically these factors are multi-level and occur at the individual (i.e., characteristics of the therapist such as knowledge and attitudes), organizational (i.e., characteristics of the provider setting including organizational culture and climate), intervention (i.e., characteristics of the treatment intervention such as usability) and system levels (i.e., characteristics of the service system within which implementation is occurring such as financing levels; Aarons, Hurlburt, & Horwitz, 2011; Damschroder et al., 2009; Fixsen, Blase, Naoom, & Wallace, 2009; Southam-Gerow, Rodriguez, Chorpita, & Daleiden, 2012). Understanding multi-level barriers and facilitators from the perspective of all stakeholders can be greatly informative to improving the implementation process.

A number of studies have been conducted to examine barriers and facilitators of implementation (Proctor, Powell, Baumann, Hamilton, & Santens, 2012). These studies have identified a host of barriers and facilitators at the individual therapist level (e.g., attitudes and training; Pagoto et al., 2007; Palinkas et al., 2008; Stirman et al., 2013); organizational level (e.g., organizational structure and support; Langley, Nadeem, Kataoka, Stein, & Jaycox, 2010; Ringle et al., 2015); intervention level (e.g., structure; Ringle et al., 2015) and system level (e.g., social networks; Palinkas et al., 2011). Although these studies have provided important insights about barriers and facilitators to implementation, most of the studies only interviewed therapists (for an exception, see Palinkas et al., 2011) and did not interview different stakeholder groups involved in the implementation process such as agency leaders, system leaders, and treatment developers.

Few studies have focused on multiple stakeholder perspectives of barriers and facilitators. One exception is a study of EBP implementation that utilized concept mapping with six stakeholder groups to identify fourteen facilitators and barriers to implementation (Aarons, Wells, Zagursky, Fettes, & Palinkas, 2009). Perceptions of the importance and malleability of these factors varied across stakeholder groups, with differences identified between policy and clinically-focused stakeholders (Green & Aarons, 2011). The importance of including multiple stakeholders is underscored by recent findings that greater consensus among leaders within organizations about barriers and facilitators was associated with greater adoption of EBPs (Palinkas et al., 2014).

In Philadelphia, the Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) has been supporting implementation of several EBPs, (Beidas et al., 2013; Beidas et al., 2015) providing an opportunity to study the implementation process in a large publicly funded mental health system that serves more than 100,000 people annually. Beginning in 2007, cognitive-behavioral therapy (CBT), an EBP for the treatment of several child and adult psychiatric disorders, was implemented (Creed, Stirman, Evans, & Beck, 2014; Stirman et al., 2010). DBHIDS subsequently implemented other EBPs, including Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT; Cohen, Deblinger, Mannarino, & Steer, 2004); Prolonged Exposure (Foa et al., 2005); and Dialectical Behavior Therapy (DBT; Linehan et al., 2006). Implementation of EBPs in the City of Philadelphia has occurred in an iterative process and has differed by initiative. Early on, selection of agencies for participation was largely guided by DBHIDS whereas, more recently, agencies applied for requests for proposals (RFPs) and were selected by DBHIDS based on their applications. DBHIDS provides (i.e., finances) training and consultation in line with treatment developer recommendations and a city employee who coordinates implementation, training, and ongoing consultation by treatment developers for each of the four initiatives. A fiscal incentive was provided for implementation of TF-CBT (see Powell et al., this issue for more specifics on the implementation strategies used by DBHIDS).

Guided by the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework (Aarons, Hurlburt, et al., 2011), the purpose of the present study is to determine the barriers and facilitators to implementation that intervention developers, agency administrators, and system leaders identify. This study builds upon previous work by using qualitative inquiry to generate rich data to understand the implementation process from multiple stakeholders involved in implementing multiple EBPs in several care settings (e.g., outpatient, residential treatment facilities) in a large urban public health system.

Methods

Participants and Procedure

This study was approved by the Institutional Review Board of the University of Pennsylvania and City of Philadelphia and informed consent was obtained for all participants. Comprehensive purposive sampling was used to identify and recruit members of three stakeholder groups who participated in DBHIDS sponsored initiatives. Fifty-six semi-structured interviews were conducted between March and August of 2014.

Treatment developers

DBHIDS provided us with the name of all individuals identified as treatment developers and/or lead trainers for the four initiatives. We contacted each individual via email to ascertain interest in participation. Seven interviews were conducted (response rate = 100%). No compensation was offered for participation. See Table 1 for demographics of all participants. Mean age was 56.6 (SD = 15.0).

Table 1.

Participant Demographics

Demographic Variable	Treatment Developers (n = 7)	Agency Administrators (n = 33)	System Leadership (n = 16)
Gender
Male	0%	33%	31%
Female	100%	67%	69%
Ethnicity
Caucasian	100%	64%	44%
African American	0%	18%	37%
American Indian/Alaskan Native	0%	9%	0%
Hispanic/Latino	0%	0%	6%
Other	0%	9%	13%
Educational Level
Bachelors Degree’s	0%	3%	0%
Masters Degree’s	0%	67%	44%
Doctoral Degree’s	100%	27%	50%

Agency stakeholders

To date, 42 community mental health clinics (CMHCs) in Philadelphia have participated in DBHIDS EBP initiatives. In collaboration with DBHIDS, we identified the person at each agency that would be best acquainted with operations and management of the EBP initiative (e.g., executive directors, clinical directors). We contacted each individual via email to ascertain interest in participation. Thirty-nine agencies agreed to participate; of those, thirty-three agencies were interviewed (response rate = 79%; the remaining six agencies were not responsive to scheduling requests after initially agreeing to participate). Agency stakeholders received $50 for participation. Mean age was 48.7 (SD = 11.5). Participants had been in their positions for a mean of 5.9 (SD = 5.6) years. Agencies participating were in varying stages of implementation and sustainment depending on when they started an initiative.

System-level leadership and key personnel

Sixteen individuals from DBHIDS and Community Behavioral Health (CBH) were invited to participate (response rate = 100%). CBH is the non-profit behavioral health managed care organization contracted by the City of Philadelphia to administer the behavioral health benefits for Philadelphia residents receiving Medical Assistance. These individuals were identified by leadership at DBHIDS as key players in the EBP initiatives. We contacted each individual via email to ascertain interest in participation. No compensation was offered to these participants. Mean age was 44.9 (SD = 10.4).

Qualitative Interview

We developed a semi-structured interview guide to collect information about participants’ experiences with EBP initiatives with a focus on inner (i.e., agency characteristics) and outer context (i.e., system characteristics) factors (see Appendix 1; one question was adapted from another interview guide; Palinkas et al., 2014). Our interview guide was informed by the EPIS framework (Aarons, Hurlburt, et al., 2011), a well-known implementation science framework developed for implementation of evidence-based psychosocial interventions in public sector settings. Interviews lasted from 23–58 minutes and were audiotaped and transcribed.

Data Analysis

Transcripts were analyzed in an iterative process based upon an integrated approach that incorporated both inductive and deductive features (Bradley, Curry, & Devers, 2007). Through a close reading of eight transcripts, the investigators developed a set of codes that were applied to the data (i.e., inductive approach). A priori codes derived from the original research questions and previous literature (specifically: barriers, facilitators, agency characteristics, system characteristics) were also applied (i.e., deductive approach). A random subset of transcripts (20%) was coded by two investigators, and inter-rater reliability was found to be excellent (K = .98; Landis & Koch, 1977).

Through an inductive process and consistent with the integrated approach described above (Bradley et al., 2007), two raters independently read through the barrier and facilitator codes to examine themes. Each reviewer produced memos including examples and commentary to reach consensus regarding newly-derived, emergent themes that emerged from the codes (Bradley et al., 2007). Upon further analysis of the codes, we found that the findings converged into three thematic categories that were consistent with our conceptual framework: outer context, inner context, and intervention characteristics (Aarons, Hurlburt, Horwitz, 2011).

Outer context factors refer to the service environment and inter-organizational environment. Inner context factors refer to intra-organizational and individual adopter characteristics. Intervention characteristics refer to factors refer to the intervention including intervention/system and intervention/organization fit; as well as the role of treatment developers (Green & Aarons, 2011). Percentages of individuals from each group who endorsed barriers and facilitator themes in each of the major categories are reported in Table 1. For parsimony, we report only on themes that 20% or more of stakeholders identified during the interviews (See Table 2).

Table 2.

Percentages of Individuals Who Endorsed Barriers and Facilitators by Stakeholder Group

	Treatment Developers (n = 7)	Agency Administrators (n = 33)	System leadership (n = 16)
Barriers
Outer Context	57%	66%	88%
Inner Context	100%	94%	94%
Intervention Characteristics	57%	42%	13%
Facilitators
Outer Context	57%	36%	88%
Inner Context	71%	88%	75%
Intervention Characteristics	71%	79%	75%

Results

Outer Context Barriers

System-level

Agency administrators, system leaders, and treatment developers agreed that system demands were a barrier to EBP implementation (See Appendix 2 for illustrative quotes). Agency administrators and system leaders noted that many county and state requirements created a stressful environment due to paperwork burden, which was further complicated by the added complexity of EBP implementation. Agency administrators and system leaders noted that lack of alignment among system-level stakeholders led to confusion; agency leaders reported receiving mixed messages. For example, agency leaders perceived that system leadership greatly valued EBPs, yet system compliance and credentialing bodies appeared unaware of the EBP initiatives. Agency administrators and system leaders agreed that the EBP implementation approach felt punitive, mandatory, and “top down” (from the system) which removed autonomy from agency stakeholders. This was perceived as a suboptimal way to implement EBPs. All three stakeholder groups agreed that poor communication between the system administrators and agencies was problematic. Specifically, they perceived that the system did not plan adequately for implementation and that minimal guidance and communication were provided about expectations and requirements of EBP initiatives. Unlike other stakeholder groups, system leaders uniquely expressed concern and curiosity about the system’s return-on-investment and EBP penetration rates given the expense of the initiatives. Further, only system leaders discussed the importance of understanding the effect of EBP implementation on client outcomes and workforce development system-wide, specifically, high staff turnover and poor pre-service training.

Inner Context Barriers

Agency-level

Agency administrators, system leaders, and treatment developers all agreed that EBP implementation and sustainability were impeded by intra-organizational factors such as lack of resources and competing demands. Agency stakeholders reported that staff have many responsibilities and that EBPs represent an addition without any subtraction of duties. Logistical challenges to implementation noted included lack of skilled supervision or trained supervisors, space constraints, high caseloads, and a dearth of equipment, technology infrastructure, and support for outcome monitoring. All three stakeholder groups agreed that staff turnover within agencies was the biggest challenge to sustainability. Differing attributions were made regarding why staff left, including increased marketability with new EBP skills and leaving the field due to burnout.

Compared to other stakeholders, system leaders noted that some agencies lacked the infrastructure needed to implement EBPs, particularly supervision, and did not take the time to fully understand what it would take logistically and financially for successful implementation and sustainability of a new EBP. Several treatment developers uniquely noted that agency administrators played a key role in implementation and that problems arose when leadership was not on board with the implementation process, or turned over as the initiative was ongoing.

Financing

Agency administrators and system leaders spoke about financial barriers to EBP implementation. They noted that the extra time required for EBPs creates significant financial strain for agencies. However, both system-leaders and agency administrators noted that this financial strain could be offset by the financial value of increased patient referrals, engagement, and retention resulting from practicing an EBP. All three stakeholder groups reported challenges of the independent contractor staffing model, noting that independent contracts are only paid for face-to-face client contact and cannot be mandated to attend trainings or supervision. Agency stakeholders uniquely reported being less likely to invest in independent contractors because they are highly transient.

Agency administrators uniquely identified the financial struggles of CMHCs as a major barrier. Many noted that revenue from outpatient services, the most common level of care in which EBPs are implemented, does not cover the operating costs of delivering these services, and that the investment required to successfully implement an EBP was almost insurmountable given their operating deficits.

Therapist-level

Only agency administrators and treatment developers reported therapist-level barriers to implementation of EBPs. Staff resistance was cited as a barrier by agency administrators who noted the challenges inherent in asking staff with allegiance to a specific therapeutic modality to change the type of treatment they delivered. Further, they opined that there was a sense of suspicion about EBPs. This was echoed by treatment developers who noted that clinicians resisted collecting client outcome data, believed that their clients are different from those participating in research trials, and did not have “EBP-oriented attitudes.” Treatment developers also said it was challenging to work with staff who did not have advanced degrees because the treatment models assume a certain level of foundational skills.

Intervention Characteristics Barriers

Intervention-population fit

Only agency administrators endorsed the fit between their client population and EBPs as a barrier. Specifically, agency administrators noted that they did not have clients that fit the criteria of the specific EBP and that clients who presented with additional complex psychosocial issues (e.g., homelessness, drug addiction, and chronic community violence) made it difficult to implement EBPs.

Outer Context Facilitators

System-level

Only system leaders mentioned system-level facilitators to the implementation of EBPs. The majority of system leaders described system collaborations with treatment developers and provider agencies as essential to EBP implementation. Inter-agency learning collaboratives and meetings facilitated by the system were lauded, as was system leadership for the foresight to develop collaborations with treatment developers and connect them with CMHCs. System leaders highlighted the development of an overall strategy to promote EBP. They noted that infrastructure was created within the system to support EBPs, specifically an EBP department to centralize and coordinate EBP efforts as well as system-level coordinating personnel dedicated to particular EBP initiatives. A number of system leaders stated that “buy-in” from system leadership facilitated EBP implementation and paved the way for system-wide implementation of EBP.

Financing

System-leaders and treatment developers championed the financial support of the system as the greatest system-level financial facilitator. This system investment included: payment for training through initiatives, funding of technical support through system personnel, reimbursement for lost time due to training, and in the case of one particular treatment, an enhanced reimbursement rate. System leaders believed that agencies would be largely unable to acquire costly training in EBPs without system support. System-level financing was not highlighted by the agency leaders.

Inner Context Facilitators

Agency-level

All three stakeholder groups agreed that intra-organizational factors facilitated implementation, particularly agency buy-in. Agencies committed to successful implementation of EBPs were enthusiastic and had clear champions. Agency leadership and dedication to the EBP resulted in providing the resources necessary for successful implementation, including training materials, EBP–focused supervision, and protected time (e.g., for reading and supervision). Stakeholders noted that successful agencies changed their culture to fit the needs of the EBP and integrated it into their daily operations. For example, in these agencies, specific EBP language (i.e., cognitive restructuring) became the common lexicon across the agency. Additional EBP-devoted supervision sessions were phased out because all supervision was in the service of supporting the practice of the EBP. Agency administrators and treatment developers also reported that EBP implementation was most effective if being an agency that provides “the best” care (via an EBP) was a deeply held value related to the agency’s mission.

Agency administrators highlighted intra-agency teamwork as critical to EBP implementation. Communication between leadership and clinical staff and camaraderie around providing the EBP to their clients despite multiple challenges were frequently mentioned as enabling successful implementation. A significant minority of agency administrators believed that having an administrator with clinical training was essential because s/he “gets it.”

Financing

Financial facilitators were raised by agency administrators and treatment developers albeit in different manners. A minority of agency administrators believed that EBPs are a source of financial support for their agency because improved services lead to more engaged consumers (i.e., lower no-show rates), happier and empowered clinicians (i.e., less turnover), and more referrals both within that service and beyond. Treatment developers were also confident that agency financial support was an essential facilitator and that agencies with successful implementation had paid clinicians for training, supervision, and meeting time, and had reduced productivity requirements. Several treatment developers noted that working with salaried therapists facilitated implementation.

Therapist-level

Parallel to agency buy-in, all three stakeholder groups echoed the importance of therapist buy-in as a facilitator. There was a sense that motivated and enthusiastic staff who are “hungry to learn” and willing to attend trainings greatly facilitated efforts. These clinicians were also described as “empowered” and satisfied with their EBP learning and the positive results they achieved with clients.

Intervention Characteristics Facilitators

Support of the treatment developer team

Agency administrators and treatment developers noted the importance of the support of the treatment developer team. Many provider stakeholders reflected that the structure of formalized training enhanced learning of the treatment. Agency leaders commented on the enthusiasm and availability of the treatment development staff as a facilitator to implementation. Treatment developers echoed this statement, emphasizing that their support played a critical role in the implementation process.

Benefits to consumers

Agency administrators noted that the EBP they implemented improved clinical care in their agencies and highlighted the fit between their client population and the EBP was a primary facilitator. The EBPs relevance to their particular population was seen as an enhancement to clinical care delivered, due not only to the therapists’ newfound tools and efficacy with this population, but also the clients’ beliefs that the therapists understood them. When therapists and providers observed clients improving, it enhanced buy-in and therefore implementation.

Discussion

This study elucidates the shared perspectives of stakeholders involved in the implementation of EBPs in a large publicly funded mental health system. Understanding implementation from the perspectives of multiple stakeholder groups has been identified as a foundational component of partnered research (Chambers & Azrin, 2013). We found that barriers and facilitators clustered around inner context factors, outer context factors, and intervention characteristics, consistent with leading implementation frameworks (Aarons, Hurlburt, Horwitz, 2011; Damschroder et al., 2009), and specifically the EPIS framework which guided our research questions (Aarons, Hurlburt, Horwitz, 2011). Identified barriers and facilitators were consistent across stakeholder group, lending confidence to the results. We noted relatively few points of divergence across the stakeholder groups; not surprisingly stakeholders were most likely to endorse barriers and facilitators in their sphere (e.g., treatment developers are more likely to endorse intervention characteristics) and while all stakeholders agreed finance was a major facilitator and barrier, they did not always agree on the cause or solution (Stewart et al., in press). Perhaps the most interesting omission, rather than point of divergence, was the lack of discussion of system-level facilitators, endorsed by the majority of system leaders and treatment developers, but not identified from the perspective of agency administrators. Nonetheless, the overwhelming converging message from stakeholders was that implementing EBPs is complex, and requires coordination, cooperation, and communication within the agency and between the agency and the system. There was also an important warning about the financial realities of EBP implementation in CMHCs.

A delicate dance was described by stakeholders around the collaboration and coordination needed across the system for implementation to be successful. This finding is corroborated by previous research, which has found that intra-organization synergy is necessary for successful implementation (Rapp et al., 2010), but this study extends beyond the organization to include all key stakeholders within a behavioral health system. Our results suggest that synergy, while important, is a complex process that involves collaboration, negotiation, planfulness, and resolution of stakeholder interests and preferences (Aarons et al., 2014). Future research that provides a better understanding of how to build cross-stakeholder teams within a system is warranted. This work could both characterize the function of teams that span organizations and whole systems, as well as delineate how to improve team functioning (Kutash et al., 2014). The Interagency Collaborative Team model, an implementation strategy to support implementation through increasing coordination between organizations while building system-based infrastructure to address implementation challenges (e.g., fidelity support and coaching) is also a potential area for future research (Hurlburt et al., 2014).

Despite the promising narrative about coordination and support across stakeholder groups, an important theme emerged relating to the financial challenges of CMHCs as a clear threat to the implementation and sustainability of EBPs. A number of recent papers have identified funding as a critical lever to the implementation of EBPs in systems (Bond et al., 2014; Green & Aarons, 2011; Isett et al., 2007; Willging, Green, Gunderson, Chaffin, & Aarons, 2015), and it is not surprising that stakeholders from all three groups raised financing as a barrier. It is important to provide the context in which these financial concerns were raised. The City of Philadelphia allocates a large yearly budget towards supporting agencies in implementing EBPs. This payment covers training and internal coordinator positions at DBHIDS dedicated to each implementation effort. In some cases the system reimburses for lost time, pays for ongoing consultation, and an enhanced reimbursement rate. Yet, this has still not been enough from the perspective of agency stakeholders in covering the losses in revenue associated with implementation. As noted above, agency stakeholders were the only stakeholder group that did not mention the systemic financial support as a facilitator to the implementation of EBP, which may be due to agency stakeholders experience of the lack of resources within the system despite generous financial allocations from DBHIDS. Literature from the organizational management area suggests that scarcity of resources result in organizational rigidity which may have resulted in agency administrators overlooking system-level facilitators (Staw, Sandelands, & Dutton, 1981). This is relevant given that, agency administrators identified a more ominous and pervasive issue that overshadows this important question of how to best implement EBPs – the dire fiscal situation of community mental health. There was an overwhelming sense of hopelessness about the future of CMHCs, particularly outpatient services, and the ability to continue operations. Considering this from the perspective of an agency’s hierarchy of needs, (Maslow, 1943) it seems that adding complex EBPs to fiscally challenged organizations may be a recipe for implementation failure (Aarons, Sommerfeld, & Willging, 2011). Future work investigating the impact of improving the fiscal effectiveness of CMHCs in concert with implementation efforts, as has been done in New York State, is an encouraging avenue for research (Lloyd, 1998). Additionally, questions around sequencing are paramount such as: is it necessary to improve agency fiscal health occur prior to implementation or can implementation of EBPs improve agency functioning?

Perhaps in response to dire financial straits, the three groups of stakeholders endorsed their concerns about an increasingly common shift to using independent contractors in CMHCs as a threat to the implementation and sustainment of EBPs. Independent contractors present unique challenges to implementation and sustainment because they are not employees; preliminary research suggests that they may have less positive attitudes towards and knowledge of EBPs (Beidas et al., 2015). This is likely due to the fact that organizations are less likely to invest in independent contractors attending professional development opportunities because they are not employees and are perceived as transient. There is a dearth of literature on the impact of the relying on independent contractors on both therapist and consumer outcomes, and future research would be well poised to better understand this phenomenon particularly in light of the Affordable Care Act (ACA) which may impact traditionally structured mental health services (Croft & Parish, 2013). Further, taking into account this workforce issue is paramount in future initiatives implementing EBPs in publicly-funded mental health agencies when structuring training and ongoing consultation and when considering staff selection for participation in EBP initiatives.

Surprisingly, the three groups of stakeholders did not raise intervention characteristics as a barrier to implementation as frequently as inner and outer context factors (although they did raise it as a facilitator). This finding diverges from other literature documenting the impact of intervention characteristics (Ringle et al., 2015) and intervention-setting fit (Damschroder et al., 2009; Green & Aarons, 2011; Lyon et al., 2014; Sackett et al., 1996). However, this may have to do with who was queried about implementation barriers; as well as our specific focus on inner and outer context factors in the interview guide. If therapists had been included in our sample, they may have raised issues relating to intervention characteristics (Aarons et al., 2009). The stakeholders we interviewed likely had a perspective less proximal to the client-treatment-therapist interaction and were more focused on inner and outer context factors. However, intervention characteristics were seen as facilitators to the implementation process and included the support of the treatment development team and consumer benefit.

This current study contributes to the existing literature by examining stakeholder perspectives of the implementation process, and extends previous research by understanding the implementation process from multiple stakeholders involved in implementing multiple EBPs in multiple care settings. The themes identified are likely influenced by the implementation approach used by DBHIDS. However, it is not without its limitations. First, we only included the perspectives of three stakeholder groups; it would have been interesting to include the perspective of other stakeholders including therapists and consumers (Aarons et al., 2009), or alternatively agency and program directors given findings that their perspectives may diverge (Palinkas et al., 2014). The stakeholders that we interviewed were not best suited to provide information on therapist-level barriers and may explain the lack of findings around the intervention barriers; although this is also a strength of our study given that the included stakeholders have been seldom interviewed in previous studies. Second, the results are specific to one publicly-funded mental health system, although findings are largely convergent with results from a previous study (Aarons et al., 2009). Third, we did not systematically characterize what stage of implementation participating agencies were in (i.e., exploration, preparation, implementation, and sustainment), thus we are unable to draw conclusions about specific themes that might emerge based on stage of implementation. Strengths of the study include the perspective of multiple stakeholders, the use of qualitative methods to generate nuanced information about the implementation process, and the opportunity to evaluate the impact of a naturalistic experiment at the system-level in the City of Philadelphia.

Conclusion

The current study provides support for the importance of examining implementation from the perspective of multiple stakeholders and converges with previous research (Aarons et al., 2009; Green & Aarons, 2011). Although findings suggest that there is much work to be done to successfully implement EBPs in publically funded mental health clinics, we believe that a number of insights emerged which can inform future successful efforts.

First, and most importantly, the findings suggest that coordinated collaboration is needed across stakeholder groups to successfully implement EBPs from the very beginning of the implementation process. This lends support to the importance of multi-level implementation strategies (Powell et al., 2012) that emphasize and enhance collaboration of all stakeholders involved in implementation across the system across the four phases of implementation (i.e., exploration, preparation, implementation and sustainment). Specifically, implementation strategies focusing on communication between the numerous stakeholders involved in implementation to foster collaboration and clear expectations as part of organizational readiness hold promise (Scaccia et al., 2015). Such implementation strategies merit development and validation in future implementation trials.

Second, two workforce issues were raised which relate to the design of future initiatives to implement EBPs. Treatment developers noted difficulty in working with less educated staff because of a lack of foundational skills brought to the training process. Similarly, therapist attitudes and buy-in were noted as a barrier to implementation. Therapist readiness to learn and deliver new practices is an important factor to address and has implications for future efforts to implement EBPs. Treatment developers will need to adapt their training and consultation process to be appropriate and acceptable for the therapists and consumer populations they are working with. Additionally, it may be necessary to assess therapists for baseline knowledge, foundational competencies, and attitudes prior to participating in EBP initiatives to ensure a certain level of knowledge to build upon and to address any maladaptive attitudes prior to participation (Casper, 2008). Finally, agency administrators may also need to be planful when considering whom to select to participate in EBP initiatives and also whom to hire if they want to emphasize use of EBPs.

Third, this study adds to the literature on potential threats to implementation and sustainability including the fiscal landscape of CMHCs (Aarons et al., 2009; Green & Aarons, 2011; Isett et al., 2007; Willging et al., 2015) and a changing workforce (Hoge & Morris, 2004; Hoge et al., 2009). These matters relate to the basic infrastructure of publicly funded community behavioral health which should be addressed to ensure that CMHCs have their basic needs met before layering on complex EBPs. While outcome (or value-based) reimbursement is in its infancy in behavioral health, future implementation efforts should pilot promising innovative and creative financial structures, drawing from both medicine, health care financing research, behavioral economics, or even the private sector (i.e., social impact bonds; Trupin, Weiss, & Kerns, 2014).

Thoughtful planning and self-assessment prior to participating in implementation initiatives are highly recommended; more research on how to engage in this process is necessary. Furthermore, providing support to organizations around better business practices; marketing and financing of EBPs; and leadership and strategic climate (Aarons, Ehrhart, Farahnak, & Hurlburt, 2015; Stewart et al., in press) are other potential future directions that merits careful consideration.

Appendix A Stakeholder perspectives of EBP initiatives in Philadelphia: Qualitative Interview for Agency Stakeholders

Instructions

The Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) has sponsored several Evidence-Based Training Initiatives in recent years. Evidence-based practices, or EBP, refer to integrating “the best available research with clinical expertise in the context of patient characteristics, culture and preferences.”

We are interested in understanding your experience with DBHIDS-sponsored EBP initiatives. You are on the front line and we believe you have important information to share with us due to your role in the agency. We are particularly interested in understanding agency- and system-wide factors that impact your ability to use evidence-based practices. When we say agency-wide, we mean factors specific to your agency. When we say system-wide, we mean factors specific across agencies within the Philadelphia public community mental health system.

Let’s start by discussing the initiatives your agency has participated in.

• 1Tell us briefly about your experience with these initiatives and your responsibilities in implementing these initiatives in your agency.

When did you take on this responsibility (e.g., prior to starting, mid-way through)?

• 2What was it about [initiative] that motivated or drove your agency to adopt it?*
• 3Generally, what makes these initiatives successful and not successful at your agency? Compare and contrast if you have participated in more than one initiative.

What would you change? What could be done to make these initiatives more responsive, more relevant, and less burdensome?

Let’s talk specifically about your agency now.

• 1What kind of infrastructure is needed at the agency-level to support these EBPs?
• 2What is your perspective on the impact of these initiatives on your agency?

Is there a sense at the agency level that this EBP is clinically better (i.e., results in better outcomes) than what was being done before with clients?

• 3Tell me about agency support at all levels around these various initiatives.

Be sure to get information about how all management levels including administrators support EBPs. Who at the agency is critical to making the EBP a success? When we say support, we mean reducing productivity requirements, problem-solving, supervision, making it easy to learn the practice.

Is the EBP integrated into regular supervision, additional supervision / meeting time documentation, intake, screening or ongoing training provided by the agency?

• 4What is the plan in your agency to continue the use of EBPs for the long term?

What is the single greatest barrier or facilitator to long term sustainment of any EBP? What type of infrastructure would be needed to maintain these EBPs?

• 5Tell me about the financial realities of implementing EBPs in your agency.

What costs are you incurring to implement EBPs? What is the financial value to having EBPs at your agency?

Are your therapists fee-for-service or salaried? How does that impact implementation?

How and what kind of financial incentives would help you increase use of EBPs in your agency? Does the potential for an enhanced rate or “preferred provider” designation make these initiatives more feasible?

ONLY FOR AGENCIES PARTICIPATING IN TF-CBT: Tell me about your experience with the enhanced rate provided for TF-CBT.

Tell me about the System

• 1What kind of infrastructure is needed at the system-level to support these EBPs?
• 2What is your perspective on the impact of these initiatives on the system (i.e., across agencies in Philadelphia)?

Across agencies, not just their agency. Are single initiatives perceived as having an overall impact of moving the system towards EBPs

• 3Tell me about system-level support of these initiatives.

Be sure to get information about how/if the system supports EBPs. Who at the system-level is critical to making the EBP a success?

• 4What do you think is important for policy-makers to know/understand about how to support the implementation of EBPs through these initiatives or other efforts moving forward at a system-level?

Appendix B

Illustrative Quotes

Theme	Stakeholder	Illustrative Quotes
Outer Context Barriers
System
System Demands	Agency Administrators	Everything felt so rushed all the time. We [system] need this now. We need this now. We need this now. It wasn’t things that we could get easily. You know? Like how many sessions’ people had. Or how many training hours people had? If people are in session you can’t get it right away…
		I think that it probably comes from a good place, but it doesn’t feel very supportive. It feels very, kind of, heavy handed. And this is the way that you need to do things because we don’t trust the way you’re doing them now. We don’t trust that it’s being good…But, I think that system-wide policymakers, I think, should look for ways to be more collaborative and less prescriptive.
	System Leaders	Unfortunately our system isn’t great at system wide needs assessment, I would like for the initiatives to be in response to a defined need. So understanding the needs of our population and making sure we’re responding to them well.
		I think the other would be a collaborative process in thinking about what the implementation strategy would be with our providers. So in most of the cases our treatment developers had a strategy in mind for what the best way to train our providers would be and how that’s supposed to look. So doing these very intensive training period and then doing all this consultation–we were all trying to respond to the research reality that shows a workshop is not enough, and put something in place that was more intensive, but perhaps we were not customizing that to the environment of a community setting as well as we could have.
	Treatment Developers	I don’t know what the expectations are [from the city]. I’ve never had an opportunity to communicate with another EBP that’s doing anything in the city.
		It became clear that these agencies were not well suited [for EBP implementation], a couple of them were not well suited…and those issues could have been identified ahead of time if the [system] expectations had made clear.
Return on investment	System Leaders	If you think about our budget on an annual basis, when we are implementing a new EBP – we are spending all that year on training people who may not be seeing cases…We are paying the purveyor to do the training and the agencies for lost time in revenue. At the end of the year, we might ask “so how many people did you serve?” and the agency says “zero” or “thirty.” That is a tough thing to swallow if it is going to take that long for people to come onboard and be trained, and then the agency says “in the second year, 30% of those people left.”
Workforce issues	System Leaders	We have individuals working in our system who have not received adequate training prior to this. I feel that our training system undercuts the clinical training model that used to exist where you learned a lot of fact based information in school and then you went out there and had a very strong supervision in order to develop clinical skills. I think that that doesn’t exist anymore in our professional infrastructure, and so more needs to be put in place in the training side of things so that people come into our system more competent to deliver services, and I think that we do not have very high credentialing requirements for the people who are the professionals in our system, and so I think that we are with EBPs trying to make up for a lack of competence and quality in the workforce coming in.
Inner Context Barriers
Agency
Sustainability	Agency Administrators	When there was a turnover, maybe about two years ago, most of the clinicians who were trained, left. And, that’s one of the barriers, once people get credentials and get these tools, they end up, you know, progressing in their career.
	System Leaders	I think that functionally, the challenge is that when folks turnover, you tend to lose the people who’ve been trained, the people coming in behind them don’t necessarily receive the training because resources aren’t there anymore. And so, it starts to dissipate a bit.
	Treatment Developers	I think fee-for-service also has very high turnover rates which makes the incentive in training harder to get people behind because there’s a sense that people will leave and probably an accurate one. I also think that people who are fee-for-service, I mean, this sort of goes back to the turnover, but they’re less invested in the agency.
Agency Resources and Competing Demands	Agency Administrators	They (the therapists) also had to do the ten hour TF-CBT thing while also meeting their quota. It was just a lot more work for people that were already really stretched thin.
		We’ve made jokes about the fact that our [EBP] group has worked out of six different spaces in the two years that it’s been in existence. Because there’s not a specific group room designated. There is definitely a space issue.
	System Leaders	I think the barrier is the due diligence around it…not really doing the due diligence around what will that look like here and what it will look like without a particular set of providers. Individual providers allow the initiative to come in, have a big hurrah around it, have a lot of money into it in the beginning, and then trickle off.
	Treatment Developers	For the strugglers… there was no protected time. There was not time laid out for supervision and if anything, there’d be a communication overtly or subtly that folks were wasting time if they were coming to consultation.
		A lot of frustration and helplessness in the systems and a lot of challenging feelings about being controlled by insurance companies, by CBH, by DBH… instability. All those things that in an organization make it really difficult to implement significant change because they’re so much changing that you don’t have control over.
Supervision	Agency Administrators	One negative is that I’m clinically supervising people in something that I cannot clinically supervise them in because I’m learning at a parallel to them so this next cohort will be much better.
Administrator Issues	Treatment Developers	Some agencies had executive level turnover and philosophy and vision changed. And in one case, it appears that that philosophy and vision may actually have been a blissful thing, more supportive of the initiative. And in another case, it really made staying with the model much more difficult.
Financing
Time	Agency Administrators	The biggest limitation is…time and time. Time, time, time, money, money, money. As soon as it became an unsupported activity, that’s when the tension developed. You want us to do this, but you also want us to be seeing patients. Explain to us how to resolve that tension…Every time you turned around, the time demand seemed to increase a little bit. They (initiative leaders) wanted a little bit more, a little bit more, a little bit more, a little bit more. At some point, you have to turn back and say, ‘Guys, we actually, we have bills we have to pay.’ And as great as it is, to say that 14 out of 14 therapists are certified in the EBP, we’ll be out of business at this rate if we don’t get back to doing things that we need to do.
	System Leaders	If you are typically used to seeing clients every hour and it’s a 50 minute session and you implement a protocol where you need to be out several days training, that takes away from the time that the therapist is seeing client. Then once therapists are trained, a certain portion of their caseload needs to be dedicated to clients who have to be seen for hour and a half, all of that contributes to the decrease in the numbers that were projected will directly impact the agency’s bottom line of our outpatient. And given that so many clients don’t come in outpatient, agencies are scrambling, you know, once they begin to implement and see what the realities are.
		The challenge is sort of putting in the upfront cost to see the long term benefit. I think that that is a financial benefit to the agencies down the road. The system benefit we hope for is the less intensive services, but I think that EBPs increase the morale and cohesion for the staff, and they want to stay and that helps the ultimate financial reality of the agency. That is not something you see immediately and you have to be pretty innovative in your own thinking administratively to be able to support that and understand the benefit of it down the road.
Fee-for-Service Staffing (FFS)/Independent Contractor Model	Agency Administrators	But the truth of the matter is that they are contractors. We contract with them if they have the skills we need and if they’re interested in this work. They can terminate the contract; we can terminate the contract. Fee-for-service is not easy. We really want people to, who we pick for trainings, to go complete the training and then be able to provide the service to children or adults in Philadelphia. But with contractors, you never know.
	System Leaders	I am cautious when it comes to an agency having only fee-for-service clinicians because the turnover. Also, they’re overworked. Agencies give them a caseload of 60 or 70 and families. And therapists end up seeing one client after the other and not having the time to do case conceptualization and assessment.
	Treatment Developers	…it’s a real challenge when people are fee-for-service, partly because they’re only paid for the time they’re across from a client. So, not only do things like group supervision not get paid for but neither do any of the extra things like taking time to do case conceptualization or anything outside of the room. I think it de-incentivizes having people invest anything other than the 50 minutes they’re across from someone.
Bleak Financial Picture	Agency Administrators	I cannot tell you the financial stress of running outpatient because your financial goals were literally impossible, and you knew no matter what would do, you were going to be extremely under budget and looking at the faces of folks that are working so very hard.
Therapist
Resistance	Agency Administrators	It’s the stigma of Evidence Based Treatment. I know what I’m going to do with my client…don’t tell me what to do because you don’t know my clients…any kind of evidence based is seen as managed care coming in wanting to cut the dollars, taking the relationship out of therapy (i.e., a ten session model and then they’re out.
	Treatment Developers	The therapists have the attitude this isn’t a good fit for us or this doesn’t fit our clients.
Types of Staff	Treatment Developers	Another challenge was working with providers with a variety of different backgrounds…people with certificates and Bachelor’s degrees.
Intervention Characteristic Barriers
Intervention-Population Fit
Fit with Client Population	Agency Administrators	Our clientele here is very difficult…primarily our people deal with is stopping the use of substances, primarily opiates. They receive Methadone… however that’s not really recovery. Recovery is actually being off the Methadone and not using anything…I think that they (therapists) were torn between using techniques that were more developed and using really basic techniques with people that are still using needles in their arms, trying to get them to stop…for our clients, every day is a crisis for them.
Outer Context Facilitators
System
Collaboration	System Leaders	Having good collaborations with the treatment developers and the persons managing the EBPs.
Strategy and Infrastructure	System Leaders	The system has been helpful in that the strategy has started to develop around what system-side goals are for EBPs.
System Buy-In	System Leaders	The department buy-in has been key to both initiatives…our commissioner, from the get go, has been 100% behind the initiative and being very involved despite you know, his other responsibilities.
Financing
System Financial Support	System Leaders	This is my experience in Philadelphia - in comparison to other relatively large urban areas is that Philadelphia has made a significant commitment financially in terms of resources to EBPs. So, I think that it makes it less difficult for agencies to then move forward because, you know, part of the, you know, what agencies tend to say is, ‘We want to do it. We’re completely committed to it philosophically, otherwise, we’re just not resourced well enough to do it.’ And I think that in Philadelphia, in a number of cases, they are resourced at least well enough to get them off the ground and move forward.
	Treatment Developers	[The system financial support was] consistently staggering…
		The largest piece financially of what has made this go.
Inner Context Facilitators
Agency
Agency Buy-in	Agency Administrators	We should be doing it. This is really helpful and meaningful and something that ultimately benefits clients and supports the mission of the organization.
	System Leaders	I can say the agencies that have been the most successful with the [EBP], we saw not only early engagement and buy-in from the executive level, but continued throughout time. They really have championed this, has become a culture at the agency which is really nice to see.
	Treatment Developers	A culture that works well is one where this [the EBP] gets infused broadly.
Teamwork	Agency Administrators	So we’re a small program, so the whole clinical team went through this training together, including the director. We consider her a part of the clinical team. She’s clinically trained. So, having us go through it together created a team buy-in and then having her support from the beginning was very important in prioritizing it, and we really - that is a priority that we meet every week and that we look at how we’re utilizing that in our work here.
Administrators with Clinical Background	Agency Administrators	I think the clinical services director has to be trained and skilled themselves, and have gone through the training…I think that’s actually the critical person to keep it going. The energy, the commitment, as long as that one person pushes and pushes and keeps it going.
Financing
EBPs Source of Financial Support	Agency Administrators	This practice is going to be, in a way, a lifeblood for the agency. Because it has increased referrals, the number of referrals, that we receive. And it’s an incredible valuable marketing tool to be able to say that we practice evidence based practice.
Agency Financial Support	Treatment Developers	Agencies had to say that they would do institutional buy out for therapists’ time to do the self-study. But everybody - this was like a miracle. [Fellow treatment developer] and I almost fell on the floor. People had actually read the damn manual!
Salaried Therapists	Treatment Developers	Places where they are not fee-for-service but instead are getting paid through any other structure is a great indicator of somewhere where it will go well.
Therapist
Therapist Buy-in	Agency Administrators	I think the buy-in from the therapists is what makes it [the EBP implementation] successful.
	System Leaders	It’s helped empower them to be more effective clinicians to have this more effective program and so I think that their level of buy-in is really very strong.
	Treatment Developers	I’m always really struck by the things that as a group and on average that they say. So, I’m not talking about just specific therapists but themes I hear in those are people talking about, you know ‘I’ve been working with folks for x number of years, and where I didn’t see any change. And I started going to these new things and I started seeing change. So then, I start doing it with more people I’m working with.’ And as I hear that from multiple therapists and they hear it from each other, I see the enthusiasm group not just with that group of trainees, but I hear the enthusiasm start to build for people who are outside the initial group for more people to get on board, and we get, sort of, a ripple effect.
Intervention Characteristics Facilitators
Treatment
Treatment Developer Support	Agency Administrators	They [treatment development team] were always available.
	Treatment Developers	We’re pretty open. They can deploy us pretty much anywhere and we say ‘yes’ and figure out how to do it.
Intervention-Population Fit
Consumer Benefit	Agency Administrators	That people are really aware of what we do and how we do it and how it benefits their children and they can see results, so they’re really invested and that helps.
Fit with Client Population	Agency Administrators	We know there is a tremendous amount of trauma in the community and the individuals we serve and it’s a very unique kind of condition and therefore needs the kind of experience that can address the problem because we want to make sure that people [clients] feel they have comfortable access and that when they come in, they’re seeing someone who knows.