Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Feb 1.
Published in final edited form as: Psychol Serv. 2019 Mar 11;18(1):33–41. doi: 10.1037/ser0000335

Concept Mapping Study of Stakeholder Perceptions of Implementation of Cognitive-Behavioral Social Skills Training on Assertive Community Treatment Teams

David H Sommerfeld 1,2, Eric Granholm 1,3, Jason Holden 1,3, Chariz Seijo 1,2, Chelsea S Rapoport 1,2, Kim Mueser 4, Jeanean Naqvi 1,2, Gregory A Aarons 1,2,§
PMCID: PMC6778484  NIHMSID: NIHMS1013175  PMID: 30855157

Abstract

This study aimed to identify factors associated with implementation of Cognitive Behavioral Social Skills Training (CBSST) on Assertive Community Treatment teams in a large public sector behavioral health system. This study used Concept Mapping (a mixed-method approach) and involved a sample including diverse stakeholder participants, including patients, ACT team members, team leaders, organization leaders, and system leaders. We identified 14 distinct issues related to implementing CBSST on ACT teams: 1) CBSST Fit with ACT Structure; 2) CBSST Fit with ACT Process; 3) Provider Perceptions about CBSST; 4) Staff Pressures/Other Demands; 5) CBSST & ACT Synergy; 6) Client Characteristics; 7) Benefits of CBSST; 8) Coordination/Interaction among ACT Providers; 9) Gov./Regulatory Factors; 10) Integration of CBSST into ACT; 11) Training Support; 12) Training Resources; 13) Multi-Level Agency Leadership; 14) Provider Characteristics. Each of these dimensions were rated in regard to importance and changeability with the top five rated dimensions including effective training support, alignment of leadership across levels of the community-based organizations delivering services, perceived benefits of CBSST, CBSST and ACT synergy, and provider perceptions of CBSST. The most critical issues for CBSST implementation on ACT teams should be addressed in future studies. Implementation strategies that capitalize on enhancing leadership and organizational climate hold promise to address all of these issues.

Introduction

Severe mental illness (SMI) has drastic implications for daily functioning. Individuals with SMI often need substantial assistance in addressing everyday problems, such as maintaining housing, building social relationships, and regularly taking medication (Bond, Drake, Mueser, & Latimer, 2001). Many people with SMI require long-term care, in order to effectively live in a community. Assertive Community Treatment (ACT) is a well-established evidence-based practice (EBP) that supports individuals with SMI by reducing hospitalizations and securing stable housing (E. L. Granholm, McQuaid, & Holden, 2016). It offers a multidisciplinary team (e.g., nurse, social worker, psychiatrist, alcohol/drug specialist, etc.) to address the range of challenges faced by individuals with SMI. The ACT model allows for multiple care professionals to engage with clients over a long period of time, and allows clients to receive ongoing one-on-one attention and support (Bond et al., 2001). It has been found to successfully engage and retain clients, and has been widely accepted and disseminated throughout the United States and the world. Despite its merits, however, this model often has limited emphasis on therapeutic treatment, particularly related to improving social functioning (Burroughs & Somerville, 2013; Drake, Bond, & Essock, 2009). Evidence-based psychosocial interventions have been found to be valuable for adults with SMI (Barrowclough, Haddock, Tarrier, Moring, & Lewis, 2000; Dilk & Bond, 1996; Mueser, Deavers, Penn, & Cassisi, 2013). Therefore, the ACT model could benefit from implementation of evidence-based approaches focusing on improving patient social functioning and social skill development.

Cognitive-Behavioral Social Skills Training (CBSST) is an evidence-based psychotherapy intervention that combines cognitive behavioral therapy (CBT) and social skills training (SST) (E. L. Granholm et al., 2016). These combined psychotherapies are often unavailable in community mental health treatment settings that serve patients with SMI, due to lack of resources, time gaps or discontinuations of treatment, unaffordable costs of care, and insurance barriers and service systems not prioritizing the use of evidence-based practices (Aarons, Ehrhart, Farahnak, & Sklar, 2014; Lehman et al., 1998; Mojtabai et al., 2009; Torres-Gonzalez et al., 2014). Due to ACT’s prioritization of regular client interactions and low staff-to-client ratios, it provides an opportunity for implementing EBPs such as CBSST in a way that can benefit patient functioning and recovery.

Findings indicate, however, that EBP implementation into the ACT model may not be seamless. ACT teams often experience barriers to implementing new EBPs, such as high turnover rates and insufficient funds for training (Burroughs & Somerville, 2013; Salyers et al., 2010; Williams, 2008). Such challenges can serve to limit the effective integration of new EBPs into service structures such as ACT. Additionally, EBPs must fit well with the work culture of an organization. The field of implementation science has highlighted factors related to implementation of new practices into existing service delivery systems, as demonstrated through the Exploration, Planning, Implementation, and Sustainment (EPIS) framework. EPIS identifies structural characteristics such as system level policies (i.e., outer context) and organizational characteristics (i.e., inner context) as well as the process of implementation likely to be important for effectively implementing EBPs in organizations (Aarons, Hurlburt, & Horwitz, 2011). ACT and CBSST each have distinct guidelines for implementation fidelity. These fidelity requirements must align well in order to successfully implement CBSST into ACT. Furthermore, ACT team members must regard CBSST as a valuable and viable option for implementation into the ACT model. Inner context factors such as leadership and organizational culture and climate should be positioned to promote and support the implementation of EBPs, in order to ensure EBP sustainment (Aarons, Ehrhart, Farahnak, & Hurlburt, 2015; Aarons & Sawitzky, 2006). EPIS also stresses the importance of characteristics of the EBP to be implemented, the setting in which it is to be implemented, characteristics of clinicians, and the role of leadership and “embedding mechanisms” (i.e., actions leaders can take to support and communicate the importance of implementation).

It is important to understand the process of implementation and the factors that serve to facilitate or hinder the implementation and sustainment of EBPs in context such as ACT teams. To that end, the present study is the implementation portion of a Hybrid Type 1 study that includes both a test of the effectiveness of a clinical intervention while simultaneously examining factors that affect their potential for real-world dissemination and implementation (Curran, Bauer, Mittman, Pyne, & Stetler, 2012). Specifically, our overall study included a randomized clinical trial that assessed for improvements in client outcomes (e.g., improved functioning) due to clients receiving both CBSST and ACT treatment services (as compared to those receiving standard ACT services) and a related, but separate implementation study that examined stakeholder (e.g., clients, administrators, and providers) perspectives on the factors influencing CBSST implementation and use (E. Granholm et al., 2015).

In order to best present the range of stakeholder perspectives, this study employed concept mapping, a mixed-methods for data collection and analysis that allows for representation of perspectives of all participants, in this case clients, service providers, supervisors, administrators, and developers/trainers, in regard to factors related to CBSST on ACT teams levels (Green, Fettes, & Aarons, 2012; Trochim, Cook, & Setze, 1994). Concept mapping is particularly suited to implementation studies as it allows for a systematic approach to generation and analysis of feedback about the factors affecting CBSST implementation into ACT teams. Consistent with the EPIS framework, we hypothesized that implementation factors would be related to outer context (e.g., external policy and funding considerations), inner context issues (e.g., leadership capabilities, staff attitudes, and workload demands), and characteristics of the EBP to be implemented (e.g., ease of use, perceived benefits of use).

Methods

Participants

Our study target population comprised a diverse set of stakeholder participants involved in and/or affected by the CBSST into ACT implementation processes within two large, private, multi-service behavioral health agencies funded to provide ACT services by local government. A total of 87 participants from six different stakeholder groups participated in 14 structured concept mapping focus groups designed to identify relevant implementation factors: two ACT client groups (n=8), six ACT team service provider groups (n=54), three ACT team supervisor groups (n=11), one agency administrator group (n=5), one public sector administrator group (n=5), and one group for the integrated CBSST in ACT team developers/trainers (n=4). To encourage open and honest discussion, each group consisted of only participants from that stakeholder group (e.g., only ACT providers from a specific ACT team, only ACT team supervisors from a specific agency, etc.). All of the ACT providers and supervisors at the agencies participating the in CBSST + ACT effectiveness study were recruited for the focus groups to identify relevant CBSST implementation factors. Focus groups were held on-site at the respective agencies at times identified by the ACT teams to maximize staff participation.

Seventy-one of the 87 stakeholder participants completed at least one of the concept mapping sorting or rating activities discussed below. The average participant age was 37.9 (SD=10.4) and 74.6% (n=53) were female. The majority of the participants were ACT service providers (n=52; 73.2%), with the remaining 26.8% comprised of ACT team supervisors (n=6), agency administrators (n=6), ACT clients (n=2), public sector administrators (n=2), and CBSST in ACT team developers/trainers (n=3). Since all staff members at ACT sites were asked to participate in the concept mapping sorting and rating activities, feedback included perspectives from new hires as well as staff involved throughout the entire CBSST implementation process. For each stakeholder group, the mean number of months of experience with CBSST during this study were as follows: 15.5 (SD=9.4) for providers; 11.0 (SD=1.4) for clients; 16.6 (SD=20.1) for team supervisors; 41.0 (SD=12.2) for agency administrators; 18 (SD=8.5) for public sector administrators; and 44.0 (SD=6.9) for intervention developers/trainers.

The study protocol was reviewed and approved for the ethical treatment of human subjects by the Institutional Review Board (IRB) of the Veterans Affairs San Diego Healthcare System and the University of California, San Diego (San Diego, CA, USA). Informed consent was obtained from all participants. Ethical review and approval was also provided by the Research Committee of the local public sector behavioral health system.

Concept Mapping Approach

The CM process typically consists of six phases: 1) preparation – researchers identify stakeholder participants and develop a focus question in collaboration with stakeholder participants; 2) generation – stakeholder participants contribute statements in response to the focus question; 3) structuring – stakeholder participants sort statements based on similarity and rate statements on several a priori dimensions (e.g., importance, changeability); 4) representation – researchers conduct multidimensional scaling (MDS) and cluster analyses to create a “concept map” placing statements that were sorted together in closer proximity to each other; 5) interpretation – researchers and stakeholder participants develop cluster labels and interpretations; and 6) utilization – researchers and stakeholder participants use results to improve implementation and implementation study design. Below, we describe how this process was applied in the present study.

In the preparation phase, the study team collaboratively and iteratively developed a single focus question: “What are the factors that influenced the implementation and use of CBSST on the ACT teams?” The study team then conducted the 14 stakeholder focus groups discussed above during the generation phase. All stakeholder participants had firsthand experience either delivering or receiving CBSST in ACT or were in relevant supervision or leadership positions of behavioral health services during the implementation process. ACT provider and supervisor concept mapping focus groups were conducted after each ACT site had implemented CBSST for at least 7 months (range=7–17 months). Overall, the 87 participants in the 14 brainstorming sessions provided 934 statements regarding implementation factors. These were carefully reviewed for redundancy or similar meaning by three team members (GA, DS, JN) and consolidated into 88 distinct statements. In the structuring phase, 71 of the 87 stakeholder participants were asked to sort the 88 statements into separate groups (or “clusters”) in a manner that was meaningful to them, and using online tools from Concept Systems, Inc (Concept Systems, 2016). Participants individually suggested a name describing the meaning or general content of each cluster.

After sorting each statement, participants were asked to rate each statement (using a Likert scale) on multiple dimensions including importance: “How important is this factor for successful initial implementation of CBSST in ACT teams?” (0=not at all important; 7=very important); “How important is this factor for ongoing/sustained use of CBSST in ACT teams?” (0=not at all important; 7=very important); and changeability: “How easy would it be to change this factor?” (0=not very easy; 7=very easy).

Analysis

Statement sorting data were analyzed using multidimensional scaling and hierarchical cluster analysis within the CM program (Davison, 1983). These procedures resulted in the creation of visual representations (i.e., concept maps) for how statements were typically clustered across all participants. Multiple concept map outcomes were considered based on acceptable overall “stress” fit statistic and interpretability of each potential solution (Rosas & Kane, 2012). The ideal model would include the fewest number of clusters that also represented distinct key concepts. This process started by considering a large number of potential thematic clusters (e.g., 20) and then, in stepwise fashion, consolidating groups that were most thematically similar based on participant responses. These models were reviewed by the study team with the final concept map (14 thematic clusters) determined by a collaborative, consensus-based discussion with representatives from the ACT team agencies. Each cluster in the final model was reviewed by the study team and ACT team representatives and named according to the nature of the statements or content contained in each cluster.

The three different ratings of each implementation factor statement were used to calculate an average score, respectively, for the factors: 1) importance for initial CBSST implementation, 2) changeability, and 3) importance for CBSST sustainability on ACT teams. Following the identification of the 14 key thematic statement clusters, the rating of each thematic cluster for these three dimensions was determined by the average ratings of the individual statements contained within each thematic cluster. To identify thematic clusters that were typically rated as having high initial importance, high changeability, and high sustainment we created a rank cluster order for each dimension (with the highest cluster rating ranked as #1, second highest as #2, etc.), and then summed the three separate rank orders to calculate a total rank score for each cluster.

Results

The final model comprised 14 thematic clusters, each representing a key domain of CBSST implementation (Figure 1). These clusters were: CBSST Fit with ACT Structure; CBSST Fit with ACT Process; Provider Perceptions about CBSST; Staff Pressures/Other Demands; CBSST & ACT Synergy; Client Characteristics; Benefits of CBSST; Coordination/Interaction among ACT Providers; Gov./Regulatory Factors; Integration of CBSST into ACT; Training Support; Training Resources; Multi-Level Agency Leadership; Provider Characteristics. Each dot within a cluster represents a statement that was sorted into similar categories by focus group participants. Lower stress index values indicate better fit between the concept map and the initial statement sorting data. The stress value is a measure of how well the MDS point-map represents the original data. The value should range from 0.10 to 0.35, with lower values indicating a better fit (Concept Systems, 2016). When the MDS does not fit the original data (i.e., the stress value is too high), it means that the distances of statements on the point map are more discrepant from the values in the similarity matrix derived from the sorting activity (Kane & Trochim, 2007). The stress value of .32 for our analyses indicated acceptable model fit. Table 1 provides a listing of each of the 14 identified thematic clusters, a general definition of what each cluster represents and three example individual implementation factor statement included in each cluster.

Figure 1.

Figure 1.

Open in a new tab

Fourteen cluster thematic concept mapping solution

Table 1.

Description of the fourteen cluster themes impacting CBSST implementation on ACT teams

Cluster name Cluster definition Example statements included in cluster
1. CBSST Fit with ACT Structure The extent to which CBSST is appropriate to implement within the organizational requirements of ACT teams. Disruptions to consistent visits with ACT clients (e.g., hospitalizations)
Providing CBSST in settings with distractions (e.g., public places)
Nature of ACT visits (e.g., going to doctor appts.) can inhibit provision of CBSST
2. CBSST Fit with ACT Process The extent to which CBSST is appropriate to implement with required procedures for ACT teams. Frequency/intensity of other required, non-CBSST related service delivery changes
ACT provider’s ability to apply CBSST flexibly/“on the fly” during regular visits
Having sufficient time in ACT visits to go as “deep” as needed for CBSST
3. Provider Perceptions about CBSST ACT providers’ beliefs about CBSST. Provider beliefs about whether CBSST improves client outcomes
Provider perception of usefulness/relevance of CBSST skills to other interventions
Perceived burden of delivering CBSST
4. Staff Pressures/Other Demands Existing requirements/ burdens on ACT teams that affect CBSST implementation. Understaffed/overworked ACT teams
Additional administrative demands (e.g., documentation) needed to deliver CBSST
Impact on provider productivity requirements
5. CBSST & ACT Synergy The extent to which CBSST complements and improves ACT process and structure. Flexibility to adapt CBSST while maintaining fidelity
CBSST structure/content makes ACT visits feel more purposeful
Creation of shared expectations for both ACT clients and providers
6. Client Characteristics Elements related to the ACT clients that are perceived to impact CBSST implementation. Relevance of CBSST concepts/ideas to ACT clients
Openness of ACT clients to structured material (e.g., workbooks, homework)
ACT client motivation/buy-in to do CBSST
7. Benefits of CBSST Direct or indirect experience with positive outcomes attributed to CBSST. Ability of CBSST to empower/increase self-esteem of ACT clients
Receiving positive feedback from ACT clients about CBSST
ACT client progress setting/accomplishing goals due to CBSST
8. Coordination/ Interaction among ACT Providers Managing the team-based approach of ACT to provide CBSST. ACT providers having tools to track/monitor client progress through CBSST sessions
Seeing other ACT providers’ success delivering CBSST
Effectiveness of systems/processes to communicate client CBSST information among ACT providers
9. Gov./Regulatory Factors External rules, guidelines and resources that can be used to support CBSST implementation. Data systems to measure and report meaningful CBSST outcomes
Available funding to deliver CBSST
Recognition of CBSST by government/public sector as an effective treatment approach
10. Integration of CBSST into ACT The extent to which CBSST can be successfully implemented with ACT requirements. Compatibility of CBSST with other contractually mandated EBPs
Funding source willingness to adjust requirements (productivity, case-load, time-frames)
Staff CBSST “champion” to encourage use of CBSST
11. Training Support Initial and ongoing training support for and monitoring of CBSST implementation. Substantial time allotted for practice and feedback during training/supervision for ACT providers
Having someone with organizational power monitor implementation of CBSST
Providing incentives for ACT provider use of CBSST
12. Training Resources Materials and tools available to ACT teams to support CBSST implementation. Access to CBSST information/trainings (e.g., online materials)
Availability of on-going training opportunities for ACT providers
ACT provider access to an “on-call” CBSST trainer for emerging questions
13. Multi-Level Agency Leadership The impact of multiple levels of leadership on CBSST implementation within ACT teams. Prioritization level of CBSST by agency
Communication about CBSST importance from ACT team supervisors/leads
Agency leadership buy-in of/support for CBSST
14. Provider Characteristics Elements related to ACT providers impacting CBSST implementation. Provider openness to try new things
Provider confidence delivering CBSST
The level of ACT provider “buy-in” or enthusiasm for CBSST
Open in a new tab

Table 2 shows the mean participant ratings for each of the identified 14 thematic clusters (as determined by the average ratings of the implementation factor statements included in each thematic cluster) and the relative rank order of the thematic cluster (i.e., #1 through #14, highest to lowest) for the ratings of 1) importance for initial CBSST implementation, 2) changeability of the implementation factor, and 3) importance for CBSST sustainment. The “importance” ratings for initial implementation of CBSST ranged from a low of 4.46 (Gov./Regulatory Factors) to a high of 5.29 (Training Support). While the range of scores was somewhat restricted, this demonstrates a relative higher rating for the need for ongoing training support relative to change government/regulatory factors. All clusters were ranked substantially above the scale midpoint of 3.5 suggesting moderate to strong importance for CBSST implementation with some variability across thematic clusters. Similarly, the “importance” cluster ratings for sustainment ranged from 4.42 (Gov./Regulatory Factors) to 5.38 (Benefits of CBSST) indicating moderate to strong importance attending to the benefits of CBSST in supporting sustainment of the practice. Cluster ratings for the changeability of implementation factors were generally perceived as moderately changeable with the lowest changeability cluster rating of 3.31 (Staff Pressures/Other Demands) and the highest of 4.68 (Training Support). This suggests that training may be a higher priority relative to competing demands on ACT teams.

Table 2.

Cluster rating averages and ranks for CBSST implementation on ACT teamsa

Importance for initial CBSST implementation Changeability of implementation factor Importance for sustainability Total
Thematic cluster (# in Fig. 1) Cluster ratingb Cluster rank Cluster ratingb Cluster rank Cluster ratingb Cluster rank Rank score
Training support (#11) 5.29 1 4.68 1 5.15 3 5
Multi-level agency leadership (#13) 5.25 2 4.56 3 5.26 2 7
Benefits of CBSST (#7) 5.21 3 4.49 4 5.38 1 8
CBSST & ACT synergy (#5) 4.83 9 4.57 2 5.06 5 16
Provider perceptions about CBSST (#3) 4.97 7 4.34 6 5.10 4 17
Provider characteristics (#14) 5.04 5 4.28 8 4.89 8 21
Training resources (#12) 5.02 6 4.43 5 4.78 10 21
Coordination/interaction among ACT providers (#8) 4.93 8 4.32 7 4.96 7 22
Staff pressures/other demands (#4) 5.12 4 3.31 14 4.98 6 24
CBSST fit with ACT process (#2) 4.79 10 4.03 10 4.87 9 29
Client characteristics (#6) 4.78 11 3.79 11 4.72 11 33
Integration of CBSST into ACT (#10) 4.73 12 4.05 9 4.68 12 33
Fit with ACT structure (#1) 4.63 13 3.40 13 4.59 13 39
Gov./regulatory factors (#9) 4.46 14 3.67 12 4.42 14 40
Open in a new tab
a

CBSST, cognitive behavioral social skills training; ACT, assertive community treatment

b

Cluster ratings ranged from 0 to 7, with higher scores indicating more importance for initial implementation or sustainment or easier changeability

Table 2 provides a detailed cluster-by-cluster comparison for the ratings and relative ranking across these three dimensions. The results show a relatively strong congruence between the clusters determined to be among the most important for CBSST implementation and sustainment and the most changeable. The far right column shows the total rank score for each thematic cluster (i.e., sum of the rank order for each of the three rating dimensions). Three thematic clusters — Training Support (e.g., providers receiving specific feedback reports on their CBSST sessions), Multi-Level Agency Leadership (e.g., agency leadership buy-in of/support for CBSST), and Benefits of CBSST (e.g., receiving positive feedback from ACT clients about CBSST) — were ranked within the top four clusters for each dimension.

Discussion

We identified 14 domains impacting CBSST implementation using a mixed-method concept mapping approach. The approach represented the combined input from stakeholder participants including patients, clinicians, team leaders, agency executive, and behavioral health system administrators. There was substantial overlap among the most highly rated thematic clusters across the three rating dimensions of importance for implementation, changeability, and importance for sustainment. In keeping with the EPIS framework, the results suggest that it is important to consider sustainment and not just implementation as any EBP implementation effort begins (Aarons, Hurlburt, et al., 2011). Training Support, Multi-Level Agency Leadership, and Benefits of CBSST were rated within the top four clusters for “Importance for Initial CBSST Implementation,” “Changeability,” and “Importance for Sustainability.” This suggests that training, leadership, and the benefits of CBSST were among the most important for implementation and sustainment and perceived changeability. This is consistent with recent work focusing on the importance of leadership in shaping the context for EBP implementation (Aarons, Ehrhart, et al., 2014; Aarons, Farahnak, & Ehrhart, 2014). However, it is also critical to have alignment with behavioral health system priorities (Willging et al., 2016). These domains can be specifically targeted through strategies for leadership and organizational climate improvement to support EBP implementation at the system and organizational levels (Aarons, Ehrhart, Moullin, Torres, & Green, 2017; Richter et al., 2016).

Furthermore, increased leadership buy-in and support to create a climate for adoption of new EBPs and also to monitor the integration of distinct models (such as CBSST) may increase an organization’s investment in the implementation process. Leaders have a substantial role in creating an organizational culture and climate that is receptive to EBPs (Schein, 2010). Leaders can employ a variety of mechanisms to embed their beliefs into organizational culture and climate. These include methods such as deliberate modeling and allocation of resources. Leaders must be invested in an EBP, however, before embedding mechanisms become feasible. Therefore, leader buy-in is an essential aspect of ensuring implementation success. Researchers must flexibly adapt mechanisms to better fit specific systems and organizations, and to ultimately improve implementation outcomes. Future studies should also incorporate strong feedback models that are frequently dispersed throughout the EBP implementation. These feedback models can promote leader buy-in by measuring reactions and documenting feedback. This information can indicate what’s important to the organization, and influence the adaptation process. Similarly, rapid fidelity feedback to providers about the quality of intervention delivery, combined with suggestions for improvement, can improve training support, which received high ratings for implementation success and changeability. It is important to note that the domain Staff Pressures/Other Demands was rated relatively high on the importance dimensions but the least changeable of all 14 clusters. The perception that the pressures on staff are least likely to change despite being more important than many other domains highlights the importance of addressing demands placed on staff (e.g., understaffed teams, overworked staff). This has implications for how service systems, agencies, and programs serving individuals with SMI allocate resources and staff their programs. While job characteristics are associated with burnout reducing the load of non-patient-centered demands, such as documentation and other administrative duties, could help with allowing clinicians to focus more squarely on their engagement and interactions with patients (Rosenberg & Pace, 2006).

The CM approach was feasible and had high utility in this study. In structured groups, the CM approach can be beneficial for encouraging collaboration and discussion among stakeholder participants. For example, it is possible during brainstorming sessions to ask participants to consider a wide range of implementation strategies (e.g., training support), as well as their impact on and value to the implementation process. This participatory process can encourage buy-in from stakeholders at all levels.

The present study identified 14 domains to be addressed for implementation of CBSST on ACT teams and many of these are consistent with prior literature and recommendations for behavioral health system transformation (American Psychological Association & Jansen, 2014; Davidson et al., 2007; Rosenheck, 2001; Torrey et al., 2001). However, taken together, recommendations for may require tailoring to specific systems and organizations providing mental health services. The recommendations from the American Psychological Association are many and appropriately consider factors consistent with the EPIS framework for both the outer system and inner organizational contexts (Moullin, Dickson, Stadnick, Rabin, & Aarons, 2019). However, for each implementation effort, it is important to consider the local context and system and select specific strategies likely to be most relevant for that setting at that time (Powell et al., 2017). In addition, it is important to consider both organizational context and provider attitudes (Beidas et al., 2015), leadership impact on organizational climate (Aarons, Sommerfeld, & Willging, 2011), and also emergent considerations such as alignment of outer context policy initiatives and leadership with that of inner context of provider organizations (Lyon et al., 2018) and “bridging factors” (e.g., collaborative relationships) that connect outer and inner contexts to support more effective implementation (Moullin et al., 2019). Recommendations of the APA are an excellent starting point in identifying what factors might be malleable and most likely to enhance implementation efforts and approaches such as Concept Mapping can be useful in that process. In addition, local collaboration of relevant stakeholders should be used to further refine such factors and develop a comprehensive implementation strategy likely to enhance success.

Limitations

One limitation to the CM approach is that statement ratings across multiple dimensions can be time-consuming. In our study, each of the 88 distinct implementation factor statements were rated on three dimensions (i.e., initial importance, changeability, and ongoing/sustainment importance), which required a large number of responses (i.e., 3 × 88 = 264) for each participant. While each statement can be rated relatively quickly on the relevant Likert scale, the time needed to complete all ratings could be perceived as burdensome by some respondents (estimated at 15–30+ minutes depending upon the amount time spent considering each item). Having participants rate each determined cluster (e.g. 14 thematic clusters) instead of each individual statement (e.g. 88), multiple times could reduce this burden. Our hope is that concept mapping software developers and vendors will consider this suggested innovation. Another limitation of the study is that there were more ACT service provider participants relative to clients, supervisors, and agency and public sector service system administrators. However, while there were fewer non-provider participants, non-redundant statements generated by all participants were included in the CM sorting and rating activities.

Conclusions

Throughout the study, there were unexpected challenges that manifested during the process of real-world implementation. Aspects of CBSST that appeared to fit well on paper proved more challenging to integrate into the ACT model in practice. This integration proved especially challenging during client interactions, and during tracking and delivering the curriculum. For example, ACT is a shared caseload model where different providers meet with patients across visits, so providers needed to develop processes to share information about session content and homework assignments. The project also experienced standard implementation barriers in terms of leadership and in terms of stakeholders using a new practice. The established methods through which the team provided service, and the possibility of a culture shift, proved especially challenging to overcome. For example, providers needed to increase the frequency and duration of sessions with some clients to shift to a skills training rather than care coordination focus. The EPIS framework stresses that organizational structure plays a large role in the successful adoption of an unfamiliar practice (Aarons, Hurlburt, et al., 2011). Although researchers had demonstrated that the CBSST and ACT models fit well together (Sommerfeld et al., 2018), future steps can be taken to further facilitate a successful fit. For example, the study team should consider the timeline for an organization’s cultural shift, in terms of the services provided, to allow for EBPs to become integrated along that timeline. There were adaptations that improved the fit, such as information exchange adaptations for team and community delivery, flexibility while treating complex clients, and employing a flexible crisis management model. Despite these adaptations, however, stakeholder participants found that major challenges within their organization, such as overworked and understaffed ACT teams, would not be impacted by the implementation process. In this way, EBP implementation may have failed to address important issues within these organizations. These findings demonstrate that while the necessity of adaptation may be applicable to a practice, it also applies to systems and organizations themselves, so they can better accommodate the use of EBPs. Changes, such as decreased staff responsibilities, may need to be made in order for EBP implementation to be feasible.

CBSST implementation into ACT serves as a case example of integrating a specific EBP into an established treatment model. ACT is a unique treatment model because it utilizes cross-functional teams. ACT teams employ professionals from multiple disciplines, such as social work, psychiatry, and nursing, in order to provide clients with comprehensive treatment (Bond et al., 2001). This poses a challenge with maintaining intervention continuity. Team members must coordinate to ensure that they are successfully adhering to the CBSST curriculum. Furthermore, the CBSST curriculum may appear more relevant to some team members than to others. This may require an implementation strategy that demonstrates the applicability of CBSST for every team member, or targets specific team members who can specialize in CBSST implementation.

Regardless of the elements that distinguish and CBSST and ACT from other models, the insights and challenges of this study are relevant for the field of implementation science as a whole. Leadership buy-in, effective embedding mechanisms, flexibility, training supports, and adaptations to the practice and system can be beneficial for a variety of EBPs. Furthermore, concept mapping serves as an effective method of conveying what is important to stakeholders throughout an intervention. Future studies can use concept mapping to provide relevant stakeholder insight regarding elements of successful EBP implementation.

Acknowledgments

This study was supported by National Institute of Mental Health grants R01MH091057 and 5R01MH072961, and National Institute on Drug Abuse grant R01DA038466. We thank the community-based organizations, providers, and patients for their participation in this project.

Jason Holden receives author royalties from the book, Cognitive Behavioral Skills Training: A Practical Treatment Guide, on which this intervention is based and speaker fees for CBSST trainings. It is possible that findings from this study supporting CBSST could increase interest in CBSST training workshops and therefore income to Dr. Holden.

Dr. Granholm has an equity interest in Granholm Consulting, Inc., a company that may potentially benefit from the research results as he receives income from the company for CBSST workshops and consulting. The terms of this arrangement have been reviewed and approved by the University of California, San Diego in accordance with its conflict of interest policies.

Footnotes

The authors declare no other potential conflicts.

References

  1. Aarons GA, Ehrhart MG, Farahnak LR, & Hurlburt MS (2015). Leadership and organizational change for implementation (LOCI): A randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science, 10(1), 11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Aarons GA, Ehrhart MG, Farahnak LR, & Sklar M (2014). Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annual Review of Public Health, 35, 255–274. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Aarons GA, Ehrhart MG, Moullin JC, Torres EM, & Green AE (2017). Testing the Leadership and Organizational Change for Implementation (LOCI) Intervention in substance abuse treatment: A cluster randomized trial study protocol. Implementation Science, 12(1), 29. doi: 10.1186/s13012-014-0192-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Aarons GA, Farahnak LR, & Ehrhart MG (2014). Leadership and strategic organizational climate to support evidence-based practice implementation. In Beidas RS & Kendall PC (Eds.), Dissemination and implementation of evidence-based practices in child and adolescent mental health (pp. 82–97). New York, NY: Guilford Press. [Google Scholar]
  5. Aarons GA, Hurlburt M, & Horwitz SM (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. doi: 10.1007/s10488-010-0327-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Aarons GA, & Sawitzky AC (2006). Organizational climate partially mediates the effect of culture on work attitudes and staff turnover in mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 33(3), 289–301. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Aarons GA, Sommerfeld DH, & Willging CE (2011). The soft underbelly of system change: The role of leadership and organizational climate in turnover during statewide behavioral health reform. Psychological Services, 8(4), 269–281. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. American Psychological Association, & Jansen MA (2014). Reframing psychology for the emerging health care environment: Recovery curriculum for people with serious mental illnesses and behavioral health disorders: Washington, DC: American Psychological Association. [Google Scholar]
  9. Barrowclough C, Haddock G, Tarrier N, Moring J, & Lewis S (2000). Cognitive Behavioral Intervention for Individuals with Severe Mental Illness Who Have a Substance Misuse Problem. Psychiatric Rehabilitation Skills, 4(2), 216–233. doi: 10.1080/10973430008408407 [DOI] [Google Scholar]
  10. Beidas RS, Marcus S, Aarons GA, Hoagwood KE, Schoenwald S, Evans AC, . . . Mandell DS (2015). Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Pediatrics, 169(4), 374–382. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Bond GR, Drake RE, Mueser KT, & Latimer E (2001). Assertive community treatment for people with severe mental illness. Disease Management and Health Outcomes, 9(3), 141–159. doi: 10.2165/00115677-200109030-00003 [DOI] [Google Scholar]
  12. Burroughs T, & Somerville J (2013). Utilization of evidenced based dialectical behavioral therapy in assertive community treatment: Examining feasibility and challenges. Community Mental Health Journal, 49(1), 25–32. doi: 10.1007/s10597-012-9485-2 [DOI] [PubMed] [Google Scholar]
  13. Concept Systems I (2016). The Concept System® Global MAX™ (Build 2016.046.12) [Web-based Platform]. Ithaca, NY. Retrieved from http://www.conceptsystemsglobal.com [Google Scholar]
  14. Curran GM, Bauer M, Mittman B, Pyne JM, & Stetler C (2012). Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50(3), 217–226. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Davidson L, Tondora J, O’connell MJ, Kirk T Jr, Rockholz P, & Evans AC (2007). Creating a recovery-oriented system of behavioral health care: Moving from concept to reality. Psychiatric Rehabilitation Journal, 31(1), 23. [DOI] [PubMed] [Google Scholar]
  16. Davison ML (1983). Multidimensional scaling. New York: John Wiley & Sons, Inc. [Google Scholar]
  17. Dilk MN, & Bond GR (1996). Meta-analytic evaluation of skills training research for individuals with severe mental illness. Journal of Consulting and Clinical Psychology, 64(6), 1337–1346. doi: 10.1037/0022-006x.64.6.1337 [DOI] [PubMed] [Google Scholar]
  18. Drake RE, Bond GR, & Essock SM (2009). Implementing evidence-based practices for people with schizophrenia. Schizophrenia Bulletin, 35(4), 704–713. doi: 10.1093/schbul/sbp041 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Granholm E, Holden JL, Sommerfeld D, Rufener C, Perivoliotis D, Mueser K, & Aarons GA (2015). Enhancing assertive community treatment with cognitive behavioral social skills training for schizophrenia: Study protocol for a randomized controlled trial. Trials, 16(1), 438. doi: 10.1186/s13063-015-0967-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Granholm EL, McQuaid JR, & Holden JL (2016). Cognitive-behavioral social skills training for schizophrenia: A practical treatment guide. New York, NY: Guilford Publications, Inc. [Google Scholar]
  21. Green AE, Fettes DL, & Aarons GA (2012). A concept mapping approach to guide and understand dissemination and implementation. Journal of Behavioral Health Services & Research, 39(4), 362–373. doi: 10.1007/s11414-012-9291-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kane M, & Trochim W (2007). Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage Publications, Inc. [Google Scholar]
  23. Lehman AF, Steinwachs DM, Dixon LB, Postrado L, Scott JE, Fahey M, . . . Skinner EA (1998). Patterns of usual care for schizophrenia: Initial results from the schizophrenia Patient Outcomes Research Team (PORT) client survey. Schizophrenia Bulletin, 24(1), 11–20. [DOI] [PubMed] [Google Scholar]
  24. Lyon AR, Cook CR, Brown EC, Locke J, Davis C, Ehrhart M, & Aarons GA (2018). Assessing organizational implementation context in the education sector: Confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implementation Science, 13(1), 5. doi: 10.1186/s13012-017-0705-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Mojtabai R, Fochtmann L, Chang S, Kotov R, Craig TJ, & Bromet E (2009). Unmet need for mental health care in schizophrenia: An overview of literature and new data from a first-admission study. Schizophrenia Bulletin, 35(4), 679–695. doi: 10.1093/schbul/sbp045 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Moullin JC, Dickson KS, Stadnick NA, Rabin B, & Aarons GA (2019). Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Science, 14(1), 1. doi: 10.1186/s13012-018-0842-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Mueser KT, Deavers F, Penn DL, & Cassisi JE (2013). Psychosocial treatments for schizophrenia. Annual Review of Clinical Psychology, 9, 465–497. doi: 10.1146/annurev-clinpsy-050212-185620 [DOI] [PubMed] [Google Scholar]
  28. Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, & Mandell DS (2017). Methods to improve the selection and tailoring of implementation strategies. The Journal of Behavioral Health Services & Research, 44(2), 177–194. doi: 10.1007/s11414-015-9475-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Richter A, von Thiele Schwarz U, Lornudd C, Lundmark R, Mosson R, & Hasson H (2016). iLead—a transformational leadership intervention to train healthcare managers’ implementation leadership. Implementation Science, 11(1), 108. doi: 10.1186/s13012-016-0475-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Rosas SR, & Kane M (2012). Quality and rigor of the concept mapping methodology: a pooled study analysis. Evaluation and Program Planning, 35(2), 236–245. [DOI] [PubMed] [Google Scholar]
  31. Rosenberg T, & Pace M (2006). Burnout among mental health professionals: Special considerations for the marriage and family therapist. Journal of Marital and Family Therapy, 32(1), 87–99. doi:doi: 10.1111/j.1752-0606.2006.tb01590.x [DOI] [PubMed] [Google Scholar]
  32. Rosenheck RA (2001). Organizational process: A missing link between research and practice. Psychiatric Services, 52, 1607–1612. [DOI] [PubMed] [Google Scholar]
  33. Salyers MP, McGuire AB, Rollins AL, Bond GR, Mueser KT, & Macy VR (2010). Integrating assertive community treatment and illness management and recovery for consumers with severe mental illness. Community Mental Health Journal, 46(4), 319–329. doi: 10.1007/s10597-009-9284-6 [DOI] [PubMed] [Google Scholar]
  34. Schein E (2010). Organizational culture and leadership. San Francisco, CA: John Wiley and Sons. [Google Scholar]
  35. Sommerfeld DH, Aarons GA, Naqvi JB, Holden J, Perivoliotis D, Mueser KT, & Granholm E (2018). Stakeholder Perspectives on Implementing Cognitive Behavioral Social Skills Training on Assertive Community Treatment Teams. Adm Policy Ment Health. doi: 10.1007/s10488-018-0904-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Torres-Gonzalez F, Ibanez-Casas I, Saldivia S, Ballester D, Grandon P, Moreno-Kustner B, . . . Gomez-Beneyto M (2014). Unmet needs in the management of schizophrenia. Neuropsychiatric Disease and Treatment, 10, 97–110. doi: 10.2147/NDT.S41063 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Torrey WC, Drake RE, Dixon L, Burns BJ, Flynn L, Rush AJ, . . . Klatzker D (2001). Implementing evidence-based practices for persons with severe mental illnesses. Psychiatric Services, 52(1), 45–50. [DOI] [PubMed] [Google Scholar]
  38. Trochim WM, Cook JA, & Setze RJ (1994). Using concept mapping to develop a conceptual framework of staff’s views of a supported employment program for individuals with severe mental illness. Journal of Consulting and Clinical Psychology, 62(4), 766–775. [DOI] [PubMed] [Google Scholar]
  39. Willging C, Aarons GA, Trott E, Green AE, Finn CV, Ehrhart KH, & Hecht D (2016). Contracting and procurement for evidence-based interventions in public-sector human services: A case study. Administration and Policy in Mental Health and Mental Health Services Research, 43(5), 675–692. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Williams CHJ (2008). Cognitive behaviour therapy within assertive outreach teams: barriers to implementation: A qualitative peer audit. Journal of Psychiatric and Mental Health Nursing, 15(10), 850–856. doi:doi: 10.1111/j.1365-2850.2008.01332.x [DOI] [PubMed] [Google Scholar]

RESOURCES